Pipelines reference - Preparers
This reference documentation for Pipelines preparers includes information on the functions and views available in the aidb extension related to preparers. See Usage and Examples for more details.
Views
aidb.preparers
| Column | Type | Description |
|---|---|---|
| id | INTEGER | |
| name | TEXT | Name of the preparer. |
| operation | aidb.DataPreparationOperation | The kind of processing step to perform. |
| destination_schema | TEXT | Schema of the destination table to store the output data. |
| destination_table | TEXT | Name of the destination table to store the output data. |
| destination_key_column | TEXT | Column of the destination table that references the key in source data. |
| destination_data_column | TEXT | Column of the destination table to store the output data. |
| options | JSONB | Configuration options for the data preparation operation. Uses the same API as the data preparation primitives. |
| source_type | TEXT | Type of source data the preparer is working with. Can be either 'Table' or 'Volume'. |
| source_schema | TEXT | Schema of the table with the source data the preparer will process. Applies only to preparers of Table source type. |
| source_table | TEXT | Name of the table with the source data the preparer will process. Applies only to preparers of Table source type. |
| source_data_column | TEXT | Column in the source table with the source data the preparer will process. Applies only to preparers of Table source type. |
| source_key_column | TEXT | Name of the key column in the source table for reference with the output processed data. Applies only to preparers of Table source type. |
| source_volume_name | TEXT | Name of the volume to use as a data source. Applies only to preparers of Volume source type. |
Types
aidb.DataPreparationOperation
The aidb.DataPreparationOperation type is an enum that represents the different types of preprocessing steps that can be performed:
- ChunkText
- SummarizeText
- ParseHtml
- ParsePdf
Functions
aidb.create_preparer_for_table
Creates a preparer with a source data table.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| name | TEXT | Required | Name of the preparer |
| operation | aidb.DataPreparationOperation | Required | Type of data preparation operation |
| source_table | TEXT | Required | Name of the source data table |
| source_data_column | TEXT | Required | Column in the source table containing the raw data |
| destination_table | TEXT | Required | Name of the destination table |
| destination_data_column | TEXT | Required | Column in the destination table for processed data |
| source_key_column | TEXT | 'id' | Column to use as key to reference the rows |
| destination_key_column | TEXT | 'id' | Key column in the destination table that references the source_key_column |
| options | JSONB | '{}'::JSONB | Configuration options for the data preparation operation. Uses the same API as the data preparation primitives. |
aidb.create_preparer_for_volume
Creates a preparer for a given PGFS volume.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| name | TEXT | Required | Name of the preparer |
| operation | aidb.DataPreparationOperation | Required | Type of data preparation operation |
| source_volume_name | TEXT | Required | Name of the source volume containing the raw data |
| destination_table | TEXT | Required | Name of the destination table |
| destination_data_column | TEXT | Required | Column in the destination table for processed data |
| destination_key_column | TEXT | 'id' | Key column in the destination table that uniquely identifies the processed data |
| options | JSONB | '{}'::JSONB | Configuration options for the data preparation operation. Uses the same API as the data preparation primitives. |
aidb.bulk_data_preparation
Executes the configured data preparation operation on all data from the specified preparer’s source.
Parameters
| Parameter | Type | Description |
|---|---|---|
| preparer_name | TEXT | Name of the preparer |
aidb.delete_preparer
Deletes the preparer's configuration.
Parameters
| Parameter | Type | Description |
|---|---|---|
| preparer_name | TEXT | Name of preparer to delete |
Note
This function doesn't delete the destination table or any data in it.
Could this page be better? Report a problem or suggest an addition!