Pipelines API reference v7

This page covers the full public API for AI pipelines: the types and enums used across the API, the views for inspecting pipeline state, the core CRUD and execution functions, and the configuration helper functions for pipeline steps and vector indexes. For model configuration helpers, see the Models reference.

Types

aidb.PipelineAutoProcessingMode

Controls how a pipeline automatically processes new or changed data.

CREATE TYPE PipelineAutoProcessingMode AS ENUM (
    'Live',
    'Background',
    'Disabled'
);

aidb.PipelineDataFormat

Specifies the format of source data the pipeline processes.

CREATE TYPE PipelineDataFormat AS ENUM (
    'Text',
    'Image',
    'Pdf'
);

aidb.PipelineSourceType

Indicates the type of data source a pipeline reads from.

CREATE TYPE PipelineSourceType AS ENUM (
    'Table',
    'Volume',
    'Empty'
);

aidb.PipelineDestinationType

Indicates the type of destination a pipeline writes to.

CREATE TYPE PipelineDestinationType AS ENUM (
    'Table',
    'Volume',
    'Empty'
);

aidb.PipelineStepOperation

Defines the operation performed by a pipeline step.

CREATE TYPE PipelineStepOperation AS ENUM (
    'ChunkText',
    'SummarizeText',
    'ParseHtml',
    'ParsePdf',
    'PerformOcr',
    'KnowledgeBase',
    'PdfToImage',
    'SemanticKB'
);

aidb.PipelineStatus

Represents the current processing state of a pipeline.

CREATE TYPE PipelineStatus AS ENUM (
    'Stale',
    'Processing',
    'UpToDate',
    'NoResults',
    'Failed',
    'Unknown',
    'PartialErrors',
    'BlockingErrors'
);

aidb.ErrorBlocking

Categorizes how a logged error affects pipeline processing. Two dimensions: scope (record vs pipeline) and temporality (temporary vs permanent).

CREATE TYPE ErrorBlocking AS ENUM (
    'RecordTemporary',
    'RecordPermanent',
    'PipelineTemporary',
    'PipelinePermanent'
);

aidb.DistanceOperator

Specifies the distance metric used for vector similarity search.

CREATE TYPE DistanceOperator AS ENUM (
    'L2',
    'InnerProduct',
    'Cosine',
    'L1',
    'Hamming',
    'Jaccard'
);

Domains

aidb.pipeline_name_50

A TEXT domain enforcing that pipeline names are no longer than 50 characters.

aidb.background_sync_interval

An INTERVAL domain enforcing that background sync intervals are between 1 second and 2 days (inclusive).

Views

aidb.pipelines

Also accessible as aidb.pipes. Lists all registered pipelines and their configuration, including source, destination, processing mode, and step definitions.

Example

SELECT name, source, destination, auto_processing FROM aidb.pipelines;

aidb.pipeline_metrics

Also accessible as aidb.pipem. Shows current processing statistics for each pipeline.

Example

SELECT * FROM aidb.pipeline_metrics;

       pipeline       | auto processing | table: unprocessed rows | volume: scans completed | count(source records) | count(destination records) |  Status  | count(record errors) | count(blocking errors)
----------------------+-----------------+-------------------------+-------------------------+-----------------------+----------------------------+----------+----------------------+------------------------
 pipeline__7471a      | Background      |                       0 |                         |                     5 |                          5 | UpToDate |                    0 |                      0
 pipeline__7471b      | Background      |                       0 |                         |                     5 |                          5 | UpToDate |                    0 |                      0
 animal_facts_kb      | Disabled        |                       0 |                         |                    99 |                        372 | UpToDate |                    0 |                      0
 animal_facts_kb_bert | Disabled        |                       0 |                         |                    99 |                        372 | UpToDate |                    0 |                      0
 mpkb_pipe_int        | Disabled        |                       0 |                         |                     2 |                          4 | UpToDate |                    0 |                      0
 mpkb_pipe_text       | Disabled        |                       0 |                         |                     2 |                          4 | UpToDate |                    0 |                      0
(6 rows)

Functions

aidb.create_pipeline

Creates a new pipeline with a source, up to 10 sequential processing steps, and an optional destination.

Parameters

Returns

Example

-- Single-step pipeline: chunk text from a table into a destination table
SELECT aidb.create_pipeline(
    name                => 'my_chunker',
    source              => 'source_docs',
    source_key_column   => 'id',
    source_data_column  => 'body',
    destination         => 'chunked_docs',
    step_1              => 'ChunkText',
    step_1_options      => aidb.chunk_text_config(200, 250, 25),
    auto_processing     => 'Live'
);

-- Multi-step pipeline: parse PDF, then embed into a knowledge base
SELECT aidb.create_pipeline(
    name                => 'pdf_to_kb',
    source              => 'pdf_volume',
    destination         => 'my_kb',
    step_1              => 'ParsePdf',
    step_1_options      => aidb.pdf_parse_config(),
    step_2              => 'KnowledgeBase',
    step_2_options      => aidb.knowledge_base_config('my_model', 'Text'),
    auto_processing     => 'Background',
    background_sync_interval => '60 seconds'
);

aidb.update_pipeline

Updates the auto-processing settings for an existing pipeline.

Parameters

Example

SELECT aidb.update_pipeline('my_chunker', auto_processing => 'Background', background_sync_interval => '5 minutes');

aidb.delete_pipeline

Deletes a pipeline and its configuration. Doesn't delete the source or destination tables.

Parameters

Example

SELECT aidb.delete_pipeline('my_chunker');

aidb.run_pipeline

Manually triggers a pipeline to execute immediately, regardless of its auto_processing mode.

Parameters

Example

SELECT aidb.run_pipeline('my_chunker');

aidb.get_pipeline_metrics

Returns current processing statistics for a pipeline.

Parameters

Returns

Example

SELECT * FROM aidb.get_pipeline_metrics('my_chunker');

Error log functions

Functions in this group manage the per-pipeline error log table. See Error log for the end-to-end workflow.

aidb.get_error_logs

Returns rows from a pipeline's error log table, optionally filtered by source record, step, or error category. Results are ordered by failed_at DESC, id DESC.

Parameters

Raises an exception if the pipeline doesn't exist. Returns an empty result set if the pipeline's error log table is missing (legacy pipelines created before 7.2.0 upgrade into the table automatically).

Returns

Example

SELECT * FROM aidb.get_error_logs('my_pipeline');

-- Errors for a specific source record
SELECT * FROM aidb.get_error_logs('my_pipeline', p_source_id => 'doc_42');

-- Blocking errors at step 2
SELECT * FROM aidb.get_error_logs(
    'my_pipeline',
    p_pipeline_step  => 2::SMALLINT,
    p_error_category => 'PipelinePermanent'
);

aidb.clear_error_logs

Deletes the specified entries from a pipeline's error log table and returns the count of rows actually removed.

Parameters

Raises an exception if the pipeline doesn't exist.

Returns

BIGINT — number of rows deleted.

Example

SELECT aidb.clear_error_logs('my_pipeline', ARRAY[1, 3, 7]);

-- Clear by filter via composition
SELECT aidb.clear_error_logs(
    'my_pipeline',
    ARRAY(SELECT id FROM aidb.get_error_logs(
        'my_pipeline',
        p_pipeline_step  => 1::SMALLINT,
        p_error_category => 'RecordPermanent'
    ))
);

aidb.get_error_log_summary

Returns error counts grouped by (pipeline_step, step_operation, error_category) for a single pipeline. One row per combination that has at least one error.

Parameters

Raises an exception if the pipeline doesn't exist. Returns an empty result set if the pipeline's error log table is missing.

Returns

Example

SELECT * FROM aidb.get_error_log_summary('my_pipeline');

-- Total error count
SELECT sum(error_count) FROM aidb.get_error_log_summary('my_pipeline');

aidb.get_all_error_summaries

Returns the same row shape as aidb.get_error_log_summary with pipeline_name prepended, rolled up across every pipeline. Pipelines with zero errors, or whose error log table is missing, are omitted.

Parameters

None.

Returns

Example

SELECT pipeline_name, sum(error_count) AS total_errors
FROM aidb.get_all_error_summaries()
GROUP BY pipeline_name;

aidb.requeue_pipeline_errors

Introduced in 7.5.0. Deletes the specified record-level error log entries and marks their source records dirty on the pipeline's state surface. The call itself does no processing; the next normal pipeline run re-processes the requeued records. See Requeuing failed records for the workflow.

Parameters

Raises if the pipeline doesn't exist. Ids that are absent from the error log are reported as not_found rather than raising.

Returns

One row per input id, ordered by ascending error_id.

• requeued: record-level row deleted and source marked dirty. Table sources enqueue a state event; volume sources set a one-shot retry_requested_at marker.
• skipped_pipeline_level: entry has a NULL source_id and is left in place. Fix the underlying issue and re-run, or dismiss with aidb.clear_error_logs.
• not_found: id was absent from the error log, or the pipeline has no error log table (legacy pipelines or Empty sources).

Concurrent requeues on overlapping ids serialize on per-row FOR UPDATE locks; non-overlapping id sets proceed in parallel. Requeue doesn't block aidb.run_pipeline or the background worker.

Background pipelines drain requeued rows automatically on the next worker tick. Disabled or Live pipelines need a follow-up aidb.run_pipeline call. On a Disabled-mode volume source, requeue sets retry_requested_at and emits a NOTICE because it can't schedule a drain; manual aidb.run_pipeline re-reads the whole volume.

Example

SELECT * FROM aidb.requeue_pipeline_errors('my_pipeline', ARRAY[1, 2, 99]);

 error_id | action
----------+------------------------
        1 | requeued
        2 | requeued
       99 | not_found
(3 rows)

Pipeline step config helpers

These functions return a JSONB configuration object for use in step_N_options parameters of aidb.create_pipeline. None of them have a parameter for intermediate_destination — see Intermediate storage for how to add it to a step's options.

aidb.chunk_text_config

Configures a ChunkText step to split text into smaller segments.

Parameters

Example

-- Chunk into ~200 character segments, max 250, with 25-character overlap
SELECT aidb.chunk_text_config(200, 250, 25, 'chars');

aidb.summarize_text_config

Configures a SummarizeText step to summarize text using a language model.

Parameters

Example

SELECT aidb.summarize_text_config(
    'my_llm',
    chunk_config => aidb.chunk_text_config(100, 100, 10, 'words'),
    prompt       => 'Summarize concisely',
    strategy     => 'reduce',
    reduction_factor => 4
);

aidb.ocr_config

Configures a PerformOcr step to extract text from images using a model.

Parameters

Example

SELECT aidb.ocr_config('my_ocr_model');

aidb.html_parse_config

Configures a ParseHtml step to extract text from HTML content.

Parameters

Example

SELECT aidb.html_parse_config();

aidb.pdf_parse_config

Configures a ParsePdf step to extract content from PDF documents.

Parameters

Example

SELECT aidb.pdf_parse_config(allow_partial_parsing => true);

aidb.knowledge_base_config

Configures a KnowledgeBase step to compute and store embeddings.

Parameters

Example

SELECT aidb.knowledge_base_config(
    'my_embedding_model',
    'Text',
    distance_operator => 'Cosine',
    vector_index      => aidb.vector_index_hnsw_config(m => 16, ef_construction => 64)
);

aidb.knowledge_base_config_from_kb

Configures a KnowledgeBase step to attach a pipeline to an existing knowledge base rather than creating a new one.

Parameters

Example

SELECT aidb.knowledge_base_config_from_kb('Text');

aidb.inference_config

Builds an inference configuration object for use with language model steps such as SummarizeText. All parameters are optional.

Parameters

Example

SELECT aidb.inference_config(
    system_prompt => 'You are a technical summarizer.',
    temperature   => 0.3,
    max_tokens    => 512
);

Vector index config helpers

These functions return a JSONB configuration for the vector_index parameter of aidb.knowledge_base_config.

aidb.vector_index_hnsw_config

Configures an HNSW index (pgvector).

Parameters

The following table shows how each distance_operator value maps to a pgvector ops class:

Example

SELECT aidb.vector_index_hnsw_config(m => 16, ef_construction => 64);

aidb.vector_index_ivfflat_config

Configures a pgvector IVFFlat index.

Parameters

Example

SELECT aidb.vector_index_ivfflat_config(lists => 100);

aidb.vector_index_chord_hnsw_config

Configures a VectorChord HNSW index.

Parameters

Example

SELECT aidb.vector_index_chord_hnsw_config(m => 16, ef_construction => 64);

aidb.vector_index_chord_vchordq_config

Configures a VectorChord Vchordq index.

Parameters

Example

SELECT aidb.vector_index_chord_vchordq_config(lists => '100', spherical_centroids => true);

aidb.vector_index_hsphere_optimized_config

Configures an HSphere Optimized index.

Parameters

Example

SELECT aidb.vector_index_hsphere_optimized_config(clusters => 256, precision_val => 0.95);

aidb.vector_index_disabled_config

Disables automatic vector index creation. Use this when your embedding dimensions exceed 2000 or when you want to manage indexes manually.

Example

SELECT aidb.vector_index_disabled_config();

Model config helpers

Model config helpers have moved to the Models reference page.

Configuration

Session-settable GUCs that control AI pipeline behavior.

aidb.pipeline_error_warnings

Superusers can change this setting per session, per role, per database, or in postgresql.conf. Other roles need an explicit GRANT SET ON PARAMETER aidb.pipeline_error_warnings before they can SET it.

When enabled, each error logged to a pipeline's error log table also emits a Postgres WARNING. Errors persist to the log regardless of this setting. Disable to silence the warnings without affecting persistence:

SET aidb.pipeline_error_warnings = false;