Pipelines PGFS

Pipelines has a Postgres File System (PGFS) extension that allows you to access external storage. In particular, it allows you to store and retrieve files from either a public or private S3-compatible bucket or a local file system. The aidb extension addresses external storage mapped through PGFS as storage locations and abstracts away the details of the underlying storage.

PGFS is a key component of Pipelines, allowing you to work with data that isn't stored in the database, either because it's too large or because it isn't structured in a way that's suitable for a relational database.

PGFS is implemented as a foreign data wrapper (FDW) and installed as an extension. The extension provides a set of functions to interact with the file system from the database.

PGFS storage locations and AIDB volumes

To use a PGFS storage location with AIDB, a volume must be created in AIDB:

• PGFS storage location: Represents an external storage provider with all necessary configuration like a path and credentials.
• AIDB volume: Connects a sotrage location to AIDB. Multiple volumes for the same storage location are possible. In a volume, you configure the data format in the destination and optionally, a sub-path.

Installing the PGFS extension

See Installing to learn how to install the PGFS extension as part of the Pipelines installation.

Using PGFS

See the PGFS management functions reference for details on the functions available in the PGFS extension. With these functions you can create and manage storage locations, to be later used through AIDB volumes.