Pipelines PGFS with S3-compatible storage

Overview: S3-compatible storage

PGFS uses the s3: prefix to indicate an S3-compatible storage system.

The general syntax for using S3 is this:

select pgfs.create_storage_location(
               'storage_location_name',
               's3://bucket_name',
               options => '{}'::JSONB,
               credentials => '{}'::JSONB
       );

The `options` argument in JSON format offers the following settings:

Option	Description
`bucket`	The bucket name. Used to explicitly provide the bucket name if it can't be passed in the url.
`skip_signature`	Disable HMAC authentication (set this to "true" when you're not providing access_key_id/secret_access_key in the credentials).
`region`	The region of the S3-compatible storage system. If the region is not specified, the client will attempt auto-discovery.
`endpoint`	The endpoint of the S3-compatible storage system.

The `credentials` argument in JSON format offers the following settings:

Option	Description
`access_key_id`	HMAC credentials (this is often the "username" in non-AWS S3 providers)
`secret_access_key`	HMAC credentials (this is often the "password" in non-AWS S3 providers)
`session_token`	A session token can be used instead of HMAC credentials

Example: AWS S3 public bucket

This is an example of using a public bucket on AWS S3. Public buckets do not require authentication.

SELECT pgfs.create_storage_location('edb_ai_example_images', 's3://public-ai-team',
                                    options => '{"region": "eu-central-1", "skip_signature": "true"}'
       );

Example: AWS S3 private bucket

This is an example of using a private bucket on AWS S3. Private buckets require authentication. Here, we're using HMAC credentials.

SELECT pgfs.create_storage_location('internal_ai_project', 's3://my-company-ai-images',
                                    options => '{"region": "eu-central-1"}',
                                    credentials => '{"access_key_id": "secret", "secret_access_key":"secret!"}'
       );

Example: non-AWS S3 / S3-compatible

This is an example of using an S3-compatible system like minIO. The endpoint must be provided in this case; it can only be omitted when using AWS S3.

SELECT pgfs.create_storage_location('ai_images_local_minio', 's3://my-ai-images',
                                    options => '{"endpoint": "https://minio-api.apps.local"}',
                                    credentials => '{"access_key_id": "my_username", "secret_access_key":"my_password"}'
       );

← Prev

Pipelines PGFS with local file storage

↑ Up

PGFS functions for Pipelines

PGFS settings

Could this page be better? Report a problem or suggest an addition!