Pipelines PGFS with S3-compatible storage

Overview: S3-compatible storage

PGFS uses the s3: prefix to indicate an S3-compatible storage system.

The general syntax for using S3 is:

select pgfs.create_storage_location(
               'storage_location_name',
               's3://bucket_name',
               options => '{}',
               credentials => '{}'
       );

Settings for the `options` argument in JSON format settings

Option	Description
`bucket`	The bucket name. Used to explicitly provide the bucket name if it can't be passed in the url.
`skip_signature`	Disable HMAC authentication. (Set this to "true" when you're not providing access_key_id/secret_access_key in the credentials.)
`region`	The region of the S3-compatible storage system. If the region isn't specified, the client attempts auto-discovery.
`endpoint`	The endpoint of the S3-compatible storage system.
`allow_http`	Whether the endpoint uses plain HTTP (rather than HTTPS/TLS). Set this to `true` if your endpoint starts with `http://`.

Settings for the `credentials` argument in JSON format

Option	Description
`access_key_id`	HMAC credentials (often the "username" in non-AWS S3 providers)
`secret_access_key`	HMAC credentials (often the "password" in non-AWS S3 providers)
`session_token`	A session token that can be used instead of HMAC credentials

Example: AWS S3 public bucket

This example uses a public bucket on AWS S3. Public buckets don't require authentication.

SELECT pgfs.create_storage_location('edb_ai_example_images', 's3://public-ai-team',
                                    options => '{"region": "eu-central-1", "skip_signature": "true"}'
       );

Example: AWS S3 private bucket

This example uses a private bucket on AWS S3. Private buckets require authentication. The example uses HMAC credentials.

SELECT pgfs.create_storage_location('internal_ai_project', 's3://my-company-ai-images',
                                    options => '{"region": "eu-central-1"}',
                                    credentials => '{"access_key_id": "secret", "secret_access_key":"secret!"}'
       );

Example: non-AWS S3 / S3-compatible with HTTPS

This example uses an S3-compatible system like minIO. The endpoint must be provided in this case. You can omit it only when using AWS S3.

SELECT pgfs.create_storage_location('ai_images_local_minio', 's3://my-ai-images',
                                    options => '{"endpoint": "https://minio-api.apps.local"}',
                                    credentials => '{"access_key_id": "my_username", "secret_access_key":"my_password"}'
       );

Example: non-AWS S3 / S3-compatible with HTTP

This example uses an S3-compatible system like minIO. The endpoint must be provided in this case. You can omit it only be when using AWS S3.

In this case, the server doesn't use TLS encryption, so the code configures a plain HTTP connection.

SELECT pgfs.create_storage_location('ai_images_local_minio', 
                                    's3://my-ai-images',
                                    options => '{"endpoint": "http://minio-api.apps.local", "allow_http":"true"}',
                                    credentials => '{"access_key_id": "my_username", "secret_access_key":"my_password"}'
       );

← Prev

Pipelines PGFS with local file storage

↑ Up

PGFS functions for Pipelines

PGFS settings

Could this page be better? Report a problem or suggest an addition!