Pipelines PGFS with Google Cloud Storage

Overview: Google Cloud Storage

PGFS uses the gs: prefix to indicate a Google Cloud Storage bucket.

The general syntax for using GCS is:

select pgfs.create_storage_location(
               'storage_location_name',
               'gs://bucket_name',
               credentials => '{}'
       );

credentials

The credentials argument is optional. It takes a value in JSON format and offers the following settings.

See the Google Cloud documentation for more information on how to manage service account keys.

You can also set up these options via the equivalent environment variables to facilitate authentication in managed environments such as Google Kubernetes Engine.

Example: private GCS bucket

SELECT pgfs.create_storage_location('edb_ai_example_images', 'gs://my-company-ai-images',
                                    credentials => '{"google_service_account_key_file": "/var/run/gcs.json"}'
       );

Example: authentication in GKE

Ensure that the GOOGLE_APPLICATION_CREDENTIALS or the GOOGLE_SERVICE_ACCOUNT_KEY_FILE environment variable is set on your PostgreSQL pod. Then, PGFS picks them up.

SELECT pgfs.create_storage_location('edb_ai_example_images', 'gs://my-company-ai-images');