Configuring HARP for cluster management v4.3.8

The HARP configuration file follows a standard YAML-style formatting that was simplified for readability. This file is located in the /etc/harp directory by default and is named config.yml

You can explicitly provide the configuration file location to all HARP executables by using the -f/--config argument.

Standard configuration

HARP essentially operates as three components:

• HARP Manager
• HARP Proxy
• harpctl

Each of these use the same standard config.yml configuration format, which always include the following sections:

• cluster.name — The name of the cluster to target for all operations.
• dcs — DCS driver and connection configuration for all endpoints.

This means a standard preamble is always included for HARP operations, such as the following:

cluster:
  name: mycluster

dcs:
  ...

Other sections are optional or specific to the named HARP component.

Cluster name

The name entry under the cluster heading is required for all interaction with HARP. Each HARP cluster has a name for both disambiguation and for labeling data in the DCS for the specific cluster.

HARP Manager writes information about the cluster here for consumption by HARP Proxy and harpctl. HARP Proxy services direct traffic to nodes in this cluster. The harpctl management tool interacts with this cluster.

DCS settings

Configuring the consensus layer is key to HARP functionality. Without the DCS, HARP has nowhere to store cluster metadata, can't hold leadership elections, and so on. Therefore this portion of the configuration is required, and certain elements are optional.

Specify all elements under a section named dcs with these multiple supplementary entries:

• driver: Required type of consensus layer to use. Currently can be etcd or bdr. Support for bdr as a consensus layer is experimental. Using bdr as the consensus layer reduces the additional software for consensus storage but expects a minimum of three full BDR member nodes to maintain quorum during database maintenance.

• endpoints: Required list of connection strings to contact the DCS. List every node of the DCS here if possible. This ensures HARP continues to function as long as a majority of the DCS can still operate and be reached by the network.

  Format when using etcd as the consensus layer is as follows:

  → WrapCopy

  dcs:
    endpoints:
      - host1:2379
      - host2:2379
      - host3:2379

  Format when using the experimental bdr consensus layer is as follows:

  → WrapCopy

  dcs:
  # only DSN format is supported
    endpoints:
      - "host=host1 port=5432 dbname=bdrdb user=postgres"
      - "host=host2 port=5432 dbname=bdrdb user=postgres"
      - "host=host3 port=5432 dbname=bdrdb user=postgres"

  Currently, bdr consensus layer requires the first endpoint to point to the local postgres instance.

• request_timeout: Time in milliseconds to consider a request as failed. If HARP makes a request to the DCS and receives no response in this time, it considers the operation as failed. This can cause the issue to be logged as an error or retried, depending on the nature of the request. Default: 250.

The following DCS SSL settings apply only when driver: etcd is set in the configuration file:

• ssl: Either on or off to enable SSL communication with the DCS. Default: off

• ssl_ca_file: Client SSL certificate authority (CA) file.

• ssl_cert_file: Client SSL certificate file.

• ssl_key_file: Client SSL key file.

Example

This example shows how to configure HARP to contact an etcd DCS consisting of three nodes:

dcs:
  driver: etcd
  endpoints:
    - host1:2379
    - host2:2379
    - host3:2379

Health check API settings

HARP provides HTTP(S) health check APIs. Add the following settings only if health check APIs are required. Specify all elements under a section named http with these multiple supplementary entries:

• enable: Set to true to enable HTTP(S) health check APIs. Defaults to false.

• host: Host name of HTTP(S) API server. Typically the same as the hostname of the HARP service as it's running within the HARP service. Defaults to 0.0.0.0.

• port: Port number of HTTP(S) server. Defaults to 8080.

• probes.timeout: Timeout value for the API endpoints. Defaults to 10s.

• secure: Enable HTTPS. Defaults to false.

The following http settings apply and are mandatory if secure: true is set:

• cert_file: HTTP server SSL certificate file. Mandatory if secure is set to true.

• key_file: HTTP server SSL key file. Mandatory if secure is set to true.

Example

http:
  enable: true
  host: "0.0.0.0"
  port: 8080
  secure: false
  cert_file: ""
  key_file: ""
  probes:
    timeout: 10s

HARP Manager specific

Besides the generic service options required for all HARP components, Manager needs other settings:

• log_level: One of DEBUG, INFO, WARNING, ERROR, or CRITICAL, which might alter the amount of log output from HARP services.

• name: Required name of the Postgres node represented by this Manager. Since the Manager can only represent a specific node, that node is named here and it is used as the name of this Manager. If this is a BDR node, it must match the value used at node creation when executing the bdr.create_node(node_name, ...) function and as reported by the bdr.local_node_summary.node_name view column. Alphanumeric characters and underscores only.

• start_command: This can be used instead of the information in DCS for starting the database to monitor. This is required if using bdr as the consensus layer.

• status_command: This can be used instead of the information in DCS for the Harp Manager to determine whether the database is running. This is required if using bdr as the consensus layer.

• stop_command: This can be used instead of the information in DCS for stopping the database.

• db_retry_wait_min: The initial time in seconds to wait if Harp Manager cannot connect to the database before trying again. Harp Manager will increase the wait time with each attempt, up to the db_retry_wait_max value.

• db_retry_wait_max: The maximum time in seconds to wait if Harp Manager cannot connect to the database before trying again.

Thus a complete configuration example for HARP Manager might look like this:

cluster:
  name: mycluster

dcs:
  driver: etcd
  endpoints:
    - host1:2379
    - host2:2379
    - host3:2379

manager:
  name: node1
  log_level: INFO

This configuration is essentially the DCS contact information, any associated service customizations, the name of the cluster, and the name of the node. All other settings are associated with the node and is stored in the DCS.

Read the Node bootstrapping for more about specific node settings and initializing nodes to be managed by HARP Manager.

HARP Proxy specific

Some configuration options are specific to HARP Proxy. These affect how the daemon operates and thus are currently located in config.yml.

Specify Proxy-based settings under a proxy heading, and include:

• location: Required name of location for HARP Proxy to represent. HARP Proxy nodes are directly tied to the location where they are running, as they always direct traffic to the current lead master node. Specify location for any defined proxy.

• log_level: One of DEBUG, INFO, WARNING, ERROR, or CRITICAL, which might alter the amount of log output from HARP services. Defaults to INFO

• name: Name of this specific proxy. Each proxy node is named to ensure any associated statistics or operating state are available in status checks and other interactive events.

• type: Defaults to builtin to use the built-in passthrough proxy. Can also be set to pgbouncer to use the deprecated (since HARP 2.1) embedded PgBouncer support.

• pgbouncer_bin_dir: Directory where PgBouncer binaries are located. If HARP is configured to use PgBouncer, it needs to know where the PgBouncer binaries are located. This can be depend on the platform or distribution, so it has no default. Otherwise, the assumption is that the appropriate binaries are in the environment's PATH variable. Deprecated (since HARP 2.1).

Example

HARP Proxy requires the cluster name, DCS connection settings, location, and name of the proxy in operation. For example:

cluster:
  name: mycluster

dcs:
  driver: etcd
  endpoints:
    - host1:2379
    - host2:2379
    - host3:2379

proxy:
  name: proxy1
  location: dc1

All other attributes are obtained from the DCS on proxy startup.

Runtime directives

While it is possible to configure HARP Manager, HARP Proxy, or harpctl with a minimum of YAML in the config.yml file, some customizations are held in the DCS. These values must either be initialized via bootstrap or set specifically with harpctl set directives.

Cluster-wide

Set these settings under a cluster YAML heading during bootstrap, or modify them with a harpctl set cluster command.

• event_sync_interval: Time in milliseconds to wait for synchronization. When events occur in HARP, they do so asynchronously across the cluster. HARP managers start operating immediately when they detect metadata changes, and HARP proxies might pause traffic and start reconfiguring endpoints. This is a safety interval that roughly approximates the maximum amount of event time skew that exists between all HARP components.

  For example, suppose Node A goes offline and HARP Manager on Node B commonly receives this event 5 milliseconds before Node C. A setting of at least 5 ms is then needed to ensure all HARP Manager services receive the event before they begin to process it.

  This also applies to HARP Proxy.

Node directives

You can change most node-oriented settings and then apply them while HARP Manager is active. These items are retained in the DCS after initial bootstrap, and thus you can modify them without altering a configuration file.

Set these settings under a node YAML heading during bootstrap, or modify them with a harpctl set node command.

• node_type: The type of this database node, either bdr or witness. You can't promote a witness node to leader.

• camo_enforcement: Whether to strictly enforce CAMO queue state. When set to strict, HARP never allows switchover or failover to a BDR CAMO partner node unless it's fully caught up with the entire CAMO queue at the time of the migration. When set to lag_only, only standard lag thresholds such as maximum_camo_lag are applied.

• dcs_reconnect_interval: The interval, measured in milliseconds, between attempts that a disconnected node tries to reconnect to the DCS.

  • Default: 1000.

• dsn: Required full connection string to the managed Postgres node. This parameter applies equally to all HARP services and enables micro-architectures that run only one service per container.

  Note

  HARP sets the sslmode argument to require by default and prevents connections to servers that don't require SSL. To disable this behavior, explicitly set this parameter to a more permissive value such as disable, allow, or prefer.

• db_data_dir: Required Postgres data directory. This is required by HARP Manager to start, stop, or reload the Postgres service. It's also the default location for configuration files, which you can use later for controlling promotion of streaming replicas.

• db_conf_dir: Location of Postgres configuration files. Some platforms prefer storing Postgres configuration files away from the Postgres data directory. In these cases, set this option to that expected location.

• db_log_file: Location of Postgres log file. Defaults to /tmp/pg_ctl.out.

• fence_node_on_dcs_failure: If HARP can't reach the DCS, several readiness keys and the leadership lease expire. This implicitly prevents a node from routing consideration. However, such a node isn't officially fenced, and the Manager doesn't stop monitoring the database if stop_database_when_fenced is set to false. Defaults to False.

• leader_lease_duration: Amount of time in seconds the lead master lease persists if not refreshed. This allows any HARP Manager a certain grace period to refresh the lock, before expiration allows another node to obtain the lead master lock instead. Defaults to 6.

• lease_refresh_interval: Amount of time in milliseconds between refreshes of the lead master lease. This essentially controls the time between each series of checks HARP Manager performs against its assigned Postgres node and when the status of the node is updated in the consensus layer. Defaults to 2000.

• max_dcs_failures: The amount of DCS request failures before marking a node as fenced according to fence_node_on_dcs_failure. This setting prevents transient communication disruptions from shutting down database nodes. Defaults to 10.

• maximum_lag: Highest allowable variance (in bytes) between last recorded LSN of previous lead master and this node before being allowed to take the lead master lock. This setting prevents nodes experiencing terminal amounts of lag from taking the lead master lock. Set to -1 to disable this check. Defaults to -1.

• maximum_camo_lag: Highest allowable variance (in bytes) between last received LSN and applied LSN between this node and its CAMO partners. This applies only to clusters where CAMO is both available and enabled. Thus this applies only to BDR EE clusters where pg2q.enable_camo is set. For clusters with particularly stringent CAMO apply queue restrictions, set this very low or even to 0 to avoid any unapplied CAMO transactions. Set to -1 to disable this check. Defaults to -1.

• ready_status_duration: Amount of time in seconds the node's readiness status persists if not refreshed. This is a failsafe that removes a node from being contacted by HARP Proxy if the HARP Manager in charge of it stops operating. Defaults to 30.

• db_bin_dir: Directory where Postgres binaries are located. As HARP uses Postgres binaries, such as pg_ctl, it needs to know where they're located. This can depend on the platform or distribution, so it has no default. Otherwise, the assumption is that the appropriate binaries are in the environment's PATH variable.

• priority: Any numeric value. Any node where this option is set to -1 can't take the lead master role, even when attempting to explicitly set the lead master using harpctl. Defaults to 100.

• stop_database_when_fenced: Rather than removing a node from all possible routing, stop the database on a node when it is fenced. This is an extra safeguard to prevent data from other sources than HARP Proxy from reaching the database or in case proxies can't disconnect clients for some other reason. Defaults to False.

• consensus_timeout: Amount of milliseconds before aborting a read or write to the consensus layer. If the consensus layer loses quorum or becomes unreachable, you want near-instant errors rather than infinite timeouts. This prevents blocking behavior in such cases. When using bdr as the consensus layer, the highest recognized timeout is 1000 ms. Defaults to 250.

• use_unix_socket: Specifies for HARP Manager to prefer to use Unix sockets to connect to the database. Defaults to False.

All of these runtime directives can be modified via harpctl. Consider if you want to decrease the lease_refresh_interval to 100ms on node1:

harpctl set node node1 lease_refresh_interval=100

Proxy directives

You can change certain settings to the proxy while the service is active. These items are retained in the DCS after initial bootstrap, and thus you can modify them without altering a configuration file. Many of these settings are direct mappings to their PgBouncer equivalent, and we will note these where relevant.

Set these settings under a proxies YAML heading during bootstrap, or modify them with a harpctl set proxy command. Properties set by harpctl set proxy require a restart of the proxy.

• auth_file: The full path to a PgBouncer-style userlist.txt file. HARP Proxy uses this file to store a pgbouncer user that has access to PgBouncer's Admin database. You can use this for other users as well. Proxy modifies this file to add and modify the password for the pgbouncer user. Deprecated (since HARP 2.1). Defaults to /etc/harp/userlist.txt.

• auth_type: The type of Postgres authentication to use for password matching. This is actually a PgBouncer setting and isn't fully compatible with the Postgres pg_hba.conf capabilities. We recommend using md5, pam cert, or scram-sha-256. Deprecated (since HARP 2.1). Defaults to md5.

• auth_query: Query to verify a user’s password with Postgres. Direct access to pg_shadow requires admin rights. It’s better to use a non-superuser that calls a SECURITY DEFINER function instead. If using TPAexec to create a cluster, a function named pgbouncer_get_auth is installed on all databases in the pg_catalog namespace to fulfill this purpose. Deprecated (since HARP 2.1).

• auth_user: If auth_user is set, then any user not specified in auth_file is queried through the auth_query query from pg_shadow in the database, using auth_user. The password of auth_user is taken from auth_file.

• client_tls_ca_file: Root certificate file to validate client certificates. Requires client_tls_sslmode to be set.

• client_tls_cert_file: Certificate for private key. Clients can validate it. Requires client_tls_sslmode to be set.

• client_tls_key_file: Private key for PgBouncer to accept client connections. Requires client_tls_sslmode to be set.

• client_tls_protocols: TLS protocol versions allowed for client connections. Allowed values: tlsv1.0, tlsv1.1, tlsv1.2, tlsv1.3. Shortcuts: all (tlsv1.0,tlsv1.1,tlsv1.2,tlsv1.3), secure (tlsv1.2,tlsv1.3), legacy (all). Defaults to secure.

• client_tls_sslmode: Whether to enable client SSL functionality. Possible values are disable, allow, prefer, require, verify-ca, and verify-full. Defaults to disable.

• database_name: Required name that represents the database clients use when connecting to HARP Proxy. This is a stable endpoint that doesn't change and points to the current node, database name, port, etc., necessary to connect to the lead master. You can use the global value * here so all connections get directed to this target regardless of database name.

• default_pool_size: The maximum number of active connections to allow per database/user combination. This is for connection pooling purposes but does nothing in session pooling mode. This is a PgBouncer setting. Deprecated (since HARP 2.1). Defaults to 25.

• ignore_startup_parameters: By default, PgBouncer allows only parameters it can keep track of in startup packets: client_encoding, datestyle, timezone, and standard_conforming_strings. All other parameters raise an error. To allow other parameters, you can specify them here so that PgBouncer knows that they are handled by the admin and it can ignore them. Often, you need to set this to extra_float_digits for Java applications to function properly. Deprecated (since HARP 2.1). Defaults to extra_float_digits.

• listen_address: IP addresses where Proxy should listen for connections. Used by the builtin and deprecated pgbouncer proxy. Defaults to 0.0.0.0.

• listen_port: System port where Proxy listens for connections. Used by the builtin and deprecated pgbouncer proxy. Defaults to 6432.

• max_client_conn: The total maximum number of active client connections that are allowed on the proxy. This can be many orders of magnitude greater than default_pool_size, as these are all connections that have yet to be assigned a session or have released a session for use by another client connection. This is a PgBouncer setting. Deprecated (since HARP 2.1). Defaults to 100.

• monitor_interval: Time in seconds between Proxy checks of PgBouncer. Since HARP Proxy manages PgBouncer as the actual connection management layer, it needs to periodically check various status and stats to verify it's still operational. You can also log or register some of this information to the DCS. Deprecated (since HARP 2.1). Defaults to 5.

• server_tls_protocols: TLS protocol versions are allowed for server connections. Allowed values: tlsv1.0, tlsv1.1, tlsv1.2, tlsv1.3. Shortcuts: all (tlsv1.0,tlsv1.1,tlsv1.2,tlsv1.3), secure (tlsv1.2,tlsv1.3), legacy (all). Defaults to secure.

• server_tls_sslmode: Whether to enable server SSL functionality. Possible values are disable, allow, prefer, require, verify-ca, and verify-full. Defaults to disable.

• session_transfer_mode: Method by which to transfer sessions. Possible values are fast, wait, and reconnect. Deprecated (since HARP 2.1). Defaults to wait.

  This property isn't used by the builtin proxy.

• server_transfer_timeout: The number of seconds Harp Proxy waits before giving up on a PAUSE and issuing a KILL command. Defaults to 30.

The following two options apply only when using the built-in proxy.

• keepalive: The number of seconds the built-in proxy waits before sending a keepalive message to an idle leader connection. Default to 5.

• timeout: The number of seconds the built-in proxy waits before giving up on connecting to the leader. Defaults to 1.

When using harpctl to change any of these settings for all proxies, use the global keyword in place of the proxy name. For example:

harpctl set proxy global max_client_conn=1000