EDB Postgres Distributed 6.0.1 release notes v6.0.1

New built-in connection manager which handles routing of connections automatically and allows enforcing of read-only connections to non-leader.

The PGD CLI now allows initial cluster setup as well as adding nodes from command-line using pgd node setup command.

Transform the sequences in distributed based on the bdr.default_sequence_kind GUC when creating/joining a bdr group instead of when creating the node as done in older versions.

Set the startvalue for galloc sequences to the following valid number after the last used by the local sequence. With this change, when creating distributed sequences and specifically galloc, there is no need to adjust the startvalue based on what might be already used.

Automatic synchronization and reconciliation of node states is now enabled by default. This means that nodes will automatically synchronize their state with the leader node and reconcile any differences without requiring manual intervention. Read more in the documentation.

The node_uuid uniquely identifies instance of a node of a given name. Random node_uuid is generated when node is created and remains constant for the lifetime of the node. The node_id column is now derived from node_uuid instead of node name.

For the time being a node needs to be fully parted before before node of the same name can be rejoined, this may be relaxed in future releases to permit rejoin as soon as part_node process for the old instance has commenced and before it completed.

For the time being upgrades from older PGD versions and mixed-version operation in clusters with older PGD nodes are not supported. This limitation will be addressed in future releases.

Replication origin and slot names now use node uuid and thus correspond to particular incarnation of a node of a given name. Similarly node group uuid is used instead of group name. Hash of database name is used in lieu of database name.

Please note that origin and node names should be treated as opaque identifiers from user's perspective, one shouldn't rely on the structure of these names nor expect these to be particularly meaningful to a human operator.

The new naming scheme is as follows:

Slots Naming Convention

• normal slot to a node => bdr_node_<targetuuid>_<dbhash>
• join slot for node => bdr_node_<targetuuid>_<dbhash>_tmp
• group slot for a topgroup => bdr_group_<topgroupuuid>_<dbhash>
• slot for any forwarding + lead to lead => bdr_node_<targetuuid>_<originidhex>_<dbhash>
• analytics slot => bdr_analytics_<groupuuid>_<dbhash>
• decoding slot => bdr_decoder_<topgroupuuid>_<dbhash>

Origins Naming Convention:

• normal origin to a node => bdr_<originuuid>_<dbhash>
• fwd origin to a source node => bdr_<originuuid>_<sourceoidhex>_<dbhash>

Ensure that no more than three node groups (one top group and two subgroups) can exist at any given time. If the limit is exceeded, an error is raised.

Don't allow PGD Essential clusters to join more than 4 data nodes.

bdr.wait_node_confirm_lsn() will look at the confirmed_flush_lsn of the given node when available, otherwise it will query pg_replication_origin_progress() of that node, and wait for the specified LSN to be reached by said node.

In previous versions, subscriber-only nodes could only be added to node groups of type "subscriber-only". In PGD 6, a subscriber-only node can be also be added to a data node group by specifying node_kind='subscriber_only' when using create_node. The join_node_group can then be done using a data node group.

Returns name of analytics slot. This merely produces the correct name irrespective of whether analytics feature is in use.

Added to complement the addition of the node_uuid column to bdr.node and bdr.local_node

Both pk_exists and multiple_unique_conflicts conflict types can now resolve more than one conflicting row by removing any old rows that are part of the conflict. The multiple_unique_conflicts now defaults to update_if_newer resolver, so it does not throw error by default anymore.

The backend_type now shows consistent worker type for PGD workers without the extra process identification. The wait_event_type and wait_event include more wait events now, instead of showing "extension" for some events. Also, connection management related columns are added to show real client address/port and whether the session is read-only.

From PGD 6.0.0, bdr.part_node functionality is enhanced to remove the parted node’s metadata automatically from all nodes in the cluster.

• For local node, it will remove all the node metadata, including information about remote nodes.
• For remote node, it removes only metadata for that specific node. Hence with this release
• A node will remain in PART_CLEANUP state till group slots of all nodes are caught up to all the transactions originating from the PARTED node
• A node will not remain in PARTED state as the node is removed as soon as it moves to PARTED state.

Add the --summary and --options flags to pgd node show command to filter the output of the pgd node show command. This also maintains symmetry with other show commands.

Add the bdr.lock_table_locking and bdr.truncate_locking GUCs to list of GUCs verfied in pgd cluster verify command.

Changing a column's type command which causes the whole table to be rewritten and the change isn't binary coercible is now supported:

CREATE TABLE foo (c1 int,c2 int, c3 int, c4 box, UNIQUE(c1, c2) INCLUDE(c3,c4)); ALTER TABLE foo ALTER c1 TYPE bigint; – results into table rewrite

This also includes support for ALTER TYPE when using the USING clause:

CREATE TABLE foo (id serial primary key,data text); ALTER TABLE foo ALTER data TYPE BYTEA USING data::bytea;

Table rewrites can hold an AccessExclusiveLock for extended periods on larger tables.

The restrictions on non-immutable ALTER TABLE... ADD COLUMN calls have been removed.

Roles and tablespaces are now synchronized before the schema is restored from the join source node. If there are already existing roles or tablespaces (or EPAS profiles, they will be updated to have the same settings, passwords etc. as the ones from the join source node. System roles (i.e. the ones created by initdb) are not synchronized.

The new bdr.node_group_config_summary view contains detailed information about group options, including effective value, source of the effective value, default value, whether the value can be inherited, etc. This is in similar spirit to pg_settings

New lock type leader DML lock is used by default for locking DDL statements that need to block DML. This lock locks on write-leaders only, no requiring all nodes to participate in the locking operation. Old behavior can be restored by adjusting bdr.ddl_locking configuration parameter.

Functions run_on_nodes, run_on_all_nodes and run_on_group now sets bdr.xact_replication to off by default.

The auto value for bdr.default_replica_identity changed to REPLICA IDENTITY FULL. This setting prevents some edge cases in conflict detection between inserts, updates and deletes across node crashes and recovery.

When the PGD group is created and the database of the initial PGD node is not empty (i.e. has some tables with data) the REPLICA IDENTITY of all tables will be set according to bdr.default_replica_identity.

Tablespace operations CREATE/ALTER/DROP TABLESPACE are now replicated as a DDL operation. Where users are running a configuration with multiple nodes on the same machine, you will need to enable the developer option allow_in_place_tablespace.

Improve the formating of the log messages to be more readable and symmetrical with Postgres log messages.

Add the recommended_value column to the result of the pgd cluster verify --settings command. The column will not be displayed in tabular output but will be displayed in JSON output.

The output for the commands with tabular output are now sorted by the resource name. Commands that display more than one resource will sort output by each resource column in order.

Subscriber-only nodes now receive data only after it has been replicated to majority of data nodes. This does not require any special configuration. Subsequently bdr.standby_slot_names and bdr.standby_slots_min_confirmed options are removed as similar physical standby functionality is provided in pg_failover_slots extension and in PG17+.

The GUC bdr.enable_auto_sync_reconcile was off by default, but is made on by default in 6.0. This GUC setting ensures that when a node is down for some time, all other nodes get caught up equally with respect to this node automatically. It also ensures that if there are any prepared transactions that are orphaned by the node going down, they are resolved, either aborted or committed as per the rules of the commit scope that created them.

Remove the old (PGD 5 and below) CLI commands, which were deprecated but supported for backward compatibility.

Previously, non-data nodes would attempt to handle, but not process commit scope logic, which could lead to confusing, albeit harmless log messages.

This provides greater visibility into the node cloning process and assists with debugging possible issues.

Routing (and by extension raft) is now enabled by default on data-groups (subgroups with data nodes).

As it is not possible to roll back a group join, it can not form part of an idempotent transaction.

pause_in_standby has been deprecated since PGD 5.0.0. Logical standby nodes should be specified as such when executing bdr.create_node()

Unlogged BDR sequences may display unexpected behaviour following a server crash. Existing unlogged BDR sequences may be converted to logged ones.

The pgd cluster show command failed with an error for clock drift if only one node was up and running in a N node cluster. The command now returns valid output for the other components, health and summary, while reporting an appropriate error for clock-drift.

The pgd node show command crashed if a non-existent node is specified to the command. The command is fixed to fail gracefully with appropriate error message.

The pgd replication show command previously crashed when formatting EPAS timestamps.

When parting a given node, that same node may have subscriptions whose origin was already parted and the group dropped. Previously this would break PGD, and has since been fixed.

The num_writers option, used in bdr.alter_node_group_option() and bdr.alter_node_group_config() should be positive or -1.

Fixes the case where an update to a table with non-unique indexes results in the ERRORconcurrent INSERT when looking for delete rows, which breaks replication.

Ensure that any custom value set in the deprecated GUC bdr.raft_election_timeoutis applied to the replacement bdr.raft_global_election_timeout

During subscription reconfiguration, if there is no change required to a subscription, do not enable it since it could have been disabled explicitly by the user. Skip reconfiguring subscriptions if there are no leadership changes.

Subscriber-only nodes will no longer attempt to take a lock on the cluster when running DDL. The DDL will be executed locally and not replicated to other nodes.

Fixed non-transactional WAL message acknowledgment by downstream that could cause a WAL sender to never exit during fast database system shutdown.

Fixed deadlock between bdr_init_physical cleaning unwanted node data and concurrent monitoring queries.

Fixed an issue when new node joining the cluster finishes CATCHUP phase before getting its replication progress against all data nodes. This may cause new node being out of sync with the cluster.

In some cases, warning messages referred to timeshard when the sequence was actually snowflakeid.