Degrading commit scope rules v5.6

SYNCHRONOUS_COMMIT, GROUP COMMIT, and CAMO each have the optional capability of degrading the requirements for transactions when particular performance thresholds are crossed.

When a node is applying a transaction and that transaction times out, it can be useful to trigger a process of degrading the requirements of the transaction to be completed, rather than just rolling back.

DEGRADE ON offers a route for gracefully degrading the commit scope rule of a transaction. At its simplest, DEGRADE ON takes a timeout and a second set of commit scope operations that the commit scope can gracefully degrade to.

For instance, after 20ms or 30ms timeout, the requirements for satisfying a commit scope could degrade from ALL (node_group_name) GROUP COMMIT to MAJORITY (node_group_name) GROUP COMMIT, making the transactions apply more steadily.

You can also require that the write leader be the originator of a transaction in order for the degrade clause to be triggered. This can be helpful in "split brain scenarios" where you have, say, 2 data nodes and a witness node. Supposing there is a network split between the two data nodes and you have connections to both of the data nodes, only one of them will be allowed to degrade, because only one of them will be elected leader through the raft election with the witness node.

Behavior

There are two parts to how the generalized DEGRADE clause behaves as it is applied to transactions.

Once during the commit, while the commit being processed is waiting for responses that satisfy the commit scope rule, PGD checks for a timeout and, if the timeout has expired, the commit being processed is reconfigured to wait for the commit scope rule in the DEGRADE clause. In fact, by this point, the commit scope rule in the DEGRADE clause might already be satisfied.

This mechanism alone is insufficient for the intended behavior, as this alone would mean that every transaction—even those that were certain to degrade due to connectivity issues—must wait for the timeout to expire before degraded mode kicks in, which would severely affect performance in such degrading-cluster scenarios.

To avoid this, the PGD manager process also periodically (every 5s) checks the connectivity and apply rate (the one in bdr.node_replication_rates) and if there are commit scopes that would degrade at that point based on the current state of replication, they will be automatically degraded—such that any transaction using that commit scope when processing after that uses the degraded rule instead of waiting for timeout—until the manager process detects that replication is moving swiftly enough again.

SYNCHRONOUS_COMMIT and GROUP COMMIT

Both SYNCHRONOUS_COMMIT and GROUP COMMIT have timeout and require_write_lead parameters, with defaults of 0 and false respectively. You should probably always set the timeout, as the default of 0 causes an instant degrade. You can also require that the write leader be the originator of the transaction in order to switch to degraded mode (again, default is false).

Both SYNCHRONOUS_COMMIT and GROUP COMMIT also have options regarding which rule you can degrade to—which depends on which rule you are degrading from.

First of all, you can degrade to asynchronous operation:

ALL (left_dc) SYNCHRONOUS_COMMIT DEGRADE ON (timeout=20s) TO ASYNC

You can also degrade to a less restrictive commit group with the same commit scope kind (again as long as the kind is either SYNCHRONOUS_COMMIT or GROUP COMMIT). For instance, you can degrade as follows:

ALL (left_dc) SYNCHRONOUS_COMMIT DEGRADE ON (timeout=20s) TO MAJORITY (left_dc) SYNCHRONOUS_COMMIT

or as follows:

ANY 3 (left_dc) SYNCHRONOUS_COMMIT DEGRADE ON (timeout=20s) TO ANY 2 (left_dc) SYNCHRONOUS_COMMIT

But you cannot degrade from SYNCHRONOUS_COMMIT to GROUP COMMIT or the other way around.

CAMO

While CAMO supports both the same timeout and require_write_lead parameters (with the same defaults, 0 and false respectively), the options are simpler in that you can only degrade to asynchronous operation.

ALL (left_dc) CAMO DEGRADE ON (timeout=20ms, require_write_lead=true) TO ASYNC

Again, you should set the timeout parameter, as the default is 0.