0031 (ADR-009): Database failover semantics #36

New issue

Open

opened 2026-09-30 10:10:51 -03:00 by marcos-mendez · 0 comments

marcos-mendez commented

2026-09-30 10:10:51 -03:00

(Migrated from github.com)

Decision (maintainer, 2026-09-30). The replica is a hot standby and does not serve traffic. Replication is synchronous: a commit is confirmed on the primary only after the replica has received it (PostgreSQL synchronous_commit; MariaDB semi-synchronous, see below). Failback: the operator chooses automatic or manual at installation, as 0020 says, default manual; Keel implements automatic failback, which happens only after the old primary has fully caught up.

Handbook: docs/decisions/0031-database-failover-semantics.md (handbook#26)

Amends: 0020 (hard part 2: asynchronous becomes synchronous / semi-synchronous for the database; failback choice and manual default kept, amended only by committing to implement automatic; the replica's read-only web tier becomes no traffic at all), 0013 (read endpoints no longer point at replicas of a pair).

Review notes (resolved by the maintainer, 2026-09-30)

MariaDB has no fully synchronous primary/replica replication. Resolved: semi-synchronous (rpl_semi_sync, AFTER_SYNC), which waits for the replica to receive, not apply, and falls back to async after rpl_semi_sync_master_timeout. Galera is not what this chooses.
PostgreSQL: synchronous_commit with synchronous_standby_names.
Failback. Resolved: the operator keeps 0020's choice, default manual; automatic failback after full catch-up is implemented (roadmap Phases 2 and 5). 0020's rejoin choice and safeguard unchanged.

Implementation tasks

Timeout policy for both engines: how long MariaDB waits before falling back, what a primary that fell back reports, what PostgreSQL does while its replica is gone.
Automatic failback with the catch-up condition.

First tasks

Measure semi-sync on the MariaDB pair (keel 0.11.x): commit latency over the mesh, fallback behaviour at timeout.
Write the timeout policy as spec fields.

**Decision (maintainer, 2026-09-30).** The replica is a hot standby and does not serve traffic. Replication is synchronous: a commit is confirmed on the primary only after the replica has received it (PostgreSQL synchronous_commit; MariaDB semi-synchronous, see below). Failback: the operator chooses automatic or manual at installation, as 0020 says, default **manual**; Keel implements automatic failback, which happens only after the old primary has fully caught up. **Handbook:** `docs/decisions/0031-database-failover-semantics.md` (handbook#26) **Amends:** 0020 (hard part 2: asynchronous becomes synchronous / semi-synchronous for the database; failback choice and manual default kept, amended only by committing to implement automatic; the replica's read-only web tier becomes no traffic at all), 0013 (read endpoints no longer point at replicas of a pair). **Review notes** (resolved by the maintainer, 2026-09-30) - [x] MariaDB has no fully synchronous primary/replica replication. Resolved: semi-synchronous (`rpl_semi_sync`, `AFTER_SYNC`), which waits for the replica to receive, not apply, and falls back to async after `rpl_semi_sync_master_timeout`. Galera is not what this chooses. - [x] PostgreSQL: `synchronous_commit` with `synchronous_standby_names`. - [x] Failback. Resolved: the operator keeps 0020's choice, default manual; automatic failback after full catch-up is implemented (roadmap Phases 2 and 5). 0020's rejoin choice and safeguard unchanged. **Implementation tasks** - [ ] Timeout policy for both engines: how long MariaDB waits before falling back, what a primary that fell back reports, what PostgreSQL does while its replica is gone. - [ ] Automatic failback with the catch-up condition. **First tasks** - [ ] Measure semi-sync on the MariaDB pair (keel 0.11.x): commit latency over the mesh, fallback behaviour at timeout. - [ ] Write the timeout policy as spec fields.