fix: the old primary turns its database read only before it releases the VIP (keel#138) #142

Open

marcos-mendez wants to merge 1 commit from fix/planned-promote-quiesce into fix/promote-announce-in-background

marcos-mendez commented

2026-10-10 14:54:53 -03:00

(Migrated from github.com)

Fixes #138. Stacked on #141 (both bump debian/changelog); its base is #141's branch.

What waited, and why

Evidence: Keel-Linux/handbook#63, 17:31:48 to 17:32:14.

The new primary's 10 s (follow 17:31:59 to 17:32:09): not the I/O stop and not the follow lock. Follow waited for the VIP on wg0, which the promote adds only after its announcement. The announcement to web-2, a member outside the pair that was down, timed out (10 s). #141 (keel#137) takes that out of the handover.
The write that waited 15 s (seq 128): it committed on the old primary at 17:31:59.343 and waited for the replica's semi-synchronous acknowledgement. Meanwhile the old primary dropped the VIP for the release, so the answer had no address to come from; the client waited until its own 15 s timeout.

Change

keel/mesh/vipserve.py (released) now calls quiesced() before vipnode.release. On a node that declares a database, SET GLOBAL read_only = ON (keel/system/dbreadonly.py, quiesce, at most 15 s) returns once the commits in flight have ended; the replica, still connected, acknowledges them. Only then does the address go, and no new write is taken. A failure is logged and the release goes on; the database follows the VIP's state after, as before.

Also: docs/replication.md, debian/changelog 0.23.16.

Tests

The read only comes before the address is dropped.
quiesced asks the database only where one is declared, and logs a failure.
quiesce's statement, its timeout and its failures.
tests/mariadb_netns.py: two planned promotes (A to B and back) while C writes at the VIP. For each: the longest gap between two acknowledged writes under 3 s, no write waiting 10 s, no acknowledged row lost. The measured numbers go here once CI ran.

Test plan

tests / coverage
mariadb / trixie (the planned gaps are printed)
the other trixie jobs
package / changelog

Fixes #138. Stacked on #141 (both bump `debian/changelog`); its base is #141's branch. ## What waited, and why Evidence: Keel-Linux/handbook#63, 17:31:48 to 17:32:14. - **The new primary's 10 s** (follow 17:31:59 to 17:32:09): not the I/O stop and not the follow lock. Follow waited for the VIP on `wg0`, which the promote adds only after its announcement. The announcement to web-2, a member outside the pair that was down, timed out (10 s). #141 (keel#137) takes that out of the handover. - **The write that waited 15 s** (seq 128): it committed on the old primary at 17:31:59.343 and waited for the replica's semi-synchronous acknowledgement. Meanwhile the old primary dropped the VIP for the release, so the answer had no address to come from; the client waited until its own 15 s timeout. ## Change `keel/mesh/vipserve.py` (`released`) now calls `quiesced()` before `vipnode.release`. On a node that declares a database, `SET GLOBAL read_only = ON` (`keel/system/dbreadonly.py`, `quiesce`, at most 15 s) returns once the commits in flight have ended; the replica, still connected, acknowledges them. Only then does the address go, and no new write is taken. A failure is logged and the release goes on; the database follows the VIP's state after, as before. Also: `docs/replication.md`, `debian/changelog` 0.23.16. ## Tests - The read only comes before the address is dropped. - `quiesced` asks the database only where one is declared, and logs a failure. - `quiesce`'s statement, its timeout and its failures. - `tests/mariadb_netns.py`: two planned promotes (A to B and back) while C writes at the VIP. For each: the longest gap between two acknowledged writes under 3 s, no write waiting 10 s, no acknowledged row lost. The measured numbers go here once CI ran. ## Test plan - [ ] tests / coverage - [ ] mariadb / trixie (the planned gaps are printed) - [ ] the other trixie jobs - [ ] package / changelog

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin fix/planned-promote-quiesce:fix/planned-promote-quiesce

git switch fix/planned-promote-quiesce

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch fix/promote-announce-in-background

git merge --no-ff fix/planned-promote-quiesce

git switch fix/planned-promote-quiesce

git rebase fix/promote-announce-in-background

git switch fix/promote-announce-in-background

git merge --ff-only fix/planned-promote-quiesce

git switch fix/planned-promote-quiesce

git rebase fix/promote-announce-in-background

git switch fix/promote-announce-in-background

git merge --no-ff fix/planned-promote-quiesce

git switch fix/promote-announce-in-background

git merge --squash fix/planned-promote-quiesce