test: the unplanned failover of a cloud simple MariaDB pair, measured at 250 ms #124

Merged

marcos-mendez merged 2 commits from test/mariadb-cloud-simple-crash into main

2026-10-09 06:09:21 -03:00

marcos-mendez commented

2026-10-09 02:26:35 -03:00

(Migrated from github.com)

Stacked on #122 (rebased on main after #121 and #127). Measures #108 and #118 in cloud simple, the mode of the real pair: no etcd lease, the VIP moved by the members' channel.

Case

tests/mariadb_simple_netns.py on the mariadb job's nodes and links (250 ms ±25 ms, 2 % loss, measured first), every spec in cloud simple with etcd disabled. A primary, B replica, connected and semi-synchronous; C writes one row at the VIP every 0.1 s. A's server killed and A cut off; exactly 2 s later, keel database promote --old-primary-gone on B. New job: mariadb-simple / trixie.

Target

15 s from the promote's start, plus what the safety review of #121 keeps without etcd: the promote waits the channel's full 10 s connect timeout for each of its three asks of the dead old primary (epoch, release, announcement). So the assert is 15 + 3 × 10 = 45 s, and 0 acknowledged rows lost.

Measured (`0be60b6`)


promote started after the kill	2.0 s
VIP on B after the kill	32.5 s
B writable after the VIP landed	0.54 s
first acknowledged write after the kill	34.5 s (32.5 s after the promote's start)
write downtime (last ok to first ok)	35.8 s
acknowledged rows lost	0

Of the 32.5 s, 30 s are the three full timeouts to the dead old primary. Without them the case is at about 3 s from the promote's start. With the 1.5 s bound (the first version of #121, before the safety review) the same case measured 10.7 s.

CI record

Green at 0be60b6 at the first run, no rerun. The case kills A only once C's writes were acknowledged. 8f0549d (the rebase on #127) failed mariadb / trixie (#122's harness, fixed there) and vip / trixie at g_promote_b (etcd did not answer, the known failure). Earlier heads: see the history of this PR.

Stacked on #122 (rebased on main after #121 and #127). Measures #108 and #118 in cloud simple, the mode of the real pair: no etcd lease, the VIP moved by the members' channel. ## Case `tests/mariadb_simple_netns.py` on the `mariadb` job's nodes and links (250 ms ±25 ms, 2 % loss, measured first), every spec in cloud simple with etcd disabled. A primary, B replica, connected and semi-synchronous; C writes one row at the VIP every 0.1 s. A's server killed and A cut off; exactly 2 s later, `keel database promote --old-primary-gone` on B. New job: `mariadb-simple / trixie`. ## Target 15 s from the promote's start, plus what the safety review of #121 keeps without etcd: the promote waits the channel's full 10 s connect timeout for each of its three asks of the dead old primary (epoch, release, announcement). So the assert is 15 + 3 × 10 = 45 s, and 0 acknowledged rows lost. ## Measured (0be60b6) | | | | --- | --- | | promote started after the kill | 2.0 s | | VIP on B after the kill | 32.5 s | | B writable after the VIP landed | 0.54 s | | first acknowledged write after the kill | 34.5 s (32.5 s after the promote's start) | | write downtime (last ok to first ok) | 35.8 s | | acknowledged rows lost | 0 | Of the 32.5 s, 30 s are the three full timeouts to the dead old primary. Without them the case is at about 3 s from the promote's start. With the 1.5 s bound (the first version of #121, before the safety review) the same case measured 10.7 s. ## CI record Green at 0be60b6 at the first run, no rerun. The case kills A only once C's writes were acknowledged. 8f0549d (the rebase on #127) failed `mariadb / trixie` (#122's harness, fixed there) and `vip / trixie` at `g_promote_b` (`etcd did not answer`, the known failure). Earlier heads: see the history of this PR.