database: a local pg_receivewal makes a standalone PostgreSQL server read as a primary in keel diff #201

New issue

Closed

opened 2026-10-16 15:14:58 -03:00 by navigator · 3 comments

navigator commented

2026-10-16 15:14:58 -03:00

Owner

Found in run 2 of the LNPP 19.0-3 release test across BR1 and BR2, 2026-10-16 (keel 0.23.41 in the image). CT 9121 is the etcd witness: an LNPP node with database.server: {engine: postgresql, role: standalone} and no pair.

At the end of the test, keel diff on the witness exits 14:

database.server.role: drift (declared standalone, observed primary)

The witness has one streaming sender, and it is local:

$ psql -c 'select application_name,client_addr,state,backend_start from pg_stat_replication'
pg_receivewal@17-main.service | | streaming | 2026-10-16 17:40:20+00
$ systemctl status pg_receivewal@17-main   # active (running) since the first boot, slot pg_receivewal_service

pg_receivewal@17-main.service is the WAL archival unit of postgresql-common. It started at the first boot (its unit file says disabled). keel.inspect.dbengines._postgresql_role reads any row of pg_stat_replication as "a replica is streaming from here", so a standalone server reads as a primary. The same unit runs on the pair nodes (on the standby it restarts in a loop).

In run 1 (same spec, keel 0.23.39) the witness gave keel diff exit 0, so the result depends on whether the unit streams at that moment.

Expected, one of: keel does not count a sender from the local host (no client_addr) or the slot pg_receivewal_service as a replica; or the image does not start the archival unit where keel converges the role.

Found in run 2 of the LNPP 19.0-3 release test across BR1 and BR2, 2026-10-16 (keel 0.23.41 in the image). CT 9121 is the etcd witness: an LNPP node with `database.server: {engine: postgresql, role: standalone}` and no pair. At the end of the test, `keel diff` on the witness exits 14: ``` database.server.role: drift (declared standalone, observed primary) ``` The witness has one streaming sender, and it is local: ``` $ psql -c 'select application_name,client_addr,state,backend_start from pg_stat_replication' pg_receivewal@17-main.service | | streaming | 2026-10-16 17:40:20+00 $ systemctl status pg_receivewal@17-main # active (running) since the first boot, slot pg_receivewal_service ``` `pg_receivewal@17-main.service` is the WAL archival unit of postgresql-common. It started at the first boot (its unit file says disabled). `keel.inspect.dbengines._postgresql_role` reads any row of `pg_stat_replication` as "a replica is streaming from here", so a standalone server reads as a primary. The same unit runs on the pair nodes (on the standby it restarts in a loop). In run 1 (same spec, keel 0.23.39) the witness gave `keel diff` exit 0, so the result depends on whether the unit streams at that moment. Expected, one of: keel does not count a sender from the local host (no `client_addr`) or the slot `pg_receivewal_service` as a replica; or the image does not start the archival unit where keel converges the role.

keel-bridge commented

2026-10-16 15:17:34 -03:00

Member

The bridge mirrored this item to GitHub: https://github.com/Keel-Linux/keel/issues/171
Comments, labels and the open or closed state sync both ways.

The bridge mirrored this item to GitHub: https://github.com/Keel-Linux/keel/issues/171 Comments, labels and the open or closed state sync both ways.

navigator commented

2026-10-16 16:15:29 -03:00

Author

Owner

The image side is fixed by Keel-Linux/keel-lnpp#4 (LNPP 19.0-4): a preset disables pg_receivewal@ and pg_compresswal@, and the build fails without it. Run 3 of the release test showed the unit disabled and inactive on the 3 CTs, and keel diff exit 0 on the witness. Left open for keel, which still reads any local sender as a replica, and for pg_dump@17-main.timer, which the first boot presets also enable (a weekly dump into /var/backups).

navigator commented

2026-10-16 21:07:35 -03:00

Author

Owner

Fixed in keel 0.23.42 (keel#203): the role query counts only the senders with a client address. The image side is the preset of keel-lnpp#4. LNPP 19.0-5 shows no drift on all 3 nodes.