CI containers cannot reach our own public names #5

New issue

Closed

opened 2026-09-27 11:17:44 -03:00 by marcos-mendez · 1 comment

marcos-mendez commented

2026-09-27 11:17:44 -03:00

(Migrated from github.com)

A container booted by the appliance gate on the self hosted runner resolves
archive.keellinux.org to a loopback address and meets its own web server
instead of the archive. Measured 2026-09-27 in a WordPress gate run:

W: Failed to fetch https://archive.keellinux.org/dists/trixie/InRelease
   SSL connection failed: certificate verify failed [IP: 127.0.1.1 443]

Two causes stacked. The gate's containers sit on lxcbr0 with a unique local
address and no route out, so they cannot use IPv6 to reach anything of ours.
Falling back to IPv4 they ask the runner's resolver, and the public services
VM's /etc/hosts carries 127.0.1.1 keellinux.org keellinux; every public
name is a CNAME to that one, so all six answer as loopback for IPv4.

The appliance is not at fault, and the WordPress boot test now says so out
loud: it skips the four archive proofs with a notice naming this issue rather
than failing, and the update path is proved by hand on the build host, where
containers hold a global address.

Closes when a gate container can reach archive.keellinux.org and read a
signed index from it. Two candidate fixes, neither chosen: give the gate's
containers a global address, as the build host does, or point those names at
an address the containers can route to instead of loopback. The second is one
line in the VM's /etc/hosts and touches a machine that serves the site and
the forum, so it needs the maintainer.

A container booted by the appliance gate on the self hosted runner resolves `archive.keellinux.org` to a loopback address and meets its own web server instead of the archive. Measured 2026-09-27 in a WordPress gate run: W: Failed to fetch https://archive.keellinux.org/dists/trixie/InRelease SSL connection failed: certificate verify failed [IP: 127.0.1.1 443] Two causes stacked. The gate's containers sit on `lxcbr0` with a unique local address and no route out, so they cannot use IPv6 to reach anything of ours. Falling back to IPv4 they ask the runner's resolver, and the public services VM's `/etc/hosts` carries `127.0.1.1 keellinux.org keellinux`; every public name is a CNAME to that one, so all six answer as loopback for IPv4. The appliance is not at fault, and the WordPress boot test now says so out loud: it skips the four archive proofs with a notice naming this issue rather than failing, and the update path is proved by hand on the build host, where containers hold a global address. Closes when a gate container can reach `archive.keellinux.org` and read a signed index from it. Two candidate fixes, neither chosen: give the gate's containers a global address, as the build host does, or point those names at an address the containers can route to instead of loopback. The second is one line in the VM's `/etc/hosts` and touches a machine that serves the site and the forum, so it needs the maintainer.

marcos-mendez commented

2026-09-29 23:25:56 -03:00

(Migrated from github.com)

Fixed on the public services VM on 2026-09-30, the maintainer's call. The gate's containers ask lxc-net's dnsmasq on lxcbr0, which read this VM's /etc/hosts (127.0.1.1 keellinux.org keellinux, and every public name is a CNAME to it). Now /etc/default/lxc-net sets LXC_DHCP_CONFILE=/etc/lxc/dnsmasq-keel.conf with no-hosts, no-resolv and the site resolvers 2804:710:d0:10::53 and ::54 (backup: /etc/default/lxc-net.keel-bak-20260930). Containers now get 2804:710:d0:5::13, which the VM serves directly; they already had IPv6 out through the VM's NAT66. The VM's own resolution, the site and the forum are unchanged.

Proof: the WordPress boot gate on master read the archive and verified its signature (apt-get update read https://archive.keellinux.org trixie and verified its signature), and on keel-wordpress#13 the whole gate passed, the update path included. The next failure it exposed was the test's own parse of apt's *** marker, fixed in keel-wordpress#13.

Fixed on the public services VM on 2026-09-30, the maintainer's call. The gate's containers ask lxc-net's dnsmasq on lxcbr0, which read this VM's /etc/hosts (`127.0.1.1 keellinux.org keellinux`, and every public name is a CNAME to it). Now `/etc/default/lxc-net` sets `LXC_DHCP_CONFILE=/etc/lxc/dnsmasq-keel.conf` with `no-hosts`, `no-resolv` and the site resolvers `2804:710:d0:10::53` and `::54` (backup: `/etc/default/lxc-net.keel-bak-20260930`). Containers now get `2804:710:d0:5::13`, which the VM serves directly; they already had IPv6 out through the VM's NAT66. The VM's own resolution, the site and the forum are unchanged. Proof: the WordPress boot gate on master read the archive and verified its signature (`apt-get update read https://archive.keellinux.org trixie and verified its signature`), and on keel-wordpress#13 the whole gate passed, the update path included. The next failure it exposed was the test's own parse of apt's `***` marker, fixed in keel-wordpress#13.