fix: the overlay's file sets MTU = 1280 on each node (keel#119) #125

2026-10-09T03:22:48-03:00

marcos-mendez commented

2026-10-09 03:22:48 -03:00

(Migrated from github.com)

Closes #119.

Cause

The MTU of wg0. Timeouts and RTT are not the cause. keel wrote no MTU line, so wg-quick took the MTU of the route to an endpoint less 80: 1420 on web-1 and web-3 (BR1), and 1412 on web-2 (BR2, PPPoE, path 1492 bytes). WireGuard pads each packet up to the MTU of its own interface, so a full TCP segment from web-1 (1340 bytes of data) was 1500 bytes on the wire. The path dropped it.

The failure was captured at 04:53 on the real nodes. web-1 sent the segment 9 times in 26.6 s, and each time it was 1500 bytes. The ICMPv6 "packet too big" came at 04:53:39. web-2 stopped at its 30 s answer timeout at 04:53:42 (did not answer: timed out). The first fragmented packet left at 04:53:49. RTT BR1 to BR2 is 4.2 ms on wg0, with 0 % loss. The channel is plain HTTP over wg0, not TLS. When the PMTU entry is in the cache, 30 of 30 fetches answer in 0.04 s. All the data is in evidence/2026-10-09-br2-channel.md.

Fix

keel.network.wireguard.render writes MTU = 1280 (OVERLAY_MTU). This is the minimum MTU of IPv6, and it is 1360 bytes on the wire over IPv6. It is the same on every node.
parse reads that line as keel's own (Parsed.mtu). It names another value as a problem.
switch.live_peers: a change of the MTU line causes a bounce, because wg set cannot set an MTU. The first apply on a node of 0.23.7 bounces wg0 one time. A change of the peers alone stays live.
Docs: spec.md, apply.md, inspect.md. Changelog 0.23.11 (rebased onto main: 0.23.8 was taken by keel#121, 0.23.9 by keel#126, 0.23.10 by keel#118).

Test

tests/test_network_overlay_mtu_netns.py uses the real wg-quick. Node B is behind a veth of 1492 bytes that drops longer packets and sends no ICMPv6. A sends a 4 KiB answer over TCP through wg0, with a deadline of 5 s.

0.23.7 file (no MTU line): B gets 0 of 4096 bytes. wg0 is 1420 on A and 1412 on B, the same as on the real nodes. This failure was seen first (RED).
The file keel renders now: B gets 4096 bytes, and wg0 is 1280 on both nodes.

The new test is added to the mesh / trixie job. Unit tests cover the render, the parse and the bounce rule. Line coverage of wireguard.py is 100 % and of switch.py is 99 %.

Closes #119. ## Cause The MTU of wg0. Timeouts and RTT are not the cause. keel wrote no MTU line, so wg-quick took the MTU of the route to an endpoint less 80: 1420 on web-1 and web-3 (BR1), and 1412 on web-2 (BR2, PPPoE, path 1492 bytes). WireGuard pads each packet up to the MTU of its own interface, so a full TCP segment from web-1 (1340 bytes of data) was 1500 bytes on the wire. The path dropped it. The failure was captured at 04:53 on the real nodes. web-1 sent the segment 9 times in 26.6 s, and each time it was 1500 bytes. The ICMPv6 "packet too big" came at 04:53:39. web-2 stopped at its 30 s answer timeout at 04:53:42 (`did not answer: timed out`). The first fragmented packet left at 04:53:49. RTT BR1 to BR2 is 4.2 ms on wg0, with 0 % loss. The channel is plain HTTP over wg0, not TLS. When the PMTU entry is in the cache, 30 of 30 fetches answer in 0.04 s. All the data is in `evidence/2026-10-09-br2-channel.md`. ## Fix - `keel.network.wireguard.render` writes `MTU = 1280` (`OVERLAY_MTU`). This is the minimum MTU of IPv6, and it is 1360 bytes on the wire over IPv6. It is the same on every node. - `parse` reads that line as keel's own (`Parsed.mtu`). It names another value as a problem. - `switch.live_peers`: a change of the MTU line causes a bounce, because `wg set` cannot set an MTU. The first apply on a node of 0.23.7 bounces wg0 one time. A change of the peers alone stays live. - Docs: spec.md, apply.md, inspect.md. Changelog 0.23.11 (rebased onto main: 0.23.8 was taken by keel#121, 0.23.9 by keel#126, 0.23.10 by keel#118). ## Test `tests/test_network_overlay_mtu_netns.py` uses the real wg-quick. Node B is behind a veth of 1492 bytes that drops longer packets and sends no ICMPv6. A sends a 4 KiB answer over TCP through wg0, with a deadline of 5 s. - 0.23.7 file (no MTU line): B gets 0 of 4096 bytes. wg0 is 1420 on A and 1412 on B, the same as on the real nodes. This failure was seen first (RED). - The file keel renders now: B gets 4096 bytes, and wg0 is 1280 on both nodes. The new test is added to the mesh / trixie job. Unit tests cover the render, the parse and the bounce rule. Line coverage of wireguard.py is 100 % and of switch.py is 99 %.

github-advanced-security[bot] (Migrated from github.com) reviewed

2026-10-09 03:23:28 -03:00

tests/overlay_mtu_netns.py

					
				@ -0,0 +104,4 @@

				    """A's answer: 4 KiB to whoever sends a line"""

				    server = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)

				    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

				    server.bind((f"{OVERLAY}1", PORT))

CodeQL / Binding a socket to all network interfaces

Binding a socket to all interfaces (using '') is a security risk.

Show more details

## CodeQL / Binding a socket to all network interfaces Binding a socket to all interfaces (using [''](1)) is a security risk. [Show more details](https://github.com/Keel-Linux/keel/security/code-scanning/30)

Sign in to join this conversation.

No reviewers