Replicated storage: one writer at a time does not scale (multi-writer across sites) #58

New issue

Open

opened 2026-10-03 15:47:03 -03:00 by marcos-mendez · 0 comments

marcos-mendez commented

2026-10-03 15:47:03 -03:00

(Migrated from github.com)

Maintainer, 2026-10-03: 'um escritor por vez não escala'. This is an open problem to solve, not an accepted limit.

Where we are. The due diligence and the 250 ms bench (in progress) point to Syncthing for Keel Web's application files. It is the cheapest choice to run and to leave (plain files), and it works over poor networks (local reads, async copy). But it is single-writer per application: only the primary writes; the others receive. Active/passive is fine for a WordPress or a Nextcloud whose database is also single-primary, but it does not scale out writes, and it does not give active-active sites.

What a solution needs (the Keel premise: 250 ms RTT, jitter, loss, partitions):

several sites accepting writes for the same application;
correct behaviour under partition: no silent loss, and conflicts detected and resolved by a stated rule;
the per-request hot path stays local;
runs unprivileged in LXC, Debian-first, operable by a tiny team.

Candidates to evaluate:

object storage with multi-writer semantics: Garage, documented for 200 ms or less, not yet in Debian; Ceph RGW multisite, which is heavy;
per-application designs: write-once uploads with unique names, which are naturally conflict-free and could be multi-writer even over Syncthing if each site writes only its own subtree, with every site receiving all;
the database side: multi-primary is unrealistic at 250 ms, so a multi-writer file layer only helps once the application's writes are also multi-site.

Done when: a decision note records the multi-writer path, with numbers measured at 250 ms, for at least WordPress uploads and Nextcloud data. Related: decisions 0032, 0049; the storage due diligence of 2026-10-03.

Maintainer, 2026-10-03: 'um escritor por vez não escala'. This is an open problem to solve, not an accepted limit. **Where we are.** The due diligence and the 250 ms bench (in progress) point to Syncthing for Keel Web's application files. It is the cheapest choice to run and to leave (plain files), and it works over poor networks (local reads, async copy). But it is single-writer per application: only the primary writes; the others receive. Active/passive is fine for a WordPress or a Nextcloud whose database is also single-primary, but it does not scale out writes, and it does not give active-active sites. **What a solution needs** (the Keel premise: 250 ms RTT, jitter, loss, partitions): - several sites accepting writes for the same application; - correct behaviour under partition: no silent loss, and conflicts detected and resolved by a stated rule; - the per-request hot path stays local; - runs unprivileged in LXC, Debian-first, operable by a tiny team. **Candidates to evaluate:** - object storage with multi-writer semantics: Garage, documented for 200 ms or less, not yet in Debian; Ceph RGW multisite, which is heavy; - per-application designs: write-once uploads with unique names, which are naturally conflict-free and could be multi-writer even over Syncthing if each site writes only its own subtree, with every site receiving all; - the database side: multi-primary is unrealistic at 250 ms, so a multi-writer file layer only helps once the application's writes are also multi-site. **Done when:** a decision note records the multi-writer path, with numbers measured at 250 ms, for at least WordPress uploads and Nextcloud data. Related: decisions 0032, 0049; the storage due diligence of 2026-10-03.