0040 (ADR-017): Monitoring per machine with Monit, aggregated per site in etcd #45

New issue

Open

opened 2026-09-30 10:11:04 -03:00 by marcos-mendez · 0 comments

marcos-mendez commented

2026-09-30 10:11:04 -03:00

(Migrated from github.com)

Decision (maintainer, 2026-09-30). Monit stays per machine and restarts what it can. An agent in the etcd overlay writes a compact Monit summary to the node's registry key; site aggregation is a view over etcd. Prometheus optional (the same agent can expose it). Machine alerting in Monit; mesh alerting via etcd watches. Monit checks are generated from the composition manifest.

Handbook: docs/decisions/0040-monitoring-with-monit-and-etcd.md (handbook#26)

Amends: 0021 ("never restarts a service" becomes "restarts what it can"; peer watching over Monit's HTTP interface becomes etcd watches; checks also come from the manifest).

Review notes (resolved by the maintainer, 2026-09-30: Monit restarts what it can; 0021 superseded on that point)

Implementation detail: which services Monit may restart, and how many times before it stops and alerts, should be per service in the manifest.

First tasks

Summary format for the registry key (0025).
Monit check generation from a manifest (after 0034's schema).

**Decision (maintainer, 2026-09-30).** Monit stays per machine and restarts what it can. An agent in the etcd overlay writes a compact Monit summary to the node's registry key; site aggregation is a view over etcd. Prometheus optional (the same agent can expose it). Machine alerting in Monit; mesh alerting via etcd watches. Monit checks are generated from the composition manifest. **Handbook:** `docs/decisions/0040-monitoring-with-monit-and-etcd.md` (handbook#26) **Amends:** 0021 ("never restarts a service" becomes "restarts what it can"; peer watching over Monit's HTTP interface becomes etcd watches; checks also come from the manifest). **Review notes** (resolved by the maintainer, 2026-09-30: Monit restarts what it can; 0021 superseded on that point) - [x] Implementation detail: which services Monit may restart, and how many times before it stops and alerts, should be per service in the manifest. **First tasks** - [ ] Summary format for the registry key (0025). - [ ] Monit check generation from a manifest (after 0034's schema).