feat(0.27.0): real semantic clustering — local sovereign model replaces the FNV stub #9

Merged

navigator merged 2 commits from feat/consensus-real-embeddings into main

2026-07-09 23:31:53 -03:00

navigator commented

2026-07-09 23:14:37 -03:00

Owner

Item #1 of docs/PLANO-ESTRATEGICO-2026.md — the highest-leverage change in the repo: the consensus signal becomes genuinely semantic.

candle-based ModelEmbedder behind the model-embedder feature; stub remains the default and the fallback.
Calibrated on pt-BR civic pairs (threshold 0.10; calibration test prints the matrix). Known limitations measured and documented, V2 tracked in a follow-up issue.
Deployment: model artifacts via hostPath on the VM, pod memory 1536Mi.

CI note: the workspace now compiles candle under --all-features; the model semantics test self-skips when CONSENSUS_MODEL_DIR is unset (no 470MB artifact in CI).

Item #1 of docs/PLANO-ESTRATEGICO-2026.md — the highest-leverage change in the repo: the consensus signal becomes genuinely semantic. - candle-based ModelEmbedder behind the `model-embedder` feature; stub remains the default and the fallback. - Calibrated on pt-BR civic pairs (threshold 0.10; calibration test prints the matrix). Known limitations measured and documented, V2 tracked in a follow-up issue. - Deployment: model artifacts via hostPath on the VM, pod memory 1536Mi. CI note: the workspace now compiles candle under --all-features; the model semantics test self-skips when CONSENSUS_MODEL_DIR is unset (no 470MB artifact in CI).

navigator added 1 commit

2026-07-09 23:14:37 -03:00

feat(0.27.0-embeddings): real semantic clustering — local model replaces the FNV stub

ci / fmt (pull_request) Successful in 5s

Details

ci / boundaries (pull_request) Successful in 13s

Details

ci / clippy (pull_request) Successful in 3m2s

Details

ci / test (pull_request) Successful in 6m1s

Details

security / deny (pull_request) Failing after 7s

Details

security / audit (pull_request) Successful in 8s

Details

security / secrets (pull_request) Successful in 8s

Details

ci / coverage (pull_request) Successful in 4m33s

Details

64adb31abe

The thesis said "pgvector clusters proposals into one real signal"; the
embedder underneath was hashed bag-of-words, so paraphrases never merged.
This slice makes the signal real:

- ModelEmbedder (consensus, feature model-embedder): BERT-family
  sentence embeddings via candle — pure Rust, CPU, no network at
  inference (PLAN.md principle 11). Mean pooling over the attention
  mask + L2 norm, so pgvector <=> keeps its scale. Reference model:
  intfloat/multilingual-e5-small (384 dims = vector(384) of 0130).
- Runtime selection: CONSENSUS_EMBEDDER=stub|model (default stub),
  CONSENSUS_MODEL_DIR, CONSENSUS_QUERY_PREFIX (E5: "query: "),
  CONSENSUS_THRESHOLD. Model loads once per process (OnceLock); load or
  inference failure degrades loudly to the stub — the consequence loop
  never wedges.
- Threshold calibrated on real pt-BR civic pairs: paraphrases measure
  0.079-0.089, distinct asks >= 0.134 → default 0.10. Two measured
  limitations documented in the calibration test (not gated): domain
  acronyms (UBS ~ posto de saude, 0.111) miss the merge; same-street-
  different-intervention pairs (0.078) can false-merge. Both tracked
  for the V2 model upgrade.
- Deploy: artifacts on the VM at /opt/dsoc/model (hostPath -> /srv/model
  — the VM cannot reach HF over IPv6-only egress); pod memory limit
  256Mi -> 1536Mi; gateway builds with the feature ON, activation stays
  behind env.

Evaluated and rejected: paraphrase-multilingual-MiniLM-L12-v2 — wider
distance spread but misses UBS=posto entirely (0.69) and still ranks
the same-street pair as a near-duplicate (0.32).

navigator added 1 commit

2026-07-09 23:23:42 -03:00

chore(deny): ignore RUSTSEC-2024-0436 (paste, unmaintained compile-time macro via candle)

ci / fmt (pull_request) Successful in 10s

Details

ci / boundaries (pull_request) Successful in 13s

Details

ci / clippy (pull_request) Successful in 2m59s

Details

ci / test (pull_request) Successful in 5m43s

Details

security / deny (pull_request) Successful in 14s