Draft docs/benchmarks/report-v1.md head-to-head benchmarks skeleton #86

New issue

Closed

opened 2026-05-26 00:56:00 -03:00 by navigator · 2 comments

navigator commented

2026-05-26 00:56:00 -03:00

Owner

Goal

Create the head-to-head benchmark report skeleton at docs/benchmarks/report-v1.md, listed as deliverable #5 in PLAN.md Section 10. The doc structures Phase 1 simulation-based performance comparisons (baseline Gemmini vs single-chip FluidPopSoC vs 8-chip board) so benchmark work has a canonical landing spot.

Sections to include:

Overview (scope: simulation-based Phase 1 numbers only; no silicon results)
Methodology (Verilator sim clock ~10 kHz per Section 8.3; cycle accounting; what counts as 'one inference')
Baselines (upstream GemminiRocketConfig from Phase 2 — reference baseline-gemmini-rocket.md)
Single-chip FluidPopSoC results (placeholder table; expected 2.5-3.5x speedup per Section 8.3)
8-chip board results (placeholder table; expected 4-6x speedup over single chip per Section 8.3)
Workloads (ResNet-50 single-chip; Llama-7B 4-bit 8-chip partitioned per Section 8.3)
Caveats (sim-only; honest reporting per Section 9.3; analytic model vs measured per Section 14.6)
Open questions (which secondary metrics — power, BW utilization, scratchpad hit rate)

Acceptance criteria

docs/benchmarks/report-v1.md exists with the sections above
Status: Draft skeleton + Owner: TBD header
Each section has TODO markers and 2-3 line intent description
Placeholder tables present with header rows; all data cells TBD
Explicitly states sim-only scope and references Section 9.3 honest-reporting language
References Sections 7 (baseline), 8.3 (acceptance targets), 9.3 (honest reporting), 14.6 (analytic models)
docs/benchmarks/README.md created or updated to list this deliverable

Plan refs

Section 10 (deliverable #5), Section 7 (baseline benchmark), Section 8.3 (testbench plan + acceptance criteria), Section 9.3 (honest reporting), Section 14.6 (analytic performance models)

Notes

Skeleton only — actual numbers come from Phases 2-3 simulation runs. Avoid any fabricated benchmark numbers; tables stay empty until measured.

## Goal Create the head-to-head benchmark report skeleton at `docs/benchmarks/report-v1.md`, listed as deliverable #5 in PLAN.md Section 10. The doc structures Phase 1 simulation-based performance comparisons (baseline Gemmini vs single-chip FluidPopSoC vs 8-chip board) so benchmark work has a canonical landing spot. Sections to include: - Overview (scope: simulation-based Phase 1 numbers only; no silicon results) - Methodology (Verilator sim clock ~10 kHz per Section 8.3; cycle accounting; what counts as 'one inference') - Baselines (upstream `GemminiRocketConfig` from Phase 2 — reference `baseline-gemmini-rocket.md`) - Single-chip FluidPopSoC results (placeholder table; expected 2.5-3.5x speedup per Section 8.3) - 8-chip board results (placeholder table; expected 4-6x speedup over single chip per Section 8.3) - Workloads (ResNet-50 single-chip; Llama-7B 4-bit 8-chip partitioned per Section 8.3) - Caveats (sim-only; honest reporting per Section 9.3; analytic model vs measured per Section 14.6) - Open questions (which secondary metrics — power, BW utilization, scratchpad hit rate) ## Acceptance criteria - [ ] `docs/benchmarks/report-v1.md` exists with the sections above - [ ] `Status: Draft skeleton` + `Owner: TBD` header - [ ] Each section has `TODO` markers and 2-3 line intent description - [ ] Placeholder tables present with header rows; all data cells TBD - [ ] Explicitly states sim-only scope and references Section 9.3 honest-reporting language - [ ] References Sections 7 (baseline), 8.3 (acceptance targets), 9.3 (honest reporting), 14.6 (analytic models) - [ ] `docs/benchmarks/README.md` created or updated to list this deliverable ## Plan refs Section 10 (deliverable #5), Section 7 (baseline benchmark), Section 8.3 (testbench plan + acceptance criteria), Section 9.3 (honest reporting), Section 14.6 (analytic performance models) ## Notes Skeleton only — actual numbers come from Phases 2-3 simulation runs. Avoid any fabricated benchmark numbers; tables stay empty until measured.