WHAT IT IS
A pipeline check for problems the forecast call finds too late.
pipeline-hygiene is a read-only inspector for a sales pipeline. Its spec, the design document written first, says managers find hygiene failures, such as deals left stale or with vague next steps, at the forecast call, the meeting where a team commits to its numbers and it is too late to fix them. It reads CSV exports from a CRM, where sellers log deals, keeps each as a dated snapshot in SQLite, a single-file database, runs eleven rules that flag those failures, and scores each deal and owner. It writes a dated report called the desk brief in Markdown, a plain-text format, and serves an offline dashboard.

WHAT CHANGED
The dashboard rewrite was checked against frozen outputs.
The first dashboard ran on Streamlit, a Python library that turns a script into a web app, until the migration plan recorded that after three iterations its styling could go no further. The plan split the move to FastHTML, a Python framework that serves plain HTML, into a port in which nothing on screen should change and a later visual redesign. Golden files, saved copies of what the Streamlit dashboard displayed, were frozen from the live app first. A pure view model, code that prepares every figure and chart without importing either framework, was extracted next, and the port was checked against the golden files before Streamlit was retired. The redesign then regenerated the golden files on purpose.
CONTROL
The tests check the engine against an answer key it did not write.
The org simulator builds a synthetic sales org and plants known problems in it, writing each one into a manifest, a list of expected findings, without ever running the rules. The manifest test then runs the full engine over that org and compares what it finds with that list. The generator and the engine are separate code written from the same spec, so they cannot share a coding mistake, although both could still misread the spec the same way. CI, the test run GitHub performs automatically, runs this test with the rest of the suite.
LESSONS
Two patterns from this build.
- Pass the date into every rule instead of letting the code read the clock. Each of the eleven rules takes the date it judges against as an input called as_of, so the same snapshots and the same date produce the same desk brief, and tests/test_brief.py checks that byte for byte against a stored golden file of the brief.
- Present owner scores as a coaching signal and keep worst-first rankings out of the output. Owner lists sort alphabetically, a score appears only once an owner has enough deals to meet a minimum sample size, and every score carries a label saying it is a coaching signal that should not feed compensation decisions, which two test files assert.
PROJECT DESIGN
How it works.
Key modules
Ingest
CSV validation and ingestion: enforces required columns, config-driven stage_map resolution (Dynamics and HubSpot presets ship in config.yaml), forecast enum check, date parsing, duplicate opp_id rejection, and single-currency-per-file invariant. Fatal on missing columns, mixed currency, or all-rows-rejected. Per-snapshot ValidationReport is persisted to the snapshots table so the brief and dashboard surface it without re-validating.
Snapshot store
SQLite snapshot store with three tables: snapshots, opportunities, runs. Derives close_date_changes and stage_entered_date from consecutive stored snapshots when source columns are absent — enabling H3 (serial slippage), H6 (stage aging), and H11 (push analytics) from data no single CRM export can provide.
Rules engine
Eleven pure functions H1–H11, each (row, config, as_of) → Violation | None | InsufficientHistory. By written convention the engine does not call date.today(), and all functions take an explicit as_of date so the same snapshots at the same as_of always produce the same output, enabling byte-for-byte golden-file comparison and a defensible audit trail.
Scoring
Opportunity, owner, and desk scoring with one required_coverage_multiple source of truth. Coverage = open pipeline / (remaining quota × required multiple); low_coverage fires exactly when the ratio falls below 1.00x, so the shown ratio and the flag cannot contradict each other. The basis string carries the exact fraction to prevent rounding errors from corrupting napkin-check math.
Desk brief
Markdown desk brief with since-last-run diff derived from the runs table. Per-owner coaching digests (--digests) write one private file per owner with only that owner's data. Pre-forecast-call commit-scrub sheet (--commit-scrub) is generated separately. Filtered and scrub runs are never recorded, preventing them from corrupting deal-streak history.
FastHTML dashboard
Seven-tab read-only dashboard built on python-fasthtml 0.14.11 with Altair/Vega-Lite charts and inline SVG sparklines. A pure view model (pipeline_hygiene_view.py, 954 LOC) owns all formatting and chart specs with no UI framework import — the parity boundary that made the Streamlit-to-FastHTML migration a renderer swap, not a rewrite. Binds 127.0.0.1 by default; container mode requires an explicit env var.
Org simulator
Generates a synthetic 3-level org (60 sellers, 4 personas: clean operator, sandbagger, happy-ears, ghost) with 8+ pathology injectors and a ground-truth manifest. The manifest integration test compares engine output against manifest fields built field-by-field by the generator — never by running the engine — so the oracle is genuinely non-circular.
SECURITY & OPERATIONS
Security and operations.
- Synthetic data only: every name, account, and opportunity in the repo — including the four weekly CSV snapshots committed to data/ — was generated by the org simulator. A grep-based verification checklist in README.md confirms no real CRM data reached the repo.
- Upload session isolation: CSVs uploaded via the dashboard are parsed in memory and never written to the snapshot store; upload session expiry was hardened at commit a4556ab (2026-08-12).
- Per-owner coaching digests are private by design: --digests writes one file per owner containing only that owner's data, isolated to out/digests/<as_of>/<owner_slug>.md. Published rankings were explicitly rejected to avoid attrition risk.
- The Azure deploy guide documents Entra Easy Auth (Microsoft sign-in) as an optional step before real data is loaded, and says it is untested against a live subscription; the demo mode serves synthetic data and is open by default. No API keys or secrets are committed — no LLM layer has been built.
BUILDER NOTES
Builder notes.
- The parity gate between Streamlit and FastHTML lives in tests/parity/test_pipeline_hygiene_parity.py, where four tests compare the view model's text, metrics, tables and charts with the golden files and a fifth asserts that the allowlist of accepted differences is empty, so any difference fails the suite.
- Using a rounded required multiple instead of the exact fraction put a persona simulation's quick hand check of the coverage math off by roughly $285K, and a later commit fixed it. Required coverage multiple is now a single source of truth in scoring.py, and the basis string carries the exact fraction so the shown ratio and the low_coverage flag are derived from identical arithmetic.
WRITING
Field notes from this work.
No public field notes for this project yet.
Full build narrative
Read-only sales pipeline inspection agent: ingests CRM CSV exports into a SQLite snapshot store, runs 11 deterministic hygiene rules (H1–H11) — stale opportunities, slipped close dates, vague next steps, forecast-category mismatches — and scores every opportunity, owner, and desk. Outputs a dated Markdown desk brief with since-last-run diffs, per-owner coaching digests, a pre-forecast-call commit-scrub sheet, and a seven-tab FastHTML dashboard. All JS and CSS are vendored; the dashboard runs fully offline with zero CDN dependencies. 138 commits · 37 PRs · 5 active days.
Spec-first: SPEC.md defined the 11 rules, the scoring model, and the data contract before any engine code. The stack — Python, SQLite, FastHTML — was validated via a zero-CDN offline spike before committing to it (headless network capture, zero external requests confirmed). The build ran as numbered task commits: ingest and the snapshot store, the rules engine and scoring, the desk brief, a Streamlit dashboard — then Streamlit was retired and replaced via a parity-gated FastHTML migration. The org simulator (60 synthetic sellers, 4 seller personas) generated ground-truth data for the manifest integration test that checks the engine against an independent oracle rather than against itself.