2026 · ENERGY SALES INTEL

GridSignals

A published, configurable public-record pipeline that preserves evidence tiers rather than fabricating certainty.

PUBLISHED Built the ingestion, resolution, scoring, and offline operator surface.
On this page

WHAT IT IS

Public-record events turned into scored sales signals.

GridSignals watches 172 US energy companies through the public record and turns each event it finds into a scored, sourced signal card that names the Microsoft security products the event points to. Its sources include SEC filings, Federal Register notices, enforcement dockets from NERC (the body that sets reliability and security standards for the North American power grid), ransomware trackers that log attackers' leak-site posts, and security press. It is built for one operator, with no login. The backend uses only the Python standard library, the modules that ship with Python itself, and the web UI is the one part allowed third-party packages.

GridSignals screenshot

WHAT CHANGED

The account tier measured empty and was put behind a gate.

Cards have two scopes: account cards about one watchlist company, such as its own 8-K Item 1.05 filing (the SEC form for disclosing a material cybersecurity incident), and sector or regulatory cards, such as a peer company's incident or a revised NERC grid-security standard. Over 12 months the account tier came up empty: EDGAR, the SEC's filing database, held no Item 1.05 among 1,886 submissions, and none of 45 executive-change filings named a security executive. An August 15 commit repositioned the README around sector and regulatory scope and gated the account tier, and the README lists four conditions, set in code, that would open it. On August 21 the feed held 7 sector and regulatory cards and no account cards.

CONTROL

Unconfirmed incident cards stay with the operator.

Each incident source carries an evidence tier, meaning how firmly that source confirms what it reports, and the lowest tier, unconfirmed early warning, covers the ransomware trackers and press stories that repeat attacker claims. A card at that tier is shown only to the operator, and outreach on it, meaning any suggested sales contact with the company, is suppressed. The card itself also carries a label saying that no company, regulator or SEC confirmation exists.

LESSONS

Two patterns from this build.

  • Keep the list of allowed exceptions inside a test that fails in both directions. Here an exception is an import crossing between the web UI and the backend. Both of its lists are empty today, a new crossing fails the test, and a listed exception that no longer exists fails it too, so the list never claims more exceptions than the code has and adding one means editing the test.
  • Test a shutdown function against a real process tree. The container's scheduler, cron, starts each pipeline job in its own session, which the group-wide stop signal never reaches, so the entrypoint carries a function that signals every job directly. The packaging test cuts relay_sigterm_to_cron_descendants out of deploy/entrypoint.sh by its text, spawns a throwaway tree of processes, sends the stop signal and checks every descendant has exited, with no Docker needed.

PROJECT DESIGN

How it works.

Stdlib-only Python pipeline → SQLite → FastAPI + HTMX UI → Docker → Azure App Service
Ingestion
Python stdlib only (urllib, xml.etree, sqlite3) — no third-party packagesSource fetchers: SEC EDGAR, Federal Register, NERC, CISA KEV, NVD CVE, ransomware.live, security-press RSS, EIA plant capacity, GLEIF/Wikidata (enrichment), USAspending, EPA ECHO, PHMSA, GDELT (stored, classifier unwired)Single-writer ingestion lock; idempotent native-ID/content-hash deduplication
Data
SQLite in WAL mode, FK enforcement, 5-second busy timeout14 versioned, checksummed append-only migrations (0001_initial through 0014_combo_scoring)CSV seeds: 16 products, 21 triggers, 62 trigger-product pairs, 172 watchlist entities, scoring weights, 2 combo rules
Processing
Entity resolver: deterministic CIK/ticker/LEI match → fuzzy fallback (0.90 auto / 0.75 review); 25-case adversarial test set9 classifiers with evidence-tier assignment and outreach gatingDecay scoring: base × 0.5^(age_days / half_life) × account_fit × scope_fit [× combo_multiplier]Hand-written logic_expr combo grammar parser (no eval/exec); 2 seeded rules
UI
FastAPI 0.136.1 + Uvicorn 0.46.0 + Jinja2 3.1.6 + HTMX (vendored)Tailwind CSS (standalone CLI, compiled to committed static file)9 pages; light/dark/system theme; keyset-paginated signal cards with stable permalinks
Deploy
Docker (python:3.12-slim base, tini as PID 1)Azure App Service via one-command PowerShell deploy script (deploy/azure-deploy.ps1)cron daemon for scheduled pipeline refresh; SIGTERM relay to cron-spawned descendants
Accuracy audit
Claude Haiku 4.5 judge over urllib — no Anthropic SDK$0.50/run budget ceiling; $0.016 measured on a 5-signal runGolden-set regression gate; versioned verdict storage per run

How data moves

Click a box to read about it; stacked boxes zoom in. Use + and − to enlarge the drawing and drag to move around. Every flow was checked against the code.

Key modules

Data

Entity resolver

Deterministic CIK/ticker/LEI/alias match with fuzzy fallback (0.90 auto-accept / 0.75 review queue); collision guard for bare-name ambiguity; 25-case adversarial fixture set covering shared names ('Dominion', 'Constellation'), subsidiary/DBA overlap, CIK zero-padding, ticker case, and fuzzy typos.

Pipeline

Ingestion runner

Shared fetcher framework enforcing source policy, TTL gating, idempotent native-ID/content-hash deduplication, single-writer lock, and per-source error containment — one source failing cannot abort the run for others.

Pipeline

Classifier suite

Nine classifiers (leadership, regulatory, incident, ransomware, company statement, security RSS, environmental enforcement, capital project, PHMSA enforcement) that assign an evidence tier and gate outreach scope per trigger type.

Pipeline

Scoring engine

Decay formula (score = base × 0.5^(age/half_life) × account_fit × scope_fit × combo_multiplier) with five persisted score components; operator-tunable weights via Admin UI; decay threshold flips a card's status to 'decayed'.

Pipeline

Combo engine

Hand-written logic_expr grammar parser — never eval/exec — composing trigger_any, obligation:any, and not_keyword clauses with AND. Two seeded combo rules; score-inert until a first account-tier signal fires against a watchlist entity.

Accuracy

Audit judge

Claude Haiku 4.5 accuracy judge called over urllib (no Anthropic SDK), budget-capped at $0.50 per run, with a golden-set regression gate and versioned verdict storage. Skips cleanly with exit 0 when no API key is present.

UI

FastAPI + HTMX UI

Nine-page multi-page app (Signal Feed, Explore, Digest, Account 360, Review Queue, Feedback/Precision, Recent Re-tiers, Regulatory Monitor, Admin/Config) with light/dark/system theme toggle, keyset-paginated signal cards, and stable per-card permalinks.

SECURITY & OPERATIONS

Security and operations.

Evidence-tiered incident card pipeline — source to outreach gate
Public-record sources (EDGAR, NERC, ransomware.live, security press, CISA KEV, …)
Ingestion: source policy check → idempotent deduplication → entity resolution to 172 watchlist entities
Classification: default evidence tier set per classifier, and the operator can re-tier an incident card, logged on Recent Re-tiers — confirmed (8-K Item 1.05 / company statement) / corroborated (The Record) / unconfirmed early-warning (ransomware tracker / leak-adjacent press)
Outreach gate: confirmed → account-level plays enabled; corroborated → sector plays; unconfirmed → operator-only view, outreach suppressed, card labeled 'no company/regulator/SEC confirmation'
  • All data sources are public-record only — GET/RSS/JSON, no ToS-restricted scraping, no credential bypass. User-agent identifies the tool: GridSignals/0.1 (+https://github.com/Doogit/GridSignals).
  • Single-writer ingestion lock serializes all database writes and is released on SIGTERM via a two-pass /proc relay to cron-spawned descendants, so `docker stop` no longer forcibly ends an in-flight cron job before its lock cleanup runs.
  • Incident outreach is gated by evidence tier: confirmed (8-K Item 1.05 or company statement) enables account-level product plays; corroborated (security journalism) enables sector-level plays; unconfirmed early-warning (ransomware tracker) is operator-only, outreach suppressed, card labeled with the absence of confirmation.
  • Security Copilot plays are suppressed for known or likely US government cloud tenants.
  • PII guard enforced at commit time via .git/hooks/pre-commit and commit-msg hooks scanning staged content and commit messages against a pattern file.
  • Every Admin config edit is appended to a config_audit table — an immutable provenance trail of every weight or half-life change, with the config version stamped per scoring run (migration 0011) for reproducibility.

BUILDER NOTES

Builder notes.

  • The SIGTERM relay in deploy/entrypoint.sh required a two-pass /proc walk to signal every cron-spawned job descendant — tini's -g flag does not reach processes that cron places in separate sessions. The function is tested against a real three-level process tree in test_packaging.py:623–727.
  • Entity resolution uses a 25-case adversarial fixture set (tests/fixtures/adversarial_cases.csv) covering bare-name collisions ('Dominion', 'Chord', 'Range', 'Constellation'), CIK zero-padding, ticker case-insensitivity, fuzzy typos, suffix-strip near-twins, and an unknown-company case.
  • The Streamlit UI was replaced mid-build with FastAPI + HTMX + Tailwind; test_packaging.py pins Streamlit's absence at line ~332 as a packaging regression guard so the Dockerfile and entrypoint cannot quietly bring it back.

WRITING

Field notes from this work.

No public field notes for this project yet.

Full build narrative

GridSignals turns the public record around US energy companies into scored, sourced signal cards — each mapped to one of 16 Microsoft security products across 62 trigger-product pairs — and surfaces them through a nine-page FastAPI + HTMX web UI. The ingestion pipeline fetches from classified or stored public-record sources (SEC EDGAR submissions and full-text search, Federal Register, NERC enforcement dockets, CISA KEV, NVD CVE, ransomware.live, security-press RSS, EIA plant capacity, and more), resolves each item to one of 172 watchlist entities via deterministic CIK/ticker/LEI match with fuzzy fallback, classifies it into a trigger type, and applies a decay-adjusted score across five persisted components. The ingestion, scoring and audit pipeline runs on the Python standard library with no third-party packages (the web UI is the one exception), and the only recurring external cost is a budget-capped Claude Haiku 4.5 accuracy judge ($0.016 measured on a 5-signal run; $0.50 ceiling). 338 commits, 115 PRs, 14 versioned migrations, 84 hermetic test modules, in 10 active days.

The build runs config-as-data throughout: 16 products, 21 trigger types, 172 watchlist entities, scoring weights, and two combo rules all live as CSV seeds loaded into SQLite at startup, so the operator can tune the system through the Admin UI without a code deploy. A 1,224-line packaging test (`test_packaging.py`) covers the Docker, Azure, scheduler and SIGTERM packaging, and it pins Streamlit's absence as a regression guard after the UI was rebuilt mid-build on FastAPI + HTMX + Tailwind. The combo engine needed a composable rule grammar; the written stdlib-only, no-dynamic-code rule ruled out `eval`/`exec` and produced a hand-written `logic_expr` parser instead. Migrations are versioned and checksummed append-only; a mismatch on an already-applied file is a hard error, preventing silent schema drift across the 14-migration history. SIGTERM handling required a two-pass `/proc` walk to relay the signal to every cron-spawned job descendant, because tini's `-g` flag does not reach processes that cron places in their own sessions, and without the relay `docker stop` forcibly ends an in-flight cron job before its lock cleanup runs.

Trust belongs in the schema, not the application.
The operating principle behind every project here. A rule the database enforces can't be forgotten in a hurry. Constraints, denied-by-default access, append-only logs — the controls that hold are the ones the system won't run without.