The cheap model failed its own exam
CareerHub grades its AI scoring against a test set with known right answers, and the agent tuning the prompts is never allowed to see those answers. That's why I believed the result when the cheaper model failed the test twice, and moved the job to a bigger model.
- CareerHub runs a job search like an outbound sales pipeline: ingest postings, score them, find the right person, draft outreach. It's local-only, single-user, and private.
- Scoring is gated by an eval with fixed thresholds. When the cheaper model came in under the recall gate twice, triage moved to the bigger model and the decision went in the log.
- The labels behind that eval are written by a separate agent that never sees the prompts or the scores. That's why the failure was believable.
- Nothing sends without a human click, and there's no config flag that changes that. 1,352 commits in 18 days, built with agents, and the agents had to ask before touching either rule.
The cheap model failed the exam. Triage is the first scoring pass in CareerHub, the one that reads every posting that survives the hard filters and decides whether it deserves a closer look. Running it on a Haiku-class model (Anthropic's smallest, cheapest tier) at low effort came back under the recall gate. Effort went up to medium and the prompt got rewritten. Still under. So triage moved up to Sonnet, the reason went into a comment in models.ts, and the reversal got its own row in the decision log.
That's a boring outcome, and boring is the point. A failed eval only means something if the eval couldn't have been gamed, and most of what's interesting about this build is how it was set up so that it couldn't be.
This post stays at the pattern level. CareerHub is the tool I use for my actual job search, so there are no companies, postings, contacts, or outcomes here. Architecture, controls, and method only.
What it is
The repo's README states the idea in one line: "Run a job search like an outbound sales pipeline." Postings come in daily from public applicant-tracking-system (ATS) APIs and job aggregators through a dozen-plus source adapters. They get deduplicated, scored against my resumes and a search profile, matched to a likely recruiter or hiring manager, and turned into a draft of direct outreach. That's the design: I approve it, it goes out from Gmail, and follow-ups and replies land on a dashboard. The scoring, guardrail, sender and eval modules are built; I haven't checked every outreach feature in the spec against the code.
LinkedIn and the job aggregators give you a few filters and their own opaque guess at whether you're a fit. I wanted an LLM-first version, where model calls through my Claude subscription do the reading and judging. It doesn't scale at all. It probably burns around $300 a month in API-equivalent credits and scraper costs. But it wasn't built to scale. It was built to be the best tool I could make for finding a job fast. It also finds contacts for good-fit roles and tracks messages and replies on one dashboard, which beats the Google Doc I used to track applications in.
It's local-only by design. The API binds to 127.0.0.1, there's no login, and it's set up to run as a Windows scheduled task on my machine. The stack is a pnpm monorepo in strict TypeScript: Express and SQLite on the back, React and Vite on the front, Drizzle as the ORM (the layer that maps tables to typed code). The dependency rule is strict too: core is pure logic with no I/O, no Express, no React, no database, which is why most of the pipeline can be tested without touching any of them.
The exam, and the wall around it
Every scoring decision is graded against an eval: a labeled set of postings, a scorer, and three gates. Triage needs precision ≥ 0.85 (of the postings it passes, how many deserved to pass) and recall ≥ 0.80 (of the postings that deserved to pass, how many it caught). Deep fit, the slower second pass that ranks the top candidates, needs a Spearman correlation ≥ 0.6 (how closely its ranking agrees with the labeled ranking). The repo's own rules say that changing a scoring prompt means re-running pnpm eval. No re-run, no change.
Recall is the gate that matters most for triage, because of where triage sits. It's the cheap filter that runs on everything. A posting it wrongly drops never reaches deep fit, never gets a score, never shows up on the dashboard. Precision mistakes cost me a few seconds of reading. Recall mistakes are invisible. So when the cheaper model missed recall twice, a third prompt rewrite and a hopeful re-run weren't the answer. The bigger model was.
Those invisible misses matter because finding what I'd otherwise miss is the whole job. Before it searches anything, CareerHub reads my resumes to pull out search keywords, asks the model which roles I'm most likely to get hired into, and drafts a list of 20 to 30 target companies where I'd fit and where company and macroeconomic signals point to strong hiring. All of that becomes the search criteria. Then it pulls several hundred postings a day from company career sites directly, from aggregators, and from scrapers. Deterministic filters (fixed rules, no model involved) clear the obvious noise, and model calls rank what's left for priority, fit, and how hireable I'd be. Only the top of that list reaches the dashboard. It has surfaced dozens of roles I wouldn't have seen on LinkedIn. At the cybersecurity leadership level, where relevant postings are hard to find, that's a big difference. A triage pass that silently drops good postings would throw that away.
Here's the claim: a test only counts if the agent being graded can't see the answer key. Setting a pass mark is easy. An agent tuning prompts against labels (the answer key) it can read will eventually fit the labels instead of the task. No bad intent required. Plain optimization gets there on its own, and the result looks exactly like progress. So the decision log set a rule before any prompt work started: labels are written by an agent in a separate context from any agent that writes or tunes scoring prompts. The labeling agent never sees packages/core/src/prompts/, the stored scores, or any scorer output. It works from a written rubric, the labels are committed before prompt work begins, and changing a frozen label needs a logged reason. That rule has been exercised since. A later label change went through the same re-labeling protocol, reason and all.
The rest of the eval setup is just as careful about what the scorer can see. Every label records where it came from: me, the labeling agent, or an agent draft I accepted. The labeling tool, pnpm eval:label, is a 127.0.0.1-only page that shows the posting and nothing the scorer thought: no score, no bucket. It renders descriptions with textContent, never innerHTML, because posting text is third-party input and doesn't get to run as HTML.
My own thumbs-up and thumbs-down ratings get the same treatment. They're split by hash into buckets of three, only bucket 0 counts as test data, and the measurement notes say in so many words that figures from the tuning buckets are "not evidence." The measurement script refuses to run against the live database and works on a .backup() copy instead. And prompts are versioned like code: 20 prompt modules, each exporting a PROMPT_VERSION stamped on every score and draft row. Triage is on triage-v7, deep fit on deep-fit-v10. That's a count of rewrites. It says nothing about quality.
None of this is exotic. It's train/test separation, the oldest rule in machine learning, applied to agents instead of datasets. The twist with agents is that the one doing the tuning can read the whole repo. Putting the labels in a separate folder won't stop it. A separate session that never opens the prompts will.
Every model call gets checked and logged
Every model call goes through one LLMProvider interface, gets validated against a zod schema (a runtime type check that rejects malformed output), and writes a row to an llm_calls table with the model, tokens, cached tokens, and cost. If validation fails, it retries once with the error appended. If that fails, the item fails and the run keeps going.
Model choice lives in a table: 21 task keys in models.ts, each mapped to a model tier and an effort level and overridable from settings. Small jobs like ranking contacts run Haiku-class at low effort. Triage, deep fit, and drafting run Sonnet-class. By default the calls go through my Claude subscription via claude -p, with a notional cost recorded anyway, so I can see what each run would cost even though I pay a flat subscription.
Posting text is untrusted input, and the prompts treat it that way. It's wrapped in explicit delimiters as reference data, and neutralizeDelimiter strips anything in the posting that would close the delimiter early (the prompt-injection move: text that pretends the data has ended and the instructions have started). And the model sees as little of me as it can: the contact block comes off the resume before any call, and drafting gets a contact's first name, title, and company, nothing more.
Nothing sends without my click
The README lists four principles, and the first one is "Human approves every send." That's enforced in the code, not left to a setting. An outbound email needs a message row with status='approved', and that row only exists because I clicked Approve on that specific message. There's no API route that sends immediately, and the rules say it outright: "There is no config flag that bypasses Approve."
Even after I click Approve, the email doesn't go out right away. The decision belongs to a pure function in guardrails/evaluate.ts (pure meaning no I/O; the current time and time zone are passed in, so tests can pin them). It returns one of three answers, send, wait, or cancel, and it checks every cancel rule before any wait rule, so a message that should never go out can't sit in a queue waiting for its turn. The spec behind it sets a daily cap of 30, a five-a-day warm-up for the first week, three minutes between sends, quiet hours from 7 PM to 7 AM, approvals that expire after 72 hours, and a circuit breaker that pauses everything on mail-server auth errors or two bounces in the last 20. It also designs a claim check for drafts that flags any number, employer, or title in the body that isn't in my resume, so the model can't promote me in an email I'm about to sign.
There's also a short list of things it won't do at all. No LinkedIn automation: no cookies, no session tokens, no browser automation against the site. LinkedIn URLs get stored, and I open them myself.
Running locally doesn't make it safe from the web
A localhost API with an Approve button is still a web target, because your browser will talk to 127.0.0.1 for any page you visit. Two guards run ahead of every route. hostGuard rejects any request whose Host header isn't localhost, 127.0.0.1, or [::1], which blocks DNS rebinding (a malicious site points its own domain at your loopback address so your browser treats your local API as same-origin). csrfGuard requires a custom header and rejects a mismatched Origin on anything that changes state, which blocks cross-site request forgery (a page you visit secretly firing requests at another app as you). Secrets live in .env or Windows Credential Manager, get entered through a write-only settings route, and never touch SQLite; the settings endpoint only ever says whether a key is configured.
How it got built
CareerHub went from first commit to now in 18 calendar days: 1,352 commits across 15 active days, peaking at 203 on October 19. About 117,000 lines of TypeScript including tests, 31 tables, 15 migrations, and 3,400-plus test cases by static count (that's a grep of test declarations, not a pass rate).
The first three commits are all documents: the product spec, the decision log, and a review pass that applied 32 findings across security, outreach safety, dedupe, sequencing, and budget. Spec and review came before code. That habit stuck. 599 of the 1,352 commits are docs: plans, decisions, handoffs, notes. That ratio would alarm a code reviewer and delight an auditor. The decision log reached 68 entries.
The work ran as plan units. Each unit got its own branch in its own worktree, a code review from an agent in a separate context, and a local --no-ff merge, 232 of them. The definition of done was tests, typecheck, and lint passing and the behavior checked in the running app.
The agents worked under a written rule about when they had to stop and ask me. They were told to keep moving without asking, with five exceptions: sending outreach, new paid services, writes to the live database beyond an approved script, irreversible schema changes, and changing frozen eval labels. Read that list again next to the rest of this post. The code refuses to send an email without an Approve click, and the decision log freezes the answer key; those are the same two things the agents had to stop and ask about. The list of things I wouldn't let the agents do on their own and the list of things the app protects against ended up being the same list.
What none of it can tell me is whether the search works. The eval measures whether triage agrees with the labels; it doesn't measure whether the pipeline finds the right job, and that number lives outside the repo, where it belongs. The app checks its own work everywhere it can, and it still can't tell me the one thing I care about most: whether it finds me the right job.