CASE STUDY

The triage step failed its eval on both models in one day

OCTOBER 29, 2026 · 6 MIN · CAREERHUB
TL;DR

CareerHub grades its AI triage step against frozen labels that the prompt-tuning agent never sees, and on October 14 that test failed triage on both models it was tried on.

  • CareerHub is a private, single-user job-search pipeline, so this post covers only its architecture and method.
  • Triage failed its gates on the cheaper model, did worse on recall on the larger one, and was merged that day with the gates still open.
  • Sending mail is enforced by code and tests, while the label freeze and the model choice rest on written rules only.

Triage is the first scoring pass in CareerHub, the job-search pipeline I built for myself, and it reads every posting that survives the hard filters (fixed rules that run before any model is involved) to decide which ones deserve a closer look. On October 14 triage ran against its eval for the first time and did not pass on either of the two models it was tried on that day. Since the data is my own job search, this post covers the architecture and the method and leaves out companies, postings, contacts, outcomes and scores.

What CareerHub is

The README describes it in one line: "Run a job search like an outbound sales pipeline." Postings come in from public applicant-tracking-system APIs (the hiring software companies use to publish jobs) and from job aggregators, and they get deduplicated, scored against a set of resumes and a search profile, matched to a likely recruiter or hiring manager, and turned into a draft of direct outreach. The app runs only on my machine, with the API listening on 127.0.0.1 (the loopback address, which nothing else on the network can reach), no login, and a Windows scheduled task to keep it running. It is TypeScript with Express, SQLite and React, and as of October 29 the repo had 1,352 commits over 18 calendar days, built under a written process where agents took one planned unit at a time, each reviewed by an agent in a separate context before a local merge.

How the eval grades triage

Every scoring change is graded by an eval, which here means a set of postings with agreed labels, a scorer that runs the current prompt against them, and fixed gates the result has to clear. Triage needs precision of at least 0.85 (of the postings it passes, the share that deserved to pass) and recall of at least 0.80 (of the postings that deserved to pass, the share it caught). Labels for 120 postings were written by an agent working from a rubric and committed at 08:30 on October 14, and the first triage code landed at 09:01, so the answers were fixed before any prompt existed. After that freeze, a label can only change with a reason recorded in the decision log, the dated list of design decisions kept in the repo.

Keeping the labels away from the prompt-tuning agent

Before any prompt work, the decision log set a rule that labels are written in a separate agent context from the agent that writes and tunes the scoring prompts. The labeling agent does not see the prompts, and the tuning agent never sees the label file or the answer for any single posting. The tuning side did receive something, though: a label-side agent compared the scorer's results with the labels and wrote aggregate error analyses, such as which kinds of postings the prompt was wrongly rejecting, and those analyses guided the next rewrite. This separation is a written rule carried out by running the agents in separate contexts, and no test or automated check verifies it.

How triage did on October 14

The first live run failed because the eval harness was wrong: it sent every posting to triage without applying the hard filters first. That was found by reading the harness code and fixed by 10:00. The next run, on Haiku (Anthropic's smallest and cheapest model tier) at low effort, missed the recall gate, where effort is a setting for how much reasoning the model does before it answers. An error analysis from the label side said the prompt was rejecting a large share of relevant postings, so at 10:20 the prompt was rewritten as an inclusive screen, which passes a posting along when in doubt, and effort went up to medium. That run passed recall and failed precision. At 10:50 a fixed compensation floor check was added ahead of the model and the prompt was rewritten again to fix precision, and that version, still on Haiku at medium, failed both gates.

At 14:24 triage moved to Anthropic's larger Sonnet tier at medium effort. The same prompt did worse on recall there, because Sonnet applied the prompt's hard-fail rules (conditions that reject a posting outright) more strictly. The work was merged that day with the gates still open and a follow-up track for further fixes. The switch to Sonnet is recorded in a comment in the file that maps each task to a model, and in a decision-log entry written 13 days after the code change.

What is enforced in code and what rests on written rules

The README's first principle is "Human approves every send," and the code holds to it. The approve route marks a message approved only if it is still a draft and refuses anything else. The sender loop only picks up approved messages, and no route sends mail directly. When a draft is approved, a separate set of tested checks decides when it may go out, with limits that include 30 sends a day, quiet hours from 7 PM to 7 AM, approvals that expire after 72 hours, and a circuit breaker that stops sending after two bounces in the last 20 sends. A test also fails if any file outside a short allowlist imports the mail library.

The rules around the eval have no such backing. The label freeze, the rule that any prompt change means re-running the eval, and the choice of Sonnet for triage all live in writing only. There is no CI (checks that run automatically on every change), the repo's only hook (a script that runs automatically when the coding agent takes certain actions) limits how many review-fix rounds a unit gets, and no test fails if someone sets triage back to Haiku.

A test that allows only one file to send mail

This is a generalised version of the import-allowlist test behind the mail rule. It walks the source tree and fails if any file outside the allowlist imports the mail library, and it also fails if an allowlisted file stops importing it, so the list cannot go stale without someone noticing.

// mail-imports.test.mjs: run from the project root with: node --test
import { readdirSync, readFileSync } from "node:fs";
import { join, sep } from "node:path";
import { test } from "node:test";
import assert from "node:assert/strict";

const SRC = "src";
const ALLOWED = new Set(["src/mail/send.ts"]); // the one file allowed to send mail
const IMPORTS_MAILER = /from\s+["']nodemailer["']|require\(\s*["']nodemailer["']\s*\)/;

function walk(dir) {
  return readdirSync(dir, { withFileTypes: true }).flatMap((entry) => {
    const path = join(dir, entry.name);
    if (entry.isDirectory()) return entry.name === "node_modules" ? [] : walk(path);
    if (!/\.(ts|tsx|js|mjs|cjs)$/.test(entry.name)) return [];
    return [path.split(sep).join("/")]; // normalise Windows separators to match ALLOWED
  });
}

test("only allowlisted files import the mail library", () => {
  const offenders = walk(SRC).filter(
    (file) => !ALLOWED.has(file) && IMPORTS_MAILER.test(readFileSync(file, "utf8")),
  );
  assert.deepEqual(offenders, []);

  for (const file of ALLOWED) {
    assert.ok(IMPORTS_MAILER.test(readFileSync(file, "utf8")), `${file} no longer imports the mail library`);
  }
});

It uses only Node's built-in test runner, so it needs nothing installed beyond Node itself. I would use it for any capability that should have exactly one entry point in a codebase, such as sending mail or calling a payment API, since the test fails the moment a new file imports the library, which is easier to rely on than remembering to look for that import in every review.

If you build an eval for your own prompts, I would recommend committing the labels before writing the first prompt and writing down exactly what the tuning side is allowed to see, so that the answers are fixed before anyone has a reason to adjust them.

Trust belongs in the schema, not the application.
The operating principle behind every project here. A rule the database enforces can't be forgotten in a hurry. Constraints, denied-by-default access, append-only logs — the controls that hold are the ones the system won't run without.