Back to OneCue

The measured runs.
Precision, silence, and one honest miss.

Two benchmarks, deterministically graded. One asks whether a cue changes what the agent writes. The other asks whether recall stays quiet when nothing applies. Every number here is reproduced from the repository, not illustrative.

Finding 1 · violation prevention

A relevant cue changes what the agent writes.

Twelve tasks whose governing decision is deliberately absent from the code — a “why” only project history knows. Seven of them contradict what the model does by default; those are the ones that can discriminate. Baseline versus OneCue, three trials each.

Decision (hidden from the code)BaselineOneCue
now arrives in seconds, not ms0 / 33 / 3
claim the id before applying, never release on error0 / 33 / 3
free workspace capped at 3 seats0 / 33 / 3
empty cart returns null, not 00 / 33 / 3
discount rate is basis points, not percent0 / 33 / 3
a withdrawal past zero is refused0 / 33 / 3
round half to even (banker's rounding)0 / 30 / 3

Pass counts across three trials. Grading is deterministic Node assertions, not an LLM verdict, so a violation is objective.

  • Cold agent violated the decision on every discriminating run (21 / 21). With OneCue it violated 3 / 21 — all three the same task.
  • The one miss is round half to even: the cue fired, the model still used Math.round. A cue that fires is not a guarantee. We keep this in the set on purpose.
  • On the five tasks where the model’s default already matched the decision, the cue caused no regression (still 3 / 3). It helps when it matters, and does not hurt when it does not.

Finding 2 · precision and silence

When nothing applies, it says nothing.

One store seeded with eight distinct decisions. Eight prompts that each should surface a specific one, and eight with no relevant decision stored. Recall is local retrieval — no model call — so this runs free and fast.

8 / 8100%Recall — fired when a decision applied
7 / 888%Precision@1 — fired the right decision
8 / 8100%Silence — stayed quiet when nothing fit
0 / 80%False fires on unrelated prompts

The headline is the last row: zero false fires on eight unrelated prompts. Silence is the half of the pitch the first benchmark cannot show — there every task plants a matching decision, so the cue always fires.

The one precision miss: “how many cents should we refund” surfaced the integer-cents decision instead of refunds — an honest near-overlap (both are about cents), not a random mismatch.

Method & honest caveats

What this does and does not prove.

  • Small n. Seven discriminating tasks × three trials (21 runs); an 8 + 8 recall fixture. Read the precision figures as “measured on an 8 + 8 fixture,” not a universal rate.
  • Not a clean sweep. OneCue prevents 6 of 7, not 7 of 7. We do not round that up to 100%.
  • Single model, subscription default, default temperature. Deterministic grading throughout.
  • Two arms — baseline versus OneCue. These fixtures hide the decision from the code by design, so a code-graph tool has nothing to index here; a true multi-tool comparison needs a real-repo fixture.

Reproduce

# Finding 1 — violation prevention (spends on claude -p; cached per run)
node experiments/graft-evaluation/run-decisions.mjs \
  --arms baseline,onecue --trials 3

# Finding 2 — precision and silence (local recall, free)
node experiments/graft-evaluation/run-precision.mjs

Run the benchmark yourself, then capture your first decision.

Install OneCue