Organizing your bench

Once you are running the grading loop more than a few times a week, the artifacts multiply: prompts, evidence pulls, drafts, scorecards, and the notes you took while reading them. This is the layout that keeps that navigable, and the two files that turn it from a folder of markdown into something you can actually answer questions from.

One idea carries all of it:

A run is the unit. One run = one prompt + one evidence set + the output + the scorecard. Finished runs are immutable.

Three shelves

Everything you produce belongs on exactly one of three shelves, and the shelf decides whether you are allowed to edit it later.

The frozen shelf is the one that matters. Evidence files carry a date and never get overwritten. Run folders carry a date and a time and do not get edited once done. The moment you edit a finished run, every comparison that referenced it becomes a comparison against something that no longer exists.

Layout

gtm-bench/
  CLAUDE.md                                 loads canon every session
  canon/                                    LIVING
    icp.md  product.md  brand-voice.md      your own static docs, by hand
  prompts/
    homepage-hero.md
  skills/
  evidence/                                 FROZEN
    2026-08-04-icp-pains.json               new pull = new file, no overwrites
  external/                                 FROZEN
    2026-08-04-competitor-x.md              scraped material, never graded
  runs/                                     FROZEN
    2026-08-04-1432-homepage-hero/
      manifest.json
      outputs/
        cold.md
        assisted.md
      scorecard.json
  logs/                                     ROLLING
    disagreements.md
    friction.md
  page-index.md
  coverage.md

The agent context file is what makes this a brain

Without an always-loaded context file, canon/ is three markdown files your agent reads when it happens to. With one, product truth, ICP and brand voice are in context on every session and every run, without you pasting them.

Keep it short. Point at the canon files, name the run convention, and state that finished runs are immutable. It is the highest-leverage file in the repository and it takes ten minutes.

The manifest

manifest.json records a run's inputs so you can reconstruct it later. The skill writes it, not you.

• Which prompt version and which evidence file
• The exact search query that produced that evidence, and the SQL it compiled to
• row_count and truncated from the search response
• The Eval run id and reused from the submit response
• The scope you sent, and the scope outcome that came back — whether it resolved or abstained, and on which of the seven reasons
• A hash of each canon file that was in context

Four of those earn their place:

The query line. Ask the corpus the wrong question and everything downstream is a well-written answer to something you did not mean. The verdict will not catch it, because it grades your output against the evidence you asked for. Recording the compiled SQL next to the question in English is what lets you see that the query writer understood the ask differently than you did.

truncated. It means the row set hit limit, so your evidence is a capped slice of what matched. A biased sample reads exactly like a clean one.

The scope line, both halves. This is the same class of failure one level up. A run that abstained to whole-corpus grading and a run that graded the slice you asked for are not comparable, and the score looks identical either way — so recording only what you sent leaves you unable to tell them apart later. Record the outcome, not just the request. See Scope the grade to a slice.

The canon hashes. Voice and product truth drift over six weeks. Without them, a run is reproducible on everything except the input most likely to explain a change.

coverage.md

One row per artifact you intend to grade, with the run that graded it. That is the whole file. Its job is answering "what have we not looked at yet" — the question that is impossible to answer three weeks in without it.

The disagreement log

Add a row whenever a verdict seems wrong, and tag which of three things it is. They look identical in the report and they are three different subsystems:

Filed as one undifferentiated pile, none of them get fixed.

The search response tells you which one you are looking at. On search.query these live on detail.internal rather than at the top level, since detail is where the natural-language lane's full envelope rides. A multi-part ask fans out, so check detail.groups when there is more than one sub-question — each carries its own internal.

friction.md

The other rolling log, and the one people skip. Every small annoyance, written down the moment it happens: the thing you had to copy-paste, the field you expected and did not find, the step you automated around rather than reporting.

Nobody remembers those a week later, and they are exactly what the next person hits on day one.

See also

• The grading loop — the method this layout serves
• Search — the detail.internal fields above, in full
• Evals — run handles, polling, and what a scorecard carries