Evals (coming soon)

POST /evals scores an output set against ground truth: hand it a target — a lookalike match list, a routed-search result set, a Chat answer's claims, or any list of items you assemble — plus the truth source to judge it by, and it returns an overall score, per-criterion scores, and a per-item verdict with the evidence behind each one. It is the measurement half of the endpoint surface: the other verbs produce answers; evals tells you how good those answers were.

Reach for it when a workflow's output feeds a decision you want audited — "did the expansion list we generated last quarter actually convert?", "are this agent's stated figures backed by the warehouse?" — and when you are tuning a prompt or a filter slice and need a number to move.

Request shape

Three target kinds:

• { "kind": "items", "items": [...] } — a list you assembled yourself, each { "id", "predicted", "context"? }.
• { "kind": "lookalike_matches", "entity_type": "company", "matches": [...] } — a lookalike.find result, scored by whether each match went on to convert.
• { "kind": "chat_answer", "chat_id": "…", "run_id": "…" } — a completed Chat run; every data-backed claim in the answer is re-checked against the query snapshots and the live warehouse.

Example — score an expansion list against CRM outcomes

You ran lookalike.find a quarter ago and worked the list. Did the similarity ranking predict anything?

curl -X POST "https://app.amdahl.co/api/platform/v1/evals" \
  -H "X-API-Key: $AMDAHL_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target": {
      "kind": "lookalike_matches",
      "entity_type": "company",
      "matches": [
        { "entity_id": "9b03…", "score": 0.91 },
        { "entity_id": "77aa…", "score": 0.87 }
      ]
    },
    "ground_truth": { "kind": "crm_outcomes", "window_days": 90 },
    "criteria": ["accuracy", "ranking"]
  }'

Over MCP, the evals tool's run action takes the same body:

{
  "action": "run",
  "target": { "kind": "lookalike_matches", "entity_type": "company", "matches": [ … ] },
  "ground_truth": { "kind": "crm_outcomes", "window_days": 90 },
  "criteria": ["accuracy", "ranking"]
}

The response — scores at the top, verdicts underneath, evidence on every verdict:

{
  "success": true,
  "scores": {
    "overall": 0.72,
    "per_criterion": { "accuracy": 0.78, "ranking": 0.66 }
  },
  "verdicts": [
    {
      "item_id": "9b03…",
      "verdict": "pass",
      "expected": "converted",
      "actual": "deal_won",
      "evidence": { "deal_id": "9214…", "deal_stage_status": "won", "closed_at": "2026-06-30T00:00:00Z" }
    },
    {
      "item_id": "77aa…",
      "verdict": "fail",
      "expected": "converted",
      "actual": "no_deal_opened",
      "evidence": { "interactions_in_window": 2 }
    }
  ],
  "coverage": { "items_scored": 2, "items_skipped": 0 },
  "freshness": { "source": "bigquery", "computed_at": "2026-07-22T04:12:09Z" },
  "timing": { "total_ms": 1800 }
}

Example — verify a Chat answer's figures

{
  "target": { "kind": "chat_answer", "chat_id": "3f0e2a1b-…", "run_id": "9a8b7c6d-…" },
  "ground_truth": { "kind": "crm_outcomes" },
  "criteria": ["grounding"]
}

Each stated figure in the answer becomes one verdict: pass when the claim's backing query re-runs to the same value, fail when it does not, uncertain when the claim has no backing query to re-run. An answer full of uncertain verdicts is not necessarily wrong — it is unaudited, which is its own finding.

Reading it

• verdict is three-valued: pass / fail / uncertain. uncertain means the ground truth could not settle the item (no matching CRM record, no backing query) — it is excluded from scores, counted in coverage.items_skipped, and worth rendering distinctly.
• scores are fractions over the scored set, not the submitted set. Check coverage before comparing runs: a 0.9 over 4 scored items is weaker evidence than a 0.7 over 200.
• evidence is the receipt — the CRM rows, query results, or labeled examples each verdict rests on. Store it with the score; a score without its evidence cannot be audited later.
• Ground truth has a window. crm_outcomes judges within window_days (default 90) — an account that converts on day 91 scores as a miss for that run. Pick the window to match the motion you are measuring.

The only success: false shapes are parameter validation (invalid_argument, naming the offending field) and the standard gate codes. A ground-truth source with nothing to say returns coverage.items_scored: 0 with every verdict uncertain — honest emptiness, never an error.

A prompt to hand an agent

Score yesterday's expansion list against reality — use the evals tool.

Target: the lookalike matches saved in {where you stored them}.
Ground truth: CRM outcomes over the last 90 days.
Criteria: accuracy and ranking.

Report the overall and per-criterion scores, then the per-item verdicts
with their evidence — call out every "uncertain" separately so we know
what the CRM couldn't settle. Do not average uncertain items into a score.

Tips

• Evaluate on a cadence, not once. The same target scored monthly is a trend; scored once it is trivia. Pair the call with a Routine once it ships.
• Keep the target small and labeled. 50 well-chosen items with clean ground truth beat 500 noisy ones — limit caps the call at 500 for that reason.
• ranking needs scores on the target items. It measures whether higher-ranked items out-performed lower-ranked ones; a target without per-item scores can only be scored on accuracy / coverage / grounding.

See also

• Simulate (coming soon) — the forward-looking sibling: evals scores what happened, simulate projects what would.
• Lookalike and Search — the result sets you will most often score.
• Endpoints overview — the whole surface and prerequisites.