Replay + automated scoring โ baseline vs. candidate models, per role
claude-sonnet-5 (2026-09-18)gpt-4o-mini (2026-09-18)gpt-5.6-terra (2026-09-24; gpt-4o
silently left ~1/3 of articles unscored)gpt-4o-mini (ties every check, 20x+ cheaper)eval/score.py (Phase 2, built โ run manually now that the daily
trial cron has ended) โ a "โ" means that run hasn't been scored; never populated for Helper
(its output is already a fact-check artifact, not prose to score this way). The Human
column comes from eval/review.py's blind A/B (Phase 3) โ a "โ" means nobody's
reviewed that row yet. Blind LLM-judge A/B is the one piece not built. Promotion needs all
four gates to hold โ see MODEL_EVAL_PLAN.md in the repo for the full design,
promotion rule, and decision log.
writer (HTML
briefing + podcast scripts), critic (section scorer that decides what
gets patched), helper (NY Sports fact-checker), or
sports_researcher (calls ESPN tools to verify NY team facts),
article_scorer (1-100 relevance scoring that decides the day's story lineup),
article_enricher ("why it matters" lines).anthropic/claude-sonnet-5).baseline = the model currently live in production for that role.
candidate = a model being evaluated as a possible replacement.k repeats
(writer runs use k=2+ to capture variance; critic/helper typically use k=1).eval/pricing.py's
rate table.k repeats errored or timed out, shown as x/k.X/Y unsupported โ of Y sentences checked against source
articles, X came back unsupported (a hallucination signal). This is the
raw count behind the Gate column; lower is better. For sports_researcher
rows, Y is the number of team facts stated and X counts numbers or
timing words (TONIGHT, LAST NIGHTโฆ) that the ESPN tools never returned; Format and
Editorial don't apply. For article_scorer rows, Format is the Spearman
rank correlation (ฯ) vs the baseline, Grounding is articles left unscored (default 50 after
a failed batch), and Editorial is how many of the baseline's top 8 the model also picks.
For article_enricher rows, Format counts "why it matters" lines over 30 words or
multi-sentence, and Grounding counts lines containing a number absent from the source.eval/review.py) โ baseline and candidate are
shown unlabeled as "A"/"B" and only revealed after scoring. A candidate's tag shows
whether it beat (PASS), tied
(TIE), or lost (FAIL)
to the baseline in that specific pairing โ a baseline has no tag, since it's the
reference point. A model reviewed against more than one candidate shows the average
of those scores, marked (avg of N).eval/out/<run_id>/
directory if you want to inspect the raw generated content behind any row.| Role | Fixture date | Model | Label | Avg latency | Avg cost | Failures | Gate | Format | Grounding | Editorial | Human | Run ID |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| article_enricher | 2026-09-22 | openai/gpt-4o-mini | baseline | 5.4s | $0.0005 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-22_1790079632 |
| article_enricher | 2026-09-22 | openai/gpt-5.6-luna | candidate | 10.1s | $0.0013 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-22_1790079632 |
| article_enricher | 2026-09-22 | openai/gpt-5.6-terra | candidate | 9.5s | $0.0094 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-22_1790079632 |
| article_enricher | 2026-09-22 | anthropic/claude-sonnet-5 | candidate | 9.3s | $0.0151 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-22_1790079632 |
| article_enricher | 2026-09-23 | openai/gpt-4o-mini | baseline | 6.5s | $0.0005 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-23_1790166059 |
| article_enricher | 2026-09-23 | openai/gpt-5.6-luna | candidate | 12.4s | $0.0012 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-23_1790166059 |
| article_enricher | 2026-09-23 | openai/gpt-5.6-terra | candidate | 10.2s | $0.0090 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-23_1790166059 |
| article_enricher | 2026-09-23 | anthropic/claude-sonnet-5 | candidate | 10.2s | $0.0150 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-23_1790166059 |
| article_enricher | 2026-09-24 | openai/gpt-4o-mini | baseline | 5.8s | $0.0005 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-24_1790252446 |
| article_enricher | 2026-09-24 | openai/gpt-5.6-luna | candidate | 9.8s | $0.0012 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-24_1790252446 |
| article_enricher | 2026-09-24 | openai/gpt-5.6-terra | candidate | 9.1s | $0.0088 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-24_1790252446 |
| article_enricher | 2026-09-24 | anthropic/claude-sonnet-5 | candidate | 8.4s | $0.0130 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-24_1790252446 |
| article_enricher | 2026-09-25 | openai/gpt-4o-mini | baseline | 5.3s | $0.0005 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-25_1790338875 |
| article_enricher | 2026-09-25 | openai/gpt-5.6-luna | candidate | 9.3s | $0.0011 | 0/1 | PASS | 1/15 | 0/15 | โ | โ | article_enricher_2026-09-25_1790338875 |
| article_enricher | 2026-09-25 | openai/gpt-5.6-terra | candidate | 15.5s | $0.0110 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-25_1790338875 |
| article_enricher | 2026-09-25 | anthropic/claude-sonnet-5 | candidate | 11.2s | $0.0150 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-25_1790338875 |
| article_enricher | 2026-09-26 | openai/gpt-4o-mini | baseline | 8.4s | $0.0005 | 0/1 | PASS | 2/15 | 0/15 | โ | โ | article_enricher_2026-09-26_1790425238 |
| article_enricher | 2026-09-26 | openai/gpt-5.6-luna | candidate | 10.1s | $0.0011 | 0/1 | PASS | 1/15 | 0/15 | โ | โ | article_enricher_2026-09-26_1790425238 |
| article_enricher | 2026-09-26 | openai/gpt-5.6-terra | candidate | 14.6s | $0.0102 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-26_1790425238 |
| article_enricher | 2026-09-26 | anthropic/claude-sonnet-5 | candidate | 11.0s | $0.0160 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-26_1790425238 |
| article_enricher | 2026-09-29 | openai/gpt-4o-mini | baseline | 8.1s | $0.0006 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-29_1790684458 |
| article_enricher | 2026-09-29 | openai/gpt-5.6-luna | candidate | 8.9s | $0.0011 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-29_1790684458 |
| article_enricher | 2026-09-29 | openai/gpt-5.6-terra | candidate | 12.7s | $0.0105 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-29_1790684458 |
| article_enricher | 2026-09-29 | anthropic/claude-sonnet-5 | candidate | 15.5s | $0.0163 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-29_1790684458 |
| article_enricher | 2026-09-30 | openai/gpt-4o-mini | baseline | 7.7s | $0.0005 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-30_1790770848 |
| article_enricher | 2026-09-30 | openai/gpt-5.6-luna | candidate | 8.9s | $0.0013 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-30_1790770848 |
| article_enricher | 2026-09-30 | openai/gpt-5.6-terra | candidate | 10.2s | $0.0098 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-30_1790770848 |
| article_enricher | 2026-09-30 | anthropic/claude-sonnet-5 | candidate | 9.1s | $0.0156 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-09-30_1790770848 |
| article_enricher | 2026-10-03 | openai/gpt-4o-mini | baseline | 6.3s | $0.0005 | 0/1 | PASS | 4/15 | 0/15 | โ | โ | article_enricher_2026-10-03_1791030036 |
| article_enricher | 2026-10-03 | openai/gpt-5.6-luna | candidate | 7.5s | $0.0011 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-10-03_1791030036 |
| article_enricher | 2026-10-03 | openai/gpt-5.6-terra | candidate | 8.0s | $0.0087 | 0/1 | PASS | 1/15 | 0/15 | โ | โ | article_enricher_2026-10-03_1791030036 |
| article_enricher | 2026-10-03 | anthropic/claude-sonnet-5 | candidate | 9.0s | $0.0140 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-10-03_1791030036 |
| article_enricher | 2026-10-04 | openai/gpt-4o-mini | baseline | 5.9s | $0.0006 | 0/1 | PASS | 1/15 | 0/15 | โ | โ | article_enricher_2026-10-04_1791116447 |
| article_enricher | 2026-10-04 | openai/gpt-5.6-luna | candidate | 7.4s | $0.0013 | 0/1 | PASS | 1/15 | 0/15 | โ | โ | article_enricher_2026-10-04_1791116447 |
| article_enricher | 2026-10-04 | openai/gpt-5.6-terra | candidate | 10.0s | $0.0126 | 0/1 | PASS | 2/15 | 0/15 | โ | โ | article_enricher_2026-10-04_1791116447 |
| article_enricher | 2026-10-04 | anthropic/claude-sonnet-5 | candidate | 9.6s | $0.0161 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-10-04_1791116447 |
| article_enricher | 2026-10-05 | openai/gpt-4o-mini | baseline | 5.7s | $0.0005 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-10-05_1791202843 |
| article_enricher | 2026-10-05 | openai/gpt-5.6-luna | candidate | 9.8s | $0.0013 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-10-05_1791202843 |
| article_enricher | 2026-10-05 | openai/gpt-5.6-terra | candidate | 9.8s | $0.0103 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-10-05_1791202843 |
| article_enricher | 2026-10-05 | anthropic/claude-sonnet-5 | candidate | 8.2s | $0.0156 | 0/1 | PASS | 0/15 | 0/15 | โ | โ | article_enricher_2026-10-05_1791202843 |
| article_scorer | 2026-09-22 | openai/gpt-4o | baseline | 6.7s | $0.0097 | 0/1 | PASS | โ | 6/21 | โ | โ | article_scorer_2026-09-22_1790079602 |
| article_scorer | 2026-09-22 | openai/gpt-5.6-terra | candidate | 6.3s | $0.0156 | 0/1 | PASS | ฯ 0.29 | 0/21 | 6/8 | โ | article_scorer_2026-09-22_1790079602 |
| article_scorer | 2026-09-22 | openai/gpt-5.6-sol | candidate | 9.1s | $0.0295 | 0/1 | PASS | ฯ 0.538 | 0/21 | 7/8 | โ | article_scorer_2026-09-22_1790079602 |
| article_scorer | 2026-09-22 | anthropic/claude-sonnet-5 | candidate | 5.9s | $0.0231 | 0/1 | PASS | ฯ 0.388 | 0/21 | 6/8 | โ | article_scorer_2026-09-22_1790079602 |
| article_scorer | 2026-09-23 | openai/gpt-4o | baseline | 5.8s | $0.0188 | 0/1 | PASS | โ | 17/51 | โ | โ | article_scorer_2026-09-23_1790166002 |
| article_scorer | 2026-09-23 | openai/gpt-5.6-terra | candidate | 12.7s | $0.0316 | 0/1 | PASS | ฯ 0.572 | 0/51 | 3/8 | โ | article_scorer_2026-09-23_1790166002 |
| article_scorer | 2026-09-23 | openai/gpt-5.6-sol | candidate | 24.3s | $0.0669 | 0/1 | PASS | ฯ 0.62 | 0/51 | 3/8 | โ | article_scorer_2026-09-23_1790166002 |
| article_scorer | 2026-09-23 | anthropic/claude-sonnet-5 | candidate | 12.1s | $0.0482 | 0/1 | PASS | ฯ 0.505 | 0/51 | 3/8 | โ | article_scorer_2026-09-23_1790166002 |
| article_scorer | 2026-09-24 | openai/gpt-4o | baseline | 4.9s | $0.0175 | 0/1 | PASS | โ | 15/43 | โ | โ | article_scorer_2026-09-24_1790252402 |
| article_scorer | 2026-09-24 | openai/gpt-5.6-terra | candidate | 10.1s | $0.0282 | 0/1 | PASS | ฯ 0.571 | 0/43 | 3/8 | โ | article_scorer_2026-09-24_1790252402 |
| article_scorer | 2026-09-24 | openai/gpt-5.6-sol | candidate | 16.7s | $0.0564 | 0/1 | PASS | ฯ 0.612 | 0/43 | 5/8 | โ | article_scorer_2026-09-24_1790252402 |
| article_scorer | 2026-09-24 | anthropic/claude-sonnet-5 | candidate | 10.6s | $0.0375 | 0/1 | PASS | ฯ 0.54 | 0/43 | 4/8 | โ | article_scorer_2026-09-24_1790252402 |
| article_scorer | 2026-09-25 | openai/gpt-4o | baseline | 8.1s | $0.0346 | 0/1 | PASS | โ | 52/86 | โ | โ | article_scorer_2026-09-25_1790338802 |
| article_scorer | 2026-09-25 | openai/gpt-5.6-terra | candidate | 25.6s | $0.0476 | 0/1 | PASS | ฯ 0.28 | 0/86 | 5/8 | โ | article_scorer_2026-09-25_1790338802 |
| article_scorer | 2026-09-25 | openai/gpt-5.6-sol | candidate | 21.5s | $0.0928 | 0/1 | PASS | ฯ 0.199 | 0/86 | 2/8 | โ | article_scorer_2026-09-25_1790338802 |
| article_scorer | 2026-09-25 | anthropic/claude-sonnet-5 | candidate | 15.9s | $0.0667 | 0/1 | FAIL | ฯ 0.359 | 62/86 | 3/8 | โ | article_scorer_2026-09-25_1790338802 |
| article_scorer | 2026-09-26 | openai/gpt-4o | baseline | 5.2s | $0.0185 | 0/1 | PASS | โ | 13/39 | โ | โ | article_scorer_2026-09-26_1790425203 |
| article_scorer | 2026-09-26 | openai/gpt-5.6-terra | candidate | 10.2s | $0.0264 | 0/1 | PASS | ฯ 0.428 | 0/39 | 3/8 | โ | article_scorer_2026-09-26_1790425203 |
| article_scorer | 2026-09-26 | openai/gpt-5.6-sol | candidate | 8.6s | $0.0503 | 0/1 | PASS | ฯ 0.42 | 0/39 | 4/8 | โ | article_scorer_2026-09-26_1790425203 |
| article_scorer | 2026-09-26 | anthropic/claude-sonnet-5 | candidate | 9.7s | $0.0359 | 0/1 | PASS | ฯ 0.456 | 0/39 | 4/8 | โ | article_scorer_2026-09-26_1790425203 |
| article_scorer | 2026-09-29 | openai/gpt-4o | baseline | 6.5s | $0.0233 | 0/1 | PASS | โ | 15/50 | โ | โ | article_scorer_2026-09-29_1790684403 |
| article_scorer | 2026-09-29 | openai/gpt-5.6-terra | candidate | 12.3s | $0.0304 | 0/1 | PASS | ฯ 0.681 | 0/50 | 5/8 | โ | article_scorer_2026-09-29_1790684403 |
| article_scorer | 2026-09-29 | openai/gpt-5.6-sol | candidate | 20.5s | $0.0646 | 0/1 | PASS | ฯ 0.753 | 0/50 | 4/8 | โ | article_scorer_2026-09-29_1790684403 |
| article_scorer | 2026-09-29 | anthropic/claude-sonnet-5 | candidate | 13.7s | $0.0487 | 0/1 | FAIL | ฯ 0.724 | 24/50 | 3/8 | โ | article_scorer_2026-09-29_1790684403 |
| article_scorer | 2026-09-30 | openai/gpt-4o | baseline | 8.0s | $0.0281 | 0/1 | PASS | โ | 10/50 | โ | โ | article_scorer_2026-09-30_1790770803 |
| article_scorer | 2026-09-30 | openai/gpt-5.6-terra | candidate | 11.5s | $0.0310 | 0/1 | PASS | ฯ 0.484 | 0/50 | 4/8 | โ | article_scorer_2026-09-30_1790770803 |
| article_scorer | 2026-09-30 | openai/gpt-5.6-sol | candidate | 13.6s | $0.0603 | 0/1 | PASS | ฯ 0.6 | 0/50 | 5/8 | โ | article_scorer_2026-09-30_1790770803 |
| article_scorer | 2026-09-30 | anthropic/claude-sonnet-5 | candidate | 10.0s | $0.0496 | 0/1 | PASS | ฯ 0.411 | 0/50 | 5/8 | โ | article_scorer_2026-09-30_1790770803 |
| article_scorer | 2026-10-03 | openai/gpt-4o | baseline | 5.0s | $0.0213 | 0/1 | PASS | โ | 9/45 | โ | โ | article_scorer_2026-10-03_1791030002 |
| article_scorer | 2026-10-03 | openai/gpt-5.6-terra | candidate | 7.5s | $0.0285 | 0/1 | PASS | ฯ 0.576 | 0/45 | 3/8 | โ | article_scorer_2026-10-03_1791030002 |
| article_scorer | 2026-10-03 | openai/gpt-5.6-sol | candidate | 10.7s | $0.0596 | 0/1 | PASS | ฯ 0.559 | 0/45 | 3/8 | โ | article_scorer_2026-10-03_1791030002 |
| article_scorer | 2026-10-03 | anthropic/claude-sonnet-5 | candidate | 8.4s | $0.0436 | 0/1 | PASS | ฯ 0.479 | 0/45 | 5/8 | โ | article_scorer_2026-10-03_1791030002 |
| article_scorer | 2026-10-04 | openai/gpt-4o | baseline | 3.8s | $0.0269 | 0/1 | PASS | โ | 55/63 | โ | โ | article_scorer_2026-10-04_1791116403 |
| article_scorer | 2026-10-04 | openai/gpt-5.6-terra | candidate | 10.7s | $0.0391 | 0/1 | PASS | ฯ 0.24 | 0/63 | 2/8 | โ | article_scorer_2026-10-04_1791116403 |
| article_scorer | 2026-10-04 | openai/gpt-5.6-sol | candidate | 14.4s | $0.0765 | 0/1 | PASS | ฯ 0.298 | 0/63 | 3/8 | โ | article_scorer_2026-10-04_1791116403 |
| article_scorer | 2026-10-04 | anthropic/claude-sonnet-5 | candidate | 13.9s | $0.0499 | 0/1 | PASS | ฯ 0.346 | 25/63 | 4/8 | โ | article_scorer_2026-10-04_1791116403 |
| article_scorer | 2026-10-05 | openai/gpt-4o | baseline | 4.7s | $0.0250 | 0/1 | PASS | โ | 42/53 | โ | โ | article_scorer_2026-10-05_1791202802 |
| article_scorer | 2026-10-05 | openai/gpt-5.6-terra | candidate | 8.4s | $0.0343 | 0/1 | PASS | ฯ 0.552 | 0/53 | 4/8 | โ | article_scorer_2026-10-05_1791202802 |
| article_scorer | 2026-10-05 | openai/gpt-5.6-sol | candidate | 13.7s | $0.0706 | 0/1 | PASS | ฯ 0.608 | 0/53 | 7/8 | โ | article_scorer_2026-10-05_1791202802 |
| article_scorer | 2026-10-05 | anthropic/claude-sonnet-5 | candidate | 12.4s | $0.0563 | 0/1 | PASS | ฯ 0.625 | 0/53 | 6/8 | โ | article_scorer_2026-10-05_1791202802 |
| critic | 2026-09-13 | openai/gpt-4o | baseline | 5.2s | $0.0012 | 0/1 | PASS | OK | 2/40 | 6.5 | โ | critic_2026-09-13_1789346036 |
| critic | 2026-09-13 | openai/gpt-5.6-terra | candidate | 30.1s | $0.0295 | 0/1 | PASS | OK | 2/40 | 6.5 | โ | critic_2026-09-13_1789346036 |
| critic | 2026-09-13 | anthropic/claude-sonnet-5 | candidate | 6.6s | $0.0026 | 0/1 | FAIL | OK | 3/40 | 6.5 | โ | critic_2026-09-13_1789346036 |
| critic | 2026-09-14 | openai/gpt-4o | baseline | 34.0s | $0.0330 | 0/1 | PASS | OK | 4/40 | 6.5 | 4.0 (avg of 2) | critic_2026-09-14_1789387634 |
| critic | 2026-09-14 | openai/gpt-5.6-terra | candidate | 13.0s | $0.0036 | 0/1 | PASS | OK | 2/40 | 5.5 | 2.0 FAIL | critic_2026-09-14_1789387634 |
| critic | 2026-09-14 | anthropic/claude-sonnet-5 | candidate | 36.1s | $0.0379 | 0/1 | PASS | OK | 2/40 | 6.0 | 4.0 TIE | critic_2026-09-14_1789387634 |
| critic | 2026-09-15 | openai/gpt-4o | baseline | 14.3s | $0.0013 | 0/1 | PASS | OK | 6/40 | 7.0 | โ | critic_2026-09-15_1789473934 |
| critic | 2026-09-15 | openai/gpt-5.6-terra | candidate | 15.4s | $0.0032 | 0/1 | PASS | OK | 5/40 | 7.0 | โ | critic_2026-09-15_1789473934 |
| critic | 2026-09-15 | anthropic/claude-sonnet-5 | candidate | 32.6s | $0.0384 | 0/1 | FAIL | OK | 7/40 | 7.0 | โ | critic_2026-09-15_1789473934 |
| critic | 2026-09-16 | openai/gpt-4o | baseline | 15.8s | $0.0012 | 0/1 | PASS | OK | 6/40 | 5.5 | โ | critic_2026-09-16_1789560351 |
| critic | 2026-09-16 | openai/gpt-5.6-terra | candidate | 44.5s | $0.0422 | 0/1 | PASS | OK | 1/40 | 7.0 | โ | critic_2026-09-16_1789560351 |
| critic | 2026-09-16 | anthropic/claude-sonnet-5 | candidate | 38.6s | $0.0455 | 0/1 | PASS | OK | 1/40 | 5.5 | โ | critic_2026-09-16_1789560351 |
| critic | 2026-09-17 | openai/gpt-4o | baseline | 11.0s | $0.0012 | 0/1 | PASS | OK | 5/40 | 7.0 | โ | critic_2026-09-17_1789646784 |
| critic | 2026-09-17 | openai/gpt-5.6-terra | candidate | 13.7s | $0.0027 | 0/1 | PASS | OK | 5/40 | 7.0 | โ | critic_2026-09-17_1789646784 |
| critic | 2026-09-17 | anthropic/claude-sonnet-5 | candidate | 15.1s | $0.0046 | 0/1 | PASS | OK | 5/40 | 7.0 | โ | critic_2026-09-17_1789646784 |
| critic | 2026-09-18 | openai/gpt-4o | baseline | 36.2s | $0.0393 | 0/1 | PASS | FAIL | 2/40 | 6.5 | โ | critic_2026-09-18_1789733149 |
| critic | 2026-09-18 | openai/gpt-5.6-terra | candidate | 18.8s | $0.0029 | 0/1 | PASS | FAIL | 1/40 | 5.5 | โ | critic_2026-09-18_1789733149 |
| critic | 2026-09-18 | anthropic/claude-sonnet-5 | candidate | 18.6s | $0.0041 | 0/1 | PASS | FAIL | 1/40 | 5.5 | โ | critic_2026-09-18_1789733149 |
| helper | 2026-09-13 | openai/gpt-4o-mini | baseline | 5.5s | $0.0004 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-13_1789346385 |
| helper | 2026-09-13 | openai/gpt-5.6-luna | candidate | 11.4s | $0.0010 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-13_1789346385 |
| helper | 2026-09-13 | anthropic/claude-haiku-4-5 | candidate | 8.9s | $0.0044 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-13_1789346385 |
| helper | 2026-09-14 | openai/gpt-4o-mini | baseline | 13.7s | $0.0006 | 0/1 | โ | โ | โ | โ | 3.0 (avg of 2) | helper_2026-09-14_1789395834 |
| helper | 2026-09-14 | openai/gpt-5.6-luna | candidate | 34.4s | $0.0027 | 0/1 | โ | โ | โ | โ | 2.0 FAIL | helper_2026-09-14_1789395834 |
| helper | 2026-09-14 | anthropic/claude-haiku-4-5 | candidate | 13.3s | $0.0070 | 0/1 | โ | โ | โ | โ | 3.0 TIE | helper_2026-09-14_1789395834 |
| helper | 2026-09-15 | openai/gpt-4o-mini | baseline | 10.9s | $0.0009 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-15_1789474000 |
| helper | 2026-09-15 | openai/gpt-5.6-luna | candidate | 32.0s | $0.0036 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-15_1789474000 |
| helper | 2026-09-15 | anthropic/claude-haiku-4-5 | candidate | 14.3s | $0.0094 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-15_1789474000 |
| helper | 2026-09-16 | openai/gpt-4o-mini | baseline | 13.6s | $0.0011 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-16_1789560453 |
| helper | 2026-09-16 | openai/gpt-5.6-luna | candidate | 41.0s | $0.0043 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-16_1789560453 |
| helper | 2026-09-16 | anthropic/claude-haiku-4-5 | candidate | 20.7s | $0.0117 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-16_1789560453 |
| helper | 2026-09-17 | openai/gpt-4o-mini | baseline | 9.6s | $0.0008 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-17_1789646828 |
| helper | 2026-09-17 | openai/gpt-5.6-luna | candidate | 31.5s | $0.0029 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-17_1789646828 |
| helper | 2026-09-17 | anthropic/claude-haiku-4-5 | candidate | 13.2s | $0.0080 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-17_1789646828 |
| helper | 2026-09-18 | openai/gpt-4o-mini | baseline | 14.3s | $0.0011 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-18_1789733226 |
| helper | 2026-09-18 | openai/gpt-5.6-luna | candidate | 33.9s | $0.0038 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-18_1789733226 |
| helper | 2026-09-18 | anthropic/claude-haiku-4-5 | candidate | 17.0s | $0.0112 | 0/1 | โ | โ | โ | โ | โ | helper_2026-09-18_1789733226 |
| sports_researcher | 2026-09-18 | openai/gpt-4o | baseline | 1.6s | $0.0086 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-18_1789780889 |
| sports_researcher | 2026-09-18 | anthropic/claude-sonnet-5 | candidate | 4.5s | $0.0146 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-18_1789780889 |
| sports_researcher | 2026-09-18 | openai/gpt-5.6-terra | candidate | 2.8s | $0.0075 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-18_1789780889 |
| sports_researcher | 2026-09-18 | openai/gpt-5.6-sol | candidate | 3.3s | $0.0144 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-18_1789780889 |
| sports_researcher | 2026-09-19 | openai/gpt-4o | baseline | 2.1s | $0.0091 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-19_1789872484 |
| sports_researcher | 2026-09-19 | openai/gpt-5.6-terra | candidate | 2.7s | $0.0074 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-19_1789872484 |
| sports_researcher | 2026-09-19 | openai/gpt-5.6-sol | candidate | 3.1s | $0.0147 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-19_1789872484 |
| sports_researcher | 2026-09-19 | anthropic/claude-sonnet-5 | candidate | 3.4s | $0.0149 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-19_1789872484 |
| sports_researcher | 2026-09-20 | openai/gpt-4o | baseline | 1.9s | $0.0081 | 0/2 | PASS | โ | 0/0 | โ | โ | sports_researcher_2026-09-20_1789906502 |
| sports_researcher | 2026-09-20 | openai/gpt-5.6-terra | candidate | 2.5s | $0.0065 | 0/2 | PASS | โ | 0/0 | โ | โ | sports_researcher_2026-09-20_1789906502 |
| sports_researcher | 2026-09-20 | openai/gpt-5.6-sol | candidate | 2.4s | $0.0130 | 0/2 | PASS | โ | 0/0 | โ | โ | sports_researcher_2026-09-20_1789906502 |
| sports_researcher | 2026-09-20 | anthropic/claude-sonnet-5 | candidate | 2.7s | $0.0130 | 0/2 | PASS | โ | 0/0 | โ | โ | sports_researcher_2026-09-20_1789906502 |
| sports_researcher | 2026-09-21 | openai/gpt-4o | baseline | 4.5s | $0.0099 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-21_1789992902 |
| sports_researcher | 2026-09-21 | openai/gpt-5.6-terra | candidate | 5.5s | $0.0082 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-21_1789992902 |
| sports_researcher | 2026-09-21 | openai/gpt-5.6-sol | candidate | 5.0s | $0.0162 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-21_1789992902 |
| sports_researcher | 2026-09-21 | anthropic/claude-sonnet-5 | candidate | 8.3s | $0.0175 | 0/2 | FAIL | โ | 0/0 | โ | โ | sports_researcher_2026-09-21_1789992902 |
| sports_researcher | 2026-09-22 | openai/gpt-4o | baseline | 3.5s | $0.0112 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-22_1790079302 |
| sports_researcher | 2026-09-22 | openai/gpt-5.6-terra | candidate | 5.0s | $0.0105 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-22_1790079302 |
| sports_researcher | 2026-09-22 | openai/gpt-5.6-sol | candidate | 5.8s | $0.0193 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-22_1790079302 |
| sports_researcher | 2026-09-22 | anthropic/claude-sonnet-5 | candidate | 5.8s | $0.0185 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-22_1790079302 |
| sports_researcher | 2026-09-23 | openai/gpt-4o | baseline | 4.8s | $0.0192 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-23_1790165703 |
| sports_researcher | 2026-09-23 | openai/gpt-5.6-terra | candidate | 5.3s | $0.0164 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-23_1790165703 |
| sports_researcher | 2026-09-23 | openai/gpt-5.6-sol | candidate | 7.2s | $0.0326 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-23_1790165703 |
| sports_researcher | 2026-09-23 | anthropic/claude-sonnet-5 | candidate | 11.4s | $0.0331 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-23_1790165703 |
| sports_researcher | 2026-09-24 | openai/gpt-4o | baseline | 2.9s | $0.0109 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-24_1790252103 |
| sports_researcher | 2026-09-24 | openai/gpt-5.6-terra | candidate | 4.8s | $0.0097 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-24_1790252103 |
| sports_researcher | 2026-09-24 | openai/gpt-5.6-sol | candidate | 4.4s | $0.0186 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-24_1790252103 |
| sports_researcher | 2026-09-24 | anthropic/claude-sonnet-5 | candidate | 5.4s | $0.0190 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-24_1790252103 |
| sports_researcher | 2026-09-25 | openai/gpt-4o | baseline | 3.3s | $0.0161 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-25_1790338503 |
| sports_researcher | 2026-09-25 | openai/gpt-5.6-terra | candidate | 5.5s | $0.0138 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-25_1790338503 |
| sports_researcher | 2026-09-25 | openai/gpt-5.6-sol | candidate | 5.6s | $0.0272 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-25_1790338503 |
| sports_researcher | 2026-09-25 | anthropic/claude-sonnet-5 | candidate | 9.0s | $0.0270 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-25_1790338503 |
| sports_researcher | 2026-09-26 | openai/gpt-4o | baseline | 3.0s | $0.0108 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-26_1790424903 |
| sports_researcher | 2026-09-26 | openai/gpt-5.6-terra | candidate | 3.9s | $0.0094 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-26_1790424903 |
| sports_researcher | 2026-09-26 | openai/gpt-5.6-sol | candidate | 3.6s | $0.0183 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-26_1790424903 |
| sports_researcher | 2026-09-26 | anthropic/claude-sonnet-5 | candidate | 4.7s | $0.0178 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-09-26_1790424903 |
| sports_researcher | 2026-09-29 | openai/gpt-4o | baseline | 3.5s | $0.0145 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-29_1790684102 |
| sports_researcher | 2026-09-29 | openai/gpt-5.6-terra | candidate | 4.6s | $0.0130 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-29_1790684102 |
| sports_researcher | 2026-09-29 | openai/gpt-5.6-sol | candidate | 4.1s | $0.0253 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-29_1790684102 |
| sports_researcher | 2026-09-29 | anthropic/claude-sonnet-5 | candidate | 19.8s | $0.0433 | 0/2 | PASS | โ | 0/1 | โ | โ | sports_researcher_2026-09-29_1790684102 |
| sports_researcher | 2026-09-30 | openai/gpt-4o | baseline | 3.8s | $0.0158 | 0/2 | PASS | โ | 0/2 | โ | โ | sports_researcher_2026-09-30_1790770502 |
| sports_researcher | 2026-09-30 | openai/gpt-5.6-terra | candidate | 6.5s | $0.0092 | 0/2 | PASS | โ | 0/2 | โ | โ | sports_researcher_2026-09-30_1790770502 |
| sports_researcher | 2026-09-30 | openai/gpt-5.6-sol | candidate | 6.0s | $0.0176 | 0/2 | PASS | โ | 0/2 | โ | โ | sports_researcher_2026-09-30_1790770502 |
| sports_researcher | 2026-09-30 | anthropic/claude-sonnet-5 | candidate | 4.6s | $0.0181 | 0/2 | PASS | โ | 0/2 | โ | โ | sports_researcher_2026-09-30_1790770502 |
| sports_researcher | 2026-10-03 | openai/gpt-4o | baseline | 3.3s | $0.0135 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-10-03_1791029702 |
| sports_researcher | 2026-10-03 | openai/gpt-5.6-terra | candidate | 4.2s | $0.0114 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-10-03_1791029702 |
| sports_researcher | 2026-10-03 | openai/gpt-5.6-sol | candidate | 3.2s | $0.0219 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-10-03_1791029702 |
| sports_researcher | 2026-10-03 | anthropic/claude-sonnet-5 | candidate | 5.8s | $0.0225 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-10-03_1791029702 |
| sports_researcher | 2026-10-04 | openai/gpt-4o | baseline | 2.7s | $0.0104 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-10-04_1791116103 |
| sports_researcher | 2026-10-04 | openai/gpt-5.6-terra | candidate | 3.9s | $0.0120 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-10-04_1791116103 |
| sports_researcher | 2026-10-04 | openai/gpt-5.6-sol | candidate | 4.3s | $0.0234 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-10-04_1791116103 |
| sports_researcher | 2026-10-04 | anthropic/claude-sonnet-5 | candidate | 6.2s | $0.0250 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-10-04_1791116103 |
| sports_researcher | 2026-10-05 | openai/gpt-4o | baseline | 3.4s | $0.0122 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-10-05_1791202502 |
| sports_researcher | 2026-10-05 | openai/gpt-5.6-terra | candidate | 4.1s | $0.0135 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-10-05_1791202502 |
| sports_researcher | 2026-10-05 | openai/gpt-5.6-sol | candidate | 3.5s | $0.0261 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-10-05_1791202502 |
| sports_researcher | 2026-10-05 | anthropic/claude-sonnet-5 | candidate | 6.6s | $0.0285 | 0/2 | PASS | โ | 0/3 | โ | โ | sports_researcher_2026-10-05_1791202502 |
| writer | 2026-09-13 | anthropic/claude-sonnet-4-6 | baseline | 62.3s | $0.1144 | 0/2 | PASS | OK | 7/40 | 6.5 | โ | writer_2026-09-13_1789344065 |
| writer | 2026-09-13 | anthropic/claude-sonnet-5 | candidate | 54.9s | $0.1258 | 0/2 | PASS | OK | 3/40 | 6.5 | โ | writer_2026-09-13_1789344065 |
| writer | 2026-09-13 | openai/gpt-5.6-sol | candidate | 57.3s | $0.2072 | 0/2 | PASS | OK | 4/40 | 5.0 | โ | writer_2026-09-13_1789344065 |
| writer | 2026-09-14 | anthropic/claude-sonnet-4-6 | baseline | 63.4s | $0.1126 | 0/2 | PASS | OK | 6/40 | 5.5 | 4.0 (avg of 2) | writer_2026-09-14_1789387203 |
| writer | 2026-09-14 | anthropic/claude-sonnet-5 | candidate | 58.3s | $0.1260 | 0/2 | PASS | OK | 3/40 | 6.5 | 3.0 FAIL | writer_2026-09-14_1789387203 |
| writer | 2026-09-14 | openai/gpt-5.6-sol | candidate | 92.2s | $0.2039 | 0/2 | PASS | FAIL | 6/40 | 5.0 | 3.0 FAIL | writer_2026-09-14_1789387203 |
| writer | 2026-09-15 | anthropic/claude-sonnet-4-6 | baseline | 68.4s | $0.1183 | 0/2 | PASS | OK | 4/40 | 5.0 | โ | writer_2026-09-15_1789473603 |
| writer | 2026-09-15 | anthropic/claude-sonnet-5 | candidate | 40.8s | $0.1124 | 0/2 | PASS | OK | 4/40 | 6.5 | โ | writer_2026-09-15_1789473603 |
| writer | 2026-09-15 | openai/gpt-5.6-sol | candidate | 54.5s | $0.2162 | 0/2 | FAIL | FAIL | 8/40 | 5.5 | โ | writer_2026-09-15_1789473603 |
| writer | 2026-09-16 | anthropic/claude-sonnet-4-6 | baseline | 64.7s | $0.1173 | 0/2 | PASS | FAIL | 6/40 | 5.5 | โ | writer_2026-09-16_1789560003 |
| writer | 2026-09-16 | anthropic/claude-sonnet-5 | candidate | 37.2s | $0.1084 | 0/2 | PASS | OK | 4/37 | 7.0 | โ | writer_2026-09-16_1789560003 |
| writer | 2026-09-16 | openai/gpt-5.6-sol | candidate | 70.6s | $0.2107 | 0/2 | PASS | FAIL | 5/40 | 5.5 | โ | writer_2026-09-16_1789560003 |
| writer | 2026-09-17 | anthropic/claude-sonnet-4-6 | baseline | 60.1s | $0.1092 | 0/2 | PASS | OK | 5/40 | 7.5 | โ | writer_2026-09-17_1789646402 |
| writer | 2026-09-17 | anthropic/claude-sonnet-5 | candidate | 40.9s | $0.1061 | 0/2 | PASS | OK | 4/40 | 7.5 | โ | writer_2026-09-17_1789646402 |
| writer | 2026-09-17 | openai/gpt-5.6-sol | candidate | 88.5s | $0.2086 | 0/2 | PASS | FAIL | 4/40 | 6.5 | โ | writer_2026-09-17_1789646402 |
| writer | 2026-09-18 | anthropic/claude-sonnet-4-6 | baseline | 70.4s | $0.1199 | 0/2 | PASS | FAIL | 5/40 | 6.5 | โ | writer_2026-09-18_1789732803 |
| writer | 2026-09-18 | anthropic/claude-sonnet-5 | candidate | 46.2s | $0.1159 | 0/2 | PASS | OK | 5/40 | 5.5 | โ | writer_2026-09-18_1789732803 |
| writer | 2026-09-18 | openai/gpt-5.6-sol | candidate | 54.7s | $0.1978 | 0/2 | PASS | FAIL | 4/40 | 5.5 | โ | writer_2026-09-18_1789732803 |