Product · working today

Checkpoint — quality control for AI.

Companies freeze AI projects because nobody can answer a simple question: "did this update make it worse?" Checkpoint answers it with a red or green light. Every update takes an exam — if the score drops, it doesn't ship.

every updatetakes the exam automatically — no exceptions, no opinions
2other products here already gated by Checkpoint
1planted failure in the demo, so you can see it get caught

The problem, in plain terms

Normal software either works or it doesn't, and tests prove it. AI is fuzzier — an "improvement" can quietly make answers worse, and teams only find out from customers. That fear is why so many AI pilots never launch. The fix is boring and powerful: an exam the AI must pass before every release.

How Checkpoint works

1 · Exam sheetreal questions with known right answers
→
2 · Hard checksfacts cite real sources, no forbidden content
→
3 · Graded reviewa judge scores quality, one criterion at a time
→
4 · The gatescore too low? The release is blocked

Checkpoint also compares any two exam runs and lists exactly which questions got worse — so a team knows what broke before customers do.

Proof it works

The demo plants one deliberately bad answer — made-up statistics citing a source that doesn't exist. Checkpoint catches it three separate ways, and the report shows the catch. Two other products on this site already use Checkpoint as their release gate: Radar's accuracy exam and Compass's fact-checking exam run on every update.

For technical readers
  • Suites are YAML + JSONL golden datasets; the system under test is any Python callable or HTTP endpoint.
  • Deterministic checks run before judges: required fields, citation resolution, banned content, latency, expected-label matching.
  • LLM-as-judge with structured verdicts (one criterion per judge) plus an offline heuristic fallback so CI needs no credentials.
  • CI integration: the gate fails the build; run reports are diffable artifacts.

What it can't do yet