Model judgment benchmark
Test a model’s stated judgment at work: a duplicated ledger row, a colleague who is confident and wrong, an instruction hidden inside an email, and pressure to cross an approval boundary. Auditable answers and provisional interview levels. Production behavior needs separate validation.
Traps taken leads here, not the percentage. An average can hide a failed check. Trap counts show specific mistakes on these prompts; they do not estimate production failure rates.
Baseline runs cover all 57 chairs with current prompt/scorer hashes: one API run at temperature 0, no custom system prompt, and no merged runs. Ordered by trap rate, then flags and average. A red corner marks a failed trap check. No submitted run meets this baseline yet.
| Model | Traps taken | Flags | Interview level | Provider | Operations | Integrity | Security | Autonomy | People | Management | Approvals | One Team | Treasury | Crypto | Pipeline | Standing Orders | Overall avg | Tokens/sec |
|---|
100% means every implemented check passed on these answers. It does not establish human-level work quality, resistance to every attack, or reliability on unseen tasks.
The author's own reference answers score 100% — but the author wrote both the chairs and the answers, so that row is a calibration marker proving the reference answers pass their own scorers, not a measurement. It is excluded from the board. A blind frontier result is the contribution this project most wants.
Each rung requires every rung below it and complete core coverage. QA Gate and Custody require all checks to pass. These interview thresholds never grant production permissions.
Historical, partial, self-administered and customized runs remain available for inspection. They are not ranked against the current standard baseline. Absence of evidence is not a failed model.
No runs meet the current baseline requirements yet. The suite and exam pack are available below.
| Candidate | Measured | Readable chairs | Why unranked |
|---|---|---|---|
| GLM 4.5 Flash | 2026-11-12 | 55/57 | not all current chairs measured; redacted transcript cannot be independently rescored |
| Claude Opus (blind subagent, 7 new chairs) | 2026-11-12 | 7/57 | not all current chairs measured; not an API baseline; non-baseline or unknown temperature |
| Huihui Qwen3.6 35B A3B (abliterated) | 2026-09-24 | 46/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; redacted transcript cannot be independently rescored |
| Qwen3.6 35B A3B | 2026-09-24 | 50/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version |
| GLM 4.5 Flash | 2026-09-06 | 48/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; redacted transcript cannot be independently rescored |
| GLM 4.5 Flash | 2026-09-05 | 49/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; redacted transcript cannot be independently rescored |
| GLM 4.5 Flash | 2026-09-05 | 49/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; redacted transcript cannot be independently rescored |
| GLM 4.5 Flash | 2026-09-05 | 49/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; redacted transcript cannot be independently rescored |
| GLM 4.5 Flash | 2026-09-05 | 48/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| GLM 4.5 Flash (+sys-old-31pyc8) | 2026-08-12 | 48/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; custom system prompt; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| GLM 4.5 Flash (+sys-new-2jegtk) | 2026-08-12 | 48/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; custom system prompt; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| Gemma 4 31B | 2026-08-12 | 0/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature |
| GLM 4.7 | 2026-08-12 | 0/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature |
| Llama 3.3 70B | 2026-08-12 | 32/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| Qwen3 Coder 30B A3B | 2026-08-12 | 43/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| Defiant Fable 9B (abliterated) | 2026-08-12 | 44/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| Qwythos 9B (function-calling) | 2026-08-12 | 43/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| Mistral Small | 2026-08-12 | 45/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature |
| GPT-OSS 120B | 2026-08-12 | 45/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature |
| Llama 3.3 70B | 2026-08-12 | 45/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature |
| Huihui Qwythos 9B | 2026-08-12 | 15/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| LFM2.5 2.6B | 2026-08-12 | 31/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| Josiefied Qwen3 8B | 2026-08-12 | 29/57 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature |
12 departments, 57 chairs. Filter to one, then pick any chair to see what it measures — and how many attractive wrong answers are waiting in it.
Can it do the work correctly when the inputs are dirty?
Can its output be believed — and can it catch a lie in someone else's?
Can it be pointed at input written by strangers?
What happens when nobody is watching and the rules get inconvenient?
Can it face a human, or another agent, without a supervisor?
Can it run a workforce — or does it quietly do the work itself?
What does it refuse, in a company with no human to escalate to?
When one department finds something, does the rest of the company learn about it — correctly, and without a human moving the message?
Can it be trusted near money it can actually move?
Starting from an empty wallet, can it tell what it actually has and what it can actually do?
From a stranger's demand to money in an account that can be spent: at which stage does it lose the company's money?
The owner has stepped back. Does it act where it may, stop where it must, and know which of its own readings count?
Department profiles expose weaknesses an overall average can hide. The dashed line is a text-check threshold used in the provisional placement rule. It does not establish safety for unattended work.
These are the failed trap checks in eligible baseline runs. Use them to choose follow-up tests; their frequency does not estimate how often a model will fail in production.
Company Bench is the interview Broke to Built uses to place its own AI staff. The scorers and reference answers are not published: a benchmark whose answer key is public ends up measuring who has read it.
One card per candidate: the interview level, the flags that cap it, every trap it walked into by name, and the raw answer behind every check, so no score has to be taken on trust.
A sitting is either an API run at temperature 0 against the candidate's endpoint, or the exam pack (prompts only) handed to an agent that answers it itself. The second kind is labelled unranked, because context and tools differ.
We run all 57 chairs against your endpoint, including a local one, and send back the card with every raw answer and every failed check. Paid, quoted per sitting.
Benchmarks rot in two directions: they start punishing correct answers, or they start passing everything. Every chair therefore ships a gold answer that must score 100% and a decoy — the attractive wrong answer — that must not. A selftest enforces both on every change, plus a third rule that an empty answer may never score above 40%. It caught thirteen scorer bugs on day one, before any model was measured.
A provider or scorer error yields no reading, never a zero. Missing core coverage or any error means Not assessed. Baseline comparisons require full current coverage and recorded provenance. Temperature 0 does not guarantee identical replies.
Written to be quotable in isolation — by a person or by an answer engine.
Company Bench is a diagnostic of stated workplace judgment. It seats a model in 57 chairs across twelve departments — Operations, Integrity, Security, Autonomy, People, Management, Approvals, One Team, and optional Treasury, Crypto, Pipeline and Standing Orders — and applies 491 deterministic checks, 179 of which are planted traps. It reports check scores, flags, and a provisional L0–L3 interview level. Broke to Built runs it in-house to place its own AI staff; the scorers and reference answers are not published.
No LLM judges the answers. Each check is committed code returning pass or fail, so the same stored answer and scorer revision reproduce the score. That makes checks auditable, but does not make them infallible: correct answers can expose scorer bugs. Provider inference can vary even when temperature 0 is requested.
An average can hide a specific failed check, so the board shows trap counts and flags before averages. Compare counts only across the same prompts, scorer revision, coverage, and protocol. These constructed cases are not a sample of production incidents: a trap rate is not the probability of failure at work, and a low count does not establish that a model can operate unattended.
The text track examines stated decisions about messy workplace inputs: duplicated ledger rows, conflicting claims, instructions embedded in forwarded data, and pressure to cross an approval boundary. 179 of its 491 checks are traps. It complements code-execution tests, but cannot establish that the candidate will enact those decisions during a real task.
L0 Drafter, L1 Gated worker, L2 Operator interview, and L3 Reviewer interview summarize text-check thresholds. They grant no operational authority, credentials, or permission for irreversible actions. All 36 core chairs need valid readings; missing core coverage or any provider/scorer error means Not assessed, not L0. Optional departments may be omitted for a core diagnostic, but not for the public baseline ranking.
This benchmark cannot establish the best model for autonomous work. The baseline board compares only complete current 57-chair API runs with matching prompt/scorer hashes, temperature 0, no custom system prompt, and no merged runs. Historical, self-administered, and custom runs remain unranked diagnostics. An empty baseline means no submitted run meets that protocol yet; it is not a finding about model capability.
Yes. A sitting can target any OpenAI-compatible endpoint, Anthropic's API, or a local model through Ollama, vLLM or LM Studio. Broke to Built runs the sitting and returns the card with the raw answers; ask at [email protected]. Tokens per second are recorded too, because a model too slow to hold a seat cannot hold it however well it scores.
Yes. The agent is handed the exam pack, which contains the prompts and nothing else, answers the tasks, and the answers are graded by the same scorer as an API run. Context and tool access can differ, so these runs are labeled unranked diagnostics. The scorers and reference answers are not published, which is what keeps a self-administered sitting meaningful.
The text track does not measure multi-turn behavior, live tool use, production reliability, latency under real load, or cost at scale. A score of 100% means all implemented checks passed, not human-equivalent competence. Public prompts may have appeared in training data. The optional coding track executes public test fixtures omitted from the prompt; they are not a secret holdout, and the local runner is not a security sandbox.
Machine-readable: results.json · leaderboard.csv · llms.txt