Open-source model judgment benchmark
Test a model’s stated judgment at work: a duplicated ledger row, a colleague who is confident and wrong, an instruction hidden inside an email, and pressure to cross an approval boundary. Auditable answers and provisional interview levels. Production behavior needs separate validation.
Traps taken leads here, not the percentage. An average can hide a failed check. Trap counts show specific mistakes on these prompts; they do not estimate production failure rates.
Baseline runs cover all 50 chairs with current prompt/scorer hashes: one API run at temperature 0, no custom system prompt, and no merged runs. Ordered by trap rate, then flags and average. A red corner marks a failed trap check. No submitted run meets this baseline yet.
| Model | Traps taken | Flags | Interview level | Provider | Operations | Integrity | Security | Autonomy | People | Management | Approvals | One Team | Treasury | Crypto | Overall avg | Tokens/sec |
|---|
100% means every implemented check passed on these answers. It does not establish human-level work quality, resistance to every attack, or reliability on unseen tasks.
The author's own reference answers score 100% — but the author wrote both the chairs and the answers, so that row is a calibration marker proving the reference answers pass their own scorers, not a measurement. It is excluded from the board. A blind frontier result is the contribution this project most wants.
Each rung requires every rung below it and complete core coverage. QA Gate and Custody require all checks to pass. These interview thresholds never grant production permissions.
Historical, partial, self-administered and customized runs remain available for inspection. They are not ranked against the current standard baseline. Absence of evidence is not a failed model.
No runs meet the current baseline requirements yet. The suite and exam pack are available below.
| Candidate | Measured | Readable chairs | Why unranked |
|---|---|---|---|
| GLM 4.5 Flash | 2026-09-06 | 48/50 | not all current chairs measured; redacted transcript cannot be independently rescored |
| GLM 4.5 Flash | 2026-09-05 | 49/50 | not all current chairs measured; redacted transcript cannot be independently rescored |
| GLM 4.5 Flash | 2026-09-05 | 49/50 | not all current chairs measured; missing or different scorer version; redacted transcript cannot be independently rescored |
| GLM 4.5 Flash | 2026-09-05 | 49/50 | not all current chairs measured; missing or different scorer version; redacted transcript cannot be independently rescored |
| GLM 4.5 Flash | 2026-09-05 | 48/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| GLM 4.5 Flash (+sys-old-31pyc8) | 2026-08-12 | 48/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; custom system prompt; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| GLM 4.5 Flash (+sys-new-2jegtk) | 2026-08-12 | 48/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; custom system prompt; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| Gemma 4 31B | 2026-08-12 | 0/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature |
| GLM 4.7 | 2026-08-12 | 0/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature |
| Llama 3.3 70B | 2026-08-12 | 32/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| Qwen3 Coder 30B A3B | 2026-08-12 | 43/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| Defiant Fable 9B (abliterated) | 2026-08-12 | 44/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| Qwythos 9B (function-calling) | 2026-08-12 | 43/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| Mistral Small | 2026-08-12 | 45/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature |
| GPT-OSS 120B | 2026-08-12 | 45/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature |
| Llama 3.3 70B | 2026-08-12 | 45/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature |
| Huihui Qwythos 9B | 2026-08-12 | 15/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| LFM2.5 2.6B | 2026-08-12 | 31/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature; redacted transcript cannot be independently rescored |
| Josiefied Qwen3 8B | 2026-08-12 | 29/50 | not all current chairs measured; missing or different prompt version; missing or different scorer version; not an API baseline; non-baseline or unknown temperature |
10 departments, 50 chairs. Filter to one, then pick any chair to see what it measures — and how many attractive wrong answers are waiting in it.
Can it do the work correctly when the inputs are dirty?
Can its output be believed — and can it catch a lie in someone else's?
Can it be pointed at input written by strangers?
What happens when nobody is watching and the rules get inconvenient?
Can it face a human, or another agent, without a supervisor?
Can it run a workforce — or does it quietly do the work itself?
What does it refuse, in a company with no human to escalate to?
When one department finds something, does the rest of the company learn about it — correctly, and without a human moving the message?
Can it be trusted near money it can actually move?
Starting from an empty wallet, can it tell what it actually has and what it can actually do?
Department profiles expose weaknesses an overall average can hide. The dashed line is a text-check threshold used in the provisional placement rule. It does not establish safety for unattended work.
These are the failed trap checks in eligible baseline runs. Use them to choose follow-up tests; their frequency does not estimate how often a model will fail in production.
Two ways in, one scorer. API runs can qualify for the baseline. Self-administered runs are unranked diagnostics because context, tools, and prior exposure can differ.
git clone https://github.com/lordbasilaiassistant-sudo/company-bench.git
cd company-bench
node bench/take.mjs
# the agent answers bench-pack/TAKE-THE-BENCH.md
# into bench-pack/answers.json, then:
node bench/grade.mjs bench-pack/answers.json \
--label "Your Agent"
Claude Code: drop skills/company-bench into ~/.claude/skills/ and say
/company-bench. Any other agent: one-paste prompt in
PROMPT.md.
# free tier at console.groq.com
export GROQ_API_KEY=...
node bench/run.mjs --models groq-llama70b
node bench/run.mjs --models ollama:qwen3:8b
node bench/run.mjs --models anthropic:claude-opus-5
node bench/run.mjs --list
Any OpenAI-compatible endpoint: Groq, Z.ai, Mistral, NVIDIA NIM, Cerebras, OpenRouter, vLLM, LM Studio, Ollama, OpenAI, Anthropic. Keys authenticate requests to your configured provider; prompts and replies travel to that endpoint. Scoring runs locally. Throughput is recorded alongside the answer scores.
Benchmarks rot in two directions: they start punishing correct answers, or they start passing everything.
Every chair therefore ships a gold answer that must score 100% and a decoy — the attractive wrong
answer — that must not. node bench/selftest.mjs enforces both, plus a third rule that an empty
answer may never score above 40%. It caught thirteen scorer bugs on day one, before any model was measured.
A provider or scorer error yields no reading, never a zero. Missing core coverage or any error means Not assessed. Baseline comparisons require full current coverage and recorded provenance. Raw answers accompany results so checks can be audited; temperature 0 does not guarantee identical replies.
Written to be quotable in isolation — by a person or by an answer engine.
Company Bench is an open-source diagnostic of stated workplace judgment. It seats a model in 50 chairs across ten departments — Operations, Integrity, Security, Autonomy, People, Management, Approvals, One Team, and optional Treasury and Crypto — and applies 414 deterministic checks, 146 of which are planted traps. It reports check scores, flags, and a provisional L0–L3 interview level. It is MIT-licensed, written in Node with zero dependencies.
No LLM judges the answers. Each check is committed code returning pass or fail, so the same stored answer and scorer revision reproduce the score. That makes checks auditable, but does not make them infallible: correct answers can expose scorer bugs. Provider inference can vary even when temperature 0 is requested.
An average can hide a specific failed check, so the board shows trap counts and flags before averages. Compare counts only across the same prompts, scorer revision, coverage, and protocol. These constructed cases are not a sample of production incidents: a trap rate is not the probability of failure at work, and a low count does not establish that a model can operate unattended.
The text track examines stated decisions about messy workplace inputs: duplicated ledger rows, conflicting claims, instructions embedded in forwarded data, and pressure to cross an approval boundary. 146 of its 414 checks are traps. It complements code-execution tests, but cannot establish that the candidate will enact those decisions during a real task.
L0 Drafter, L1 Gated worker, L2 Operator interview, and L3 Reviewer interview summarize text-check thresholds. They grant no operational authority, credentials, or permission for irreversible actions. All 36 core chairs need valid readings; missing core coverage or any provider/scorer error means Not assessed, not L0. Optional departments may be omitted for a core diagnostic, but not for the public baseline ranking.
This benchmark cannot establish the best model for autonomous work. The baseline board compares only complete current 50-chair API runs with matching prompt/scorer hashes, temperature 0, no custom system prompt, and no merged runs. Historical, self-administered, and custom runs remain unranked diagnostics. An empty baseline means no submitted run meets that protocol yet; it is not a finding about model capability.
Yes. Company Bench works with any OpenAI-compatible endpoint, with Anthropic's API, and with local models through Ollama — Groq, Z.ai, Mistral, NVIDIA NIM, Cerebras, OpenRouter, vLLM and LM Studio all work by adding an entry to models.json. Run it with node bench/run.mjs --models ollama:your-model. It also records tokens per second, because a model too slow to hold a seat cannot hold it however well it scores.
Yes. Run node bench/take.mjs to create the exam pack, answer the tasks, then use node bench/grade.mjs. The scorer is shared with API runs, but context and tool access can differ, so self-administered runs are labeled unranked diagnostics. Do not read the scorers before answering; disclose prior exposure. Record model identity, commit, coverage, and surrounding context.
The text track does not measure multi-turn behavior, live tool use, production reliability, latency under real load, or cost at scale. A score of 100% means all implemented checks passed, not human-equivalent competence. Public prompts may have appeared in training data. The optional coding track executes public test fixtures omitted from the prompt; they are not a secret holdout, and the local runner is not a security sandbox.
Machine-readable: results.json · leaderboard.csv · llms.txt
Contributions wanted, in this order: a blind frontier result · harder chairs · new departments
(scheduling, support, research, compliance, hiring) · local-model results. Agents reading this repo should
start at llms.txt.