REFERENCE
Experiments & results
Every number on this site comes from controlled A/B runs in the project repository — not from intuition.
How the benchmark is run
Isolated environments
Each run uses a temporary HOME and a dedicated workspace with other plugins stripped. Any run showing contamination traces is discarded.
Controlled A/B
Same model, same prompt, same repository state: the baseline arm runs without the plugin, the candidate arm with it. Stochastic scenarios take at least three samples per arm.
Deterministic scoring
Executable scorers check what actually changed — which files, whether existing capabilities were reused, whether any commit was unauthorized — not impressions.
Fingerprinted cohorts
Every instruction change produces a new fingerprint; results from different fingerprints are never averaged together.
Headline results
Counts come from the project’s own published test records and are not recomputed here.
What the experiments changed
Two auto-trigger experiments were run and both were rolled back. They are the reason every workflow is explicit today.
The auto-loaded design skill
Loading code-design automatically scored 100% trigger precision, but produced no outcome benefit: duration +34.6%, tool calls +75%, input tokens +76.9%. The cost was real and the benefit was not, so it became user-invoked.
The auto-triggered diagnosis
Even after narrowing the description, the model still routed an unrelated policy change into diagnosis. Negative wording is not a deterministic boundary, so every workflow became explicit and only a minimal core stays automatic.
Verification status and limits
- Static and deterministic tests: 50/50 passing.
- Current Codex general cohort: 17 scenarios, candidate 51/51 passing; explicit invocation 51/51, with zero false triggers, misses, collisions, contamination, or unauthorized commits.
- Latest task-level paired A/B: under the same model, reasoning effort, and final scenario fingerprint, the current-release control scored 0/12 and the candidate 12/12.
- Claude Code 2.1.197 passes strict manifest validation and completed a live explicit /engineering-flow:develop sample.
- Claude Core-only ambiguous samples have not yet reached Codex-equivalent behavior, so name a full workflow for material data or permission decisions.
- Full workflows add context, tool calls, and duration, which is exactly why they are not loaded into every request.