Evals of the edakit plugin¶
A source
This page is a source of edakit – a concept, a prompt or the rules of a session –, shown as it is in the repository. Read it as such; the guides say what edakit does.
Typical tasks on the reference circuits (tests/reference.py), run by claude plugin eval with a model of choice –
concept docs/concepts/done/rules-at-scale.md, section 8, and maker-coverage FT-24: edakit must work with Sonnet, not
only with the largest model.
| Case | Task | Graded by |
|---|---|---|
fault-motors, fault-audio, fault-cables |
find the fault built into a reference circuit with edakit | the FAULT: line names the place and the code; edakit check ran; no finding invented (judge) |
fault-guitar-sim |
the same, for a fault only the simulation shows | the line; edakit sim ran; no finding invented |
coverage |
what went unchecked for lack of data | the codes and the field named; edakit check --coverage ran |
add-led |
a status LED on GP8 from the block library, until check and sim are clean | the graph has it; block add, check and sim ran; the DONE: line |
requirements-inventory |
the inventory for the requirements file of led-bar with a two-line README (#97) | the README's eight LEDs and USB port are candidates; a candidate read from the circuit says inferred; the list "already checked by edakit"; no candidate edakit derived (judge) |
first-reader |
a reader new to edakit follows the start page and the quickstart to the first finding on the walkthrough's example, with only those pages | edakit check ran; PATH.reverse named; what it means and why it is accepted (judge) |
first-circuit |
a reader new to edakit makes a checked circuit of a one-line idea – a 9 V battery, a switch, two LEDs – with the site's Markdown as the only guide, no board (documentation-newcomer.md, N.5) | edakit check ran; a graph with two LEDs; the pages named as site/…; the DONE: line without an open error; a graph of its own with battery, switch and two LEDs, every acceptance with a reason (judge on the graph); the last check clean and the DONE line true to it (judge on the trace); at most six documentation pages opened with Read – a page read on from an offset not counted again –; no Read, Grep, Glob or Bash into edakit's code (edakit/ of the plugin) |
Each case's scaffold.sh writes its project into the run's empty workspace with tests/reference.py (plain
python3, no edakit needed for that). The graders are regex and tool_used – free – and one judge per fault case and
for requirements-inventory.
Running¶
git archive HEAD | tar -x -C /tmp/edakit-eval # a copy without the .venv: uv's hard links are refused
cd /tmp/edakit-eval && UV_LINK_MODE=copy uv sync # the copy's own environment, without hard links
claude plugin eval . --scaffold --allow-tools Bash Edit Write --trust-plugin --ablation none --runs 1 \
--model claude-sonnet-5-5 --max-cost-usd 6 --no-publish
On Linux the sandbox of Bash needs bubblewrap and socat, and TMPDIR set to a directory outside /tmp,
/var/tmp and /dev/shm (mkdir -p /srv/evaltmp, owned by the user running it, TMPDIR=/srv/evaltmp): the harness
puts the run's workspace under TMPDIR and denies Bash writes to those three directories, and the denial is mounted
last – with the default /tmp, Bash cannot write the workspace (the Edit tool can), which is what the runs below and
the first two runs of first-circuit (TD-102) met. The plugin directory must contain no hard links – the harness
refuses case files reachable by another name, and uv fills a .venv with hard links from its cache –, hence the copy.
Run it as an ordinary user (useradd -m eval), not as root. A run has no network: edakit part search, edakit stock
and edakit cost cannot reach the shop, which the prompt of first-circuit says.
Baseline, 2026-10-31¶
One run per case, judge Haiku, plugin at the commit after Flyback finding names the part's kind; in a cloud container
as root.
| Case | Sonnet 5.5 | Opus 5.5 |
|---|---|---|
| fault-motors | 1.00 | 1.00 |
| fault-audio | 1.00 | 1.00 |
| fault-cables | 1.00 | 1.00 |
| fault-guitar-sim | 1.00 | 1.00 |
| coverage | 1.00 | 1.00 |
| add-led | 0.40 | 1.00 |
| cost | 0.55 USD | 1.48 USD |
add-led with Sonnet: Bash could not write the workspace (TMPDIR under /tmp, see above); Sonnet stopped and asked
how to go on, Opus wrote the graph with the Edit tool. One run per case is a first reading, not a rate: repeat with
--runs 3 before reading a difference into it.
What the first runs changed in edakit: bin/edakit falls back to the plugin's .venv where uv cannot write its cache;
edakit sim reports artifacts it could not write as a note after its complete results; the SessionStart hook names the
path of bin/edakit when edakit is not on PATH; the flyback finding names the motor's winding instead of a coil, which
a run had taken for a relay.
The explorative DSL, 2026-11-03¶
Three cases for the circuit language, a beta a project switches on with [beta] dsl = true
(docs/dsl/); their test project led-bar-circuit has the switch.
| Case | Task | Graded by |
|---|---|---|
add-led-circuit |
a status LED on GP8 from the block library, on led-bar in the circuit language (led-bar-circuit) |
the source has the usage, the resolved graph the LED; block add, check and sim ran; the DONE: line |
repeat-leds |
four LEDs with their resistors on GP16 to GP19, written as one for |
the source has a loop, the resolved graph GP19's net; check and sim ran; the DONE: line |
variant-smd |
a variant in variants/smd/ with J1 assembled (bom yes) and another button |
the variant file names its base and sets bom yes; check ran; the DONE: line |
One run per case, Sonnet 5.5, the plugin at the branch of the circuit language (28b5c1c; the two cases whose grader
threw again at 9d1289e); in a cloud container as the user eval with TMPDIR=/srv/evaltmp.
| Case | Sonnet 5.5 | Turns | Cost |
|---|---|---|---|
| add-led-circuit | 1.00 | 5 | 0.13 USD |
| repeat-leds | 1.00 – first 0.80: the grader source-has-a-for threw |
8 | 0.17 USD |
| variant-smd | 1.00 – first 0.75: the grader variant-file threw |
10 | 0.18 USD |
What the runs changed: the runner compiles a regex grader as JavaScript, which knows no inline (?m) – the flag goes
to flags, and a test refuses inline flags in every grader; docs/dsl/variants.md says what a variant keeps beside
its source, which the run of variant-smd had to find out. repeat-leds wrote its loop of parts and nets by hand, not with use
led – the case asks for a loop only. The copy of the second runs was not synced again after it was unpacked:
bin/edakit wanted to rebuild edakit from PyPI, which a run cannot reach, and Sonnet took the copy's .venv instead.