AR6

Benchmark · replayed · Spicilège

Does a model guess why a link was kept?

For each saved link, the reason it was kept: to test, to learn, to read, to follow, to build on, or nothing. The truth is Julien’s own answers, one link at a time; everything else is a prediction. The same question is replayed whenever the method or the model changes.

5 runs, 13 Oct to 15 Oct 2026 · best so far 57 % m

23 % the floor: always answer “to test”

75 % the ceiling, roughly: Two independent agents agree on 75 % of reasons (kappa 0.70): the ceiling is not 100 %.

0255075100 always “to test” agents agree 26 13 Oct 0 ex. 45 13 Oct 69 ex. 32 14 Oct 30 ex. 38 14 Oct 69 ex. 57 15 Oct 70 ex.
Exact reasons guessed, in %. Circles: models in the cloud; squares: a model on the Mac. The dashed line marks a change of method and benchmark: what lies on either side does not compare.

Runs

  1. Claude agents, content only cloud

    pass 1 · 0 examples · 138 choices

    26 %m

  2. Claude agents, profile and examples cloud

    pass 2, cross-validated on two folds of 69 · 69 examples · 138 choices

    45 %m 66 % in families

  3. Qwen3.8 Flash-Next, oMLX, on the Mac local

    v2 · 30 examples · 138 choices · 5 min

    32 %m 57 % in families

  4. Qwen3.8 Flash-Next, oMLX, on the Mac local

    v2 · 69 examples · 138 choices · 8 min

    38 %m 60 % in families

  5. protocol change Method v3 merges “to integrate” into “to test” (“before integrating, you test anyway”), and the benchmark grew from 138 to 186 choices: not comparable with the runs above.

  6. Qwen3.8 Flash-Next, oMLX, on the Mac local

    v3 · 70 examples · 186 choices · 11 min

    106 of 186; 6 answers unreadable.

    57 %m 62 % in families

Replayed whenever the method or the model changes. What it does not show: a guess is only a proposal; Julien confirms or corrects each reason, and every correction joins the benchmark.