mcAgent · verityResearch
Fine-tuning earns its claim beyond tool access alone.
The control that isolates it: a prompted base model — identical
tool/oracle access, no fine-tuning — measured against the fine-tuned adapter
(v20) on the same two held-out evaluation sets, with the pass/fail
criteria for this comparison frozen before either arm was run.
Verdict
Wide margin on both sets — licenses the claim that fine-tuning on
the verified corpus contributes capability beyond tool access alone.
Results — all four arms
n = 106 held-out · n = 43 adversarial
The sharpest signal is tool-call correctness, not answer accuracy: the
base model essentially never emits a working tool call at all — 0% held-out, 2%
adversarial — regardless of how the question is phrased. Tool access alone does not
confer working tool-use behavior at this model size. The fine-tuned adapter does, at 98%
and 83%.
Full numbers
Table view — the values behind every bar above.
| Arm |
Eval set |
Tool-call |
Answer |
| base | held-out | 0/106 (0%) | 51/106 (48%) |
| base | adversarial-probe | 1/43 (2%) | 6/43 (13%) |
| v20 | held-out | 104/106 (98%) | 102/106 (96%) |
| v20 | adversarial-probe | 36/43 (83%) | 37/43 (86%) |
How this was measured
ledger frozen 2026‑09‑02
Contrast
Fine-tuned (v20 adapter) vs. prompted-base
— same base model, no adapter, identical tool/oracle access, identical harness.
Isolates whether the adapter's training contributes capability beyond what tool access
alone gives a prompted base model.
Registered before results
Primary contrast, allowed interpretations, failure criteria,
exclusions, and all four arms were written down and committed
before any result file was read. The criteria were not fitted
to the numbers after the fact — the numbers were read against
criteria that already existed.
What would undercut this
Base matching or nearing v20 on both sets would have meant no
capability-gain claim at all — describing the system, not a causal result.
Beating v20 on held-out only would have licensed a narrower recall claim, not a
robustness one. Neither happened; the wide-margin criterion is what fired.
Attempt census
All four arms launched clean on first attempt. No excluded,
invalid, or discarded runs.
Pinned state
Worktree HEAD, oracle database, training set, and probe set are
hashed and pinned in the source ledger — see link below.
Criteria applied as written, not amended after seeing results