30 models & 13 harnesses put to the test · Jul–Oct 2026
For some years I have used an Obsidian vault as my knowledge base: insights, decisions, people, tasks, ideas. Can an agent keep it up to date? I recorded a real meeting (the other participant agreed beforehand to the recording and the AI processing) and let agents update my vault, putting the right insights in the right files and folders.
Some frontier cloud models get it right in 1–6 minutes, but many other cloud models fall short. Locally, only one setup gets it right reliably, in about 15 minutes: the plain Qwen3.8 27B, 4-bit, extra-high effort, on Pi. Fine-tunes hit the top once, then fall back.
| Option | Configuration | Scores | Time | Cost/run |
|---|---|---|---|---|
| Cloud: when privacy allows | ||||
| Best quality | Claude Opus 5.5 on Pi, or GPT-6.1 Sol at medium effort. One run each. | 99 | 2–5.5 min | $0.25–0.86 |
| Best value | GPT-6 Luna, max effort. Any harness. | 80–97 | 1–6 min | ~$0.04 |
| Local: when it must stay private | ||||
| Use | Qwen3.8 27B, plain model, 4-bit, extra-high effort, 65K output limit, on Pi | 86–97 | 14–19 min | $0 |
| Runner-up | Swift, a Qwen3.8 27B fine-tune, same setup. Faster, less steady. | 40–93 | 9–11 min | $0 |
One run proves little. The same Swift setup scored 93, 93, 57, 40, then 92. Models are ranked by their median run, with the worst run beside it.
The middle and the lowest score of a model's runs. Q · J = median quality and judgment, out of 50 each.
The agent program that drives the model. Pi and OpenCode were used most; Pi + custom prompt: Pi with my own system prompt.
Self-hosted: my laptop or home server. Splash is a faster inference runtime on the same home server. Cloud: a provider's API.
Gaps under about 20 points are noise. Quantization and packaging variants are merged into one row; fine-tuned derivatives keep their own. Gray edge: cloud. Green edge: my local pick.
One dot per model: its median run. Up is better, left is cheaper. Self-hosted models have no per-run bill and sit in the $0 column. Color shows how long the run took, from green (fast) to red (slow).
Logarithmic cost axis. Cost is what the provider reported for the run; cloud runs without a reported cost are left out. Only the top models are named; hover any dot for its details.
The harness is the program around the model: its prompt, tools and limits. It decides whether a run finishes, and the same model can score 30 points apart on two harnesses. Which harness works best depends on the model.
No harness wins everywhere. Pi was best for Claude Sonnet 5.5 and the self-hosted models, OpenCode 2 for Qwen3.8 Flash Next; GPT-6 Luna and DeepSeek did well on all of them. The other harnesses mostly stalled or failed their tool checks.
For self-hosted models, more than anything else. The same model, server and harness, at two effort levels:
The agent had to read the meeting transcript and update the vault. A grader then checked every file it wrote, item by item:
| Part | What was checked | Points |
|---|---|---|
| Quality of the notes: 50 | ||
| Right files | Four outputs, each in the right folder with real content: a Journal entry, a meeting note, an insight note (one idea per note), a profile of the other participant | 15 |
| Main content | The framework the other participant explained during the call. Points for each of its ideas captured accurately. | 20 |
| House style | The vault guide's conventions: frontmatter, dated file names, links to existing notes, a takeaway and a "so what" in the insight, no text copied across notes | 10 |
| Extras | Connecting anecdotes to notes already in the vault, and listing concrete next steps | 5 |
| Judgment: 50 | ||
| Who does what | Three follow-ups agreed in the call, each with the right sender: the participant sends me a short profile to forward to one of my contacts (12); I help with an introduction (5); I send them an article (3) | 20 |
| Relationship | Describes the participant as a peer and connector | 10 |
| Biography | The participant's background, not mine: right past employers and current independent work, no invented roles (9); right family origin (6) | 15 |
| Restraint | Leaves the vault's reference files about me untouched | 5 |
| Penalties | ||
| History lost | Erasing existing content such as the Journal's past entries (−10 for a single line) | −40 |
| Invented biography | A false professional identity for the participant (−10 if isolated) | −20 |
A score can go below zero. A hand-written reference answer scores 100.
How the runs were launched and graded, and the problems met · all 204 runs (CSV)
About me: I work in AI transformation by day, at a large corporation. At night I write and run experiments like this one. More on my Substack, The Intelligence Fabric.