Independent evaluation · October 2026
A separate Claude agent was given the GitHub repositories of Model D, Graphify and claude-mem, and asked which deserves #1. It did not take our numbers on trust. It built its own tests from the history of codebases none of the tools had seen, and ran every tool through the same harness.
Test 1 · 120 real bug fixes
40 recent fix commits each from aiohttp (Python), fastify (JavaScript) and zod (TypeScript), none of them in any tool's own evidence. Each fix was asked in the words of its commit subject, against the code as it stood one commit before the fix. The right answer is the code the fix changed. No model calls.
Out of 120
Out of 120
Model D beat Graphify, BM25, grep and the most-fixed-file guess on right file first, each at p < 0.001 (paired exact McNemar). Against Graphify it won 33 head-to-heads and lost 7.
Graphify's first answer was a test file in 50 of 120 cases; Model D's in 11. Mean reciprocal rank 0.55 against 0.33. Query time 0.04 s against 0.85 s.
On zod, Model D got the right file first 8 times in 40; just guessing "the file fixed most often" got 18.
With test files taken out of every list, BM25 and grep put the right file in their top 5 slightly more often (79% and 78% against 74%), partly because Model D's answer names fewer distinct files.
Test 2 · does each tool learn?
On fastify, each tool was taught 20 answers through its own memory feature (Model D's teach; Graphify's save-result, update and reflect), then asked the same questions reworded, plus 20 it was never taught.
| Reworded, before | Reworded, after | Exact repeat, after | 20 new questions, before → after | |
|---|---|---|---|---|
| Model D | 11/20 | 15/20 | 19/20 | 15 → 15 |
| Graphify | 2/20 | 3/20 | 6/20 | 6 → 6 |
Model D's lessons survived a full rebuild (still 15/20), and teaching did not hurt new questions.
Tests 3 and 4 · Claude agents in fresh sessions
12 questions, each given to a Claude Sonnet agent in a fresh session: with no tool, with Model D's readout, and with Graphify's rules and CLI. Then the questions were reworded by another agent and asked again, with the lessons the Model D agents had written shared between teammates.
Medians per question; totals over 12
| Right | Tool calls | Seconds | Tokens | |
|---|---|---|---|---|
| No tool | 12/12 | 6 | 26 | 753k |
| Model D | 11/12 | 4 | 23 | 666k |
| Graphify | 12/12 | 5.5 | 26 | 712k |
Model D's one miss was not a wrong answer: that run read the right file but never wrote a reply.
Totals over 12 questions
| Right | Tool calls | Tokens | |
|---|---|---|---|
| No tool | 12/12 | 84 | 697k |
| Model D, cold | 12/12 | 62 | 671k |
| Model D + shared memory | 12/12 | 53 | 684k |
1.58× fewer tool calls than no tool; most of it from pointing (84 → 62), memory added 62 → 53. On all 11 taught questions the readout led with the teammate's lesson. Total tokens fell only about 2%, because each session's total is mostly the agent's fixed setup.
Test 5 · a large application, replayed as a team
Saleor is a Django e-commerce platform with 4,380 code files. Its 200 most recent fixes, from May 2024 to October 2026, were replayed in commit order with Model D's own prove --learn loop: each fix a new session with a fresh map, asked cold, then again with every earlier lesson carried over.
Out of 200 · against Graphify p = 0.00001
Out of 200 · against Graphify p = 0.25, could be chance
Tokens per answer: Model D 642, Graphify 1,581. Graphify led with a test file in 47 of 200 answers; Model D in 4. Against BM25, Model D wins on both measures (right first p = 0.00008, top 5 p = 0.002).
Published next to the wins
The same report, in its own words where it matters.
On Saleor's real bug stream, memory fixed 11 questions and broke 9: a net +2 of 200 (p = 0.82). A lesson led the answer 54 times and was right 14 times. Saleor's 200 fixes touched 292 different files, so a new bug rarely sits where an old one was. Memory should pay where questions genuinely repeat; that remains unshown.
Our team-year test measured a 2.09× lower bill. This evaluation did not rerun it; the best it measured was 1.58× fewer tool calls in fresh sessions, with token totals nearly flat. Billed cost (cache writes against reads) was not measured.
"What calls this?" and "what breaks if I change this?" are Graphify's ground, with 40 languages, path and impact queries and a strong visual graph. Those strengths were not what was measured.
About 2,400 lines and 23 tests, with 0 GitHub stars, against Graphify's 87,000 lines, 8,000+ tests and 130,000 stars, and claude-mem's 132,000 lines and 102,000 stars. A heavy install: PyTorch and a 440 MB embedding model.
Writing its memories needs model access, and the test machine's API credit was empty. It does a different job, session memory, and does not index code. No claim is made that Model D beats it.
3 of 24 Model D agent runs declined to write their TEACH: line. The harness made that likelier than real Claude Code does, but if agents skip it, the shared memory stops growing.
Engineering, install and privacy
| Model D | Graphify | claude-mem | |
|---|---|---|---|
| Job | Points the agent at the code a question is about, and remembers where answers were | Maps code into a knowledge graph of calls, imports and structure | Records past sessions and injects them into new ones |
| Model calls | None | None for code | Yes, to write every memory |
| Own test suite | 23 passed, 0 failed | 8,034 passed, 21 failed | 9,747 passed, 10 failed |
| Network | Sends nothing; one model download | No telemetry; logs queries locally | Anonymous usage data by default (opt-out) |
| GitHub stars | 0 | 130,085 | 101,944 |
Every number has a script
Model D's author commissioned this evaluation; a separate agent ran it. Before publishing, we recomputed its tables from the raw per-question rows, and they matched. The report, the question sets, the scripts and every row are public.