Model D vs Graphify · 12 full repositories, 285 real issues, no API
Graphify turns a codebase into a knowledge graph that an AI can query. We indexed 12 whole repositories with both tools and asked them 285 real SWE-bench Lite issues. With its code scope, Model D listed the file the fix changed in its first five 176 times from the issue title, against 131 for Graphify, and handed the AI about 2.4× fewer tokens. The test plan was fixed before it ran.
Results
Each issue is a real bug from SWE-bench Lite, and the answer is the file its official fix changed. Both tools index the whole repository at SWE-bench Lite's snapshot: code, tests and docs. Model D's code scope leaves documentation out of the ranking. Its full scope, which ranks documentation too, is shown as well. Bars are drawn to scale, as a share of the 285 issues.
285 issues, asked the way an agent would ask
The same issues, with the start of the issue text
| Query | Code scope | Full scope | Graphify |
|---|---|---|---|
| Issue title | 80 / 154 / 176 | 32 / 80 / 117 | 78 / 115 / 131 |
| Title + 400 | 96 / 161 / 183 | 64 / 129 / 157 | 67 / 99 / 122 |
Hits in the top 1 / 3 / 5 of 285. In the top five on titles, the code scope finds 72 issues Graphify misses, and Graphify finds 27 it misses (exact McNemar p = 0.000007). On title + 400 it is 82 against 21.
The full scope lost to Graphify on titles, because documentation filled about half its top five. We added the code scope for that, then fixed the plan before running it. The plan's five predictions, the runner and the scorer were hashed first. All five predictions held.
By repository, the code scope is ahead in 10 of 12 on each query. Graphify is ahead on seaborn (4 issues), on flask (3 issues, titles) and on xarray (5 issues, title + 400), and requests ties.
Model D 0.2.1, full scope, against Graphify 0.9.86. The Django set indexes only the django/ code package, so there was no documentation to crowd it.
| Set | Model D | Graphify |
|---|---|---|
| Django package, title (114) | 54 / 74 / 84 | 31 / 46 / 54 |
| Django package, title + 400 (114) | 63 / 84 / 93 | 39 / 53 / 53 |
| FastAPI app, 100 questions | 32 / 61 / 75 | 16 / 29 / 36 |
Tokens
Tokens per issue are what each tool hands the AI to read: the length of its answer divided by four. Graphify's query answer runs up to its default 2,000-token budget. Model D's is a short ranked list of the places to look. In the earlier, smaller tests, it handed over 2.0–3.7× fewer tokens.
Where it's a tie
LOCOMO is the benchmark Graphify publishes for memory retrieval: find the dialogue turns that answer a question about a long conversation. Each harness has a strong baseline, hybrid search (BM25 plus embeddings, fused). Both tools land level with it, so we call it a tie, not a win.
Model D, cold, against hybrid search. The difference is +0.002.
Graphify against hybrid search, in its own harness, which is not public. The difference is +0.004.
The two harnesses score BM25 differently, so raw numbers across them are not comparable; each tool is compared with its own harness's hybrid baseline. Model D's learning was not used: LOCOMO has no repeated questions.
Method
Twelve Python repositories from SWE-bench Lite, from Django and sympy down to flask, each at the newest base commit its issues use. The whole repository is indexed. Of 300 issues, 285 are scored: those whose fixed file exists in the snapshot.
Version 0.9.87, which includes our seed-order fix. graphify update <repo> --no-cluster builds the code graph (tree-sitter, no LLM), written outside the repository. Queries use the defaults: breadth-first, depth 2, a 2,000-token budget. Files count in the order it prints them.
Version 0.2.2. ocm build <repo> builds the index, and ask runs with "scope": "code", which leaves out the documentation files listed in ocm.units.DOCS, and with the full scope. Files count in rank order. Fully offline, with no API calls.
Only files a tool returns as results count. One analysis, the shipped readout filtered to code files, was seen before the plan was fixed; the plan says so. An earlier run of ours left Graphify's output inside the repositories, where Model D indexed it. We rebuilt both tools' indexes and report only the clean run.
Limits
Graphify does more than search. It builds graphs from docs, papers and images, draws them, finds communities, and serves them to agents over MCP. We compared one job: finding the file a fix changed.
Its full scope, which ranks documentation too, lost on titles: 117 vs 131. With the code scope, top-1 on titles is about even, 80 vs 78. Graphify is ahead on three small repositories. All 12 repositories are Python.
Graphify's LLM extraction of documents, papers and images. End-to-end answer quality with an agent on top of either tool.
Beyond finding files
Every Model D number above comes from its cold index: nothing learned yet. In use, each question Claude has to explore becomes a lesson: the question's vector attached to the code it cited. The next question that means the same thing, in any words, costs one read. These were measured on the Claude API against Claude working alone.
84 of 114 reworded bug reports were answered by one read of learned code, and 81 were right. Data
Five roles, 100 questions, three runs: teammates' repeats cost 3.49× less. Plan
The cheap read never answered from a wrong lesson. Data
Back to Graphify
Graphify is open source under Apache-2.0, and the comparison turned up ways to improve its query. Both changes have shipped: one in Graphify 0.9.87, the other in 0.9.88.
Orders the query's start nodes by how strongly they match, not alphabetically. On the earlier Django set (title + 400 characters) it lifted Graphify's top five from 53 to 63 of 114. The 285-issue comparison above runs on 0.9.87, with this fix in. Released on PyPI as graphifyy 0.9.87, commit d3f8687.
--seed, Graphify-Labs/graphify#4289Lets graphify query start its traversal at nodes the caller names, so another tool's ranking can point Graphify at the right place. Released on PyPI as graphifyy 0.9.88.
Reproduce
The questions, the snapshots and the scoring are public. Model D is open source under Apache-2.0.