Benchmark comparison against Graft #232

Activity

1. joyful-ii-V-I commented on Sep 16, 2026

   

   joyful-ii-V-I

   on Sep 16, 2026

   Collaborator

   More actions

   Thanks, @Mako-L! A comparison against Graft is under way. We run both tools ourselves on the same tasks, and we'll publish the wins and the losses together. We'll post here when it's out.

   ❤️React with ❤️2Reacted by Mako-L and Eliah Rusin
2. joyful-ii-V-I

   added this to the 0.6.2 milestone on Sep 16, 2026
3. Mako-L commented on Oct 25, 2026

   

   Mako-L

   on Oct 25, 2026

   Author

   More actions

   Any news on this?
4. joyful-ii-V-I commented on Oct 25, 2026

   

   joyful-ii-V-I

   on Oct 25, 2026

   Last edited by joyful-ii-V-I

   Collaborator

   More actions

   Thanks for the nudge, @Mako-L. Here's where it stands.

   We run ripwire, Graft (v0.21.1) and four other code-intelligence tools on the same set of agent questions over real open-source repos. An answer counts as complete when it names every item a full answer needs: the right files, symbols and call paths.

   Interim snapshot, on the 120 questions every tool was asked:

   tool	complete answers
   ripwire (current dev build)	87 / 120
   codegraph	35
   code-review-graph	18 (108 run)
   graphify	16
   codebase-memory-mcp	16
   Graft	13

   If you're on the current release: 0.6.8 wasn't graded directly. The nearest graded build, from two days after 0.6.8, scored 83 / 120, so 0.6.8 should be close to the dev-build figure.

   Please treat this as provisional:

   • Grading so far is mechanical. It hasn't been adjudicated by hand yet.
   • The other tools' answers were recorded in early October. The publish round re-runs every tool at its latest release.
   • Speed, answer size and follow-up calls aren't in this snapshot.
   • There are still 5 questions where another tool answers completely and ripwire doesn't. We're fixing those first.

   ripwire is still improving: new fixes land on main every few days, and we expect the numbers in 0.7.0 to be better than this snapshot. We'll publish the full table with that release, per-question results and losses included, and I'll post here when it's out.

   👀React with 👀1Reacted by Mako-L