None of these tools measure how effective they are...
It's a massive red flag to me when you could get decent data to see if your thing actually works, and they don't even attempt to...
Have the LLM use your tool, run it on several of the coding benchmarks. If you're stingy, run it on the ones that don't cost much.
Otherwise, I'm going to assume it doesn't actually work. If it did - Claude, Antigravity, Codex, Pi, or some major player would bundle tools like this into the CLI / harness.
AFAIK, none of the major players do. That's a sign to me these don't work in general.
I've tried building some tools specific to bug fixing. Intelligently feeding context massively helps smaller models. But, what I've found - surprisingly - is that a smaller, much better focused, including a lot of helpful data as well, has almost no impact on larger models compared to what they do by default.
You do save some tokens, though, which is what they're claiming - but not ~99%...
> otherwise, popular solutions would integrate the idea
None of the major players are incentivized to care about this, especially not over other opportunities. Why would you expect them to integrate it?
One of the biggest wins you can institute for your own codebase if you use agents is writing your own harness, by a huge margin. The defaults are fine, but you can do better.
They're incentivised because they're offering plans at a loss and/or pricing out potential customers. All these LLM companies are competing on accuracy and price.
If you're looking for an efficiency-focused harness, I had a pretty good time using the Dirac agent. The line-based anchors were slightly buggy though (this was a couple months ago) and would sometimes add the same line of code multiple times or leave an anchor in the output.
Creator of Dirac here. Came across this too late. The behavior you mentioned happens more commonly in smaller models, rarely in larger/frontier models. The underlying code is clean but the smaller models often make boundary errors (off by one type). Which models did you observe this with?
That was true. But actually, I think that's changed a few weeks ago since they introduced a API credit amount equivalent to your (eg. $100, $200) that will be used for such cases. So they don't ban you, they just bill you that allocated credit and then actual API cost.
Yes. That’s possible in addition to using your actual subscription. I’ve been using it via cliproxy for all harnesses and even my own code review agent hooked up to github apps. Not banned yet but I also dont do crazy stuff with openclaw or hermes
It's too hard to define what "works" even means in this case. Look at the example savings output. A lot of it is kubectl output.
Your suggestion to using coding benchmarks doesn't really capture the whole picture. I haven't seen a benchmark using kubectl.
> AFAIK, none of the major players do. That's a sign to me these don't work in general.
It's a lose/lose for major players. If it works well, it will lower their revenue. Also there's a high risk it'll significantly worsen results for some people, even if it improves results for others.
I don't think frontier model providers are going to be incentivized to invest in this much, yet. Once inference gets more competitive, sure. I haven't looked lately, but won't be surprised if tools like OpenCode do do what you're suggesting, though. Third-party coding harnesses ARE aligned to deliver this type of feature and optimization.
> I'm going to assume it doesn't actually work. If it did - Claude, Antigravity, Codex, Pi, or some major player would bundle tools like this into the CLI / harness.
So often we will burn 20% of limit in a single ill conceived agent tool call that we're simply not going to be able to or want to be able to intercept. Where I see a tool like this being a real step forward is to add a decision point. it does not have to bubble up to hard-require user to provide permission, but it can let the LLM have an intermediate checkpoint to say that it's about to get blasted with 30k tokens and here is roughly the shape of it and do you wanna adjust or whittle it down if you know what you're looking for etc.?
There is definitely tons of value to extract from this line of thinking.
You can't measure effectiveness, because you never know what kind of model will process your prompt. One request you might get full e.g. Opus and another they'll downgrade it to Sonnet or something more basic. I have this with "Opus 4.8" all the time.
The problem even attempting to develop a tool for the frontier model space is that the cost to run a statistically significant benchmark is almost certainly going to be over $100 - for a single model.
Unless something is like 25%+ more cost effective on Gemini for a task, I would not assume those savings are going to transfer to GPT.
If you need to run a test this expensive and slow for every release, hobbiests aren't going to do it.
And if you wanted any broadly specific improvements to coding like they all claim, the costs would be in the thousands per release even for a single for a single model.
And they almost certainly would not be eye popping.
If the models could be SUBSTANTIALLY better, Google and Anthropic and OpenAI wouldn't be finding that out from a hobbiest making wildly unscientific claims.
Yup, this is hitting it on the nose. But, despite the cost - the benchmark is the vital ingredient that cant be skipped. Otherwise, you don't know if what you're building is actually helping the agent rather than hindering it.
On the previous large benchmark run, i proved 40-50% cost reduction per correct answer.
I'm not sure why the vendors aren't using token filtering/compression more in their tooling, but perhaps they don't mind users feeding them more data and using more data.
It's a massive red flag to me when you could get decent data to see if your thing actually works, and they don't even attempt to...
Have the LLM use your tool, run it on several of the coding benchmarks. If you're stingy, run it on the ones that don't cost much.
Otherwise, I'm going to assume it doesn't actually work. If it did - Claude, Antigravity, Codex, Pi, or some major player would bundle tools like this into the CLI / harness.
AFAIK, none of the major players do. That's a sign to me these don't work in general.
I've tried building some tools specific to bug fixing. Intelligently feeding context massively helps smaller models. But, what I've found - surprisingly - is that a smaller, much better focused, including a lot of helpful data as well, has almost no impact on larger models compared to what they do by default.
You do save some tokens, though, which is what they're claiming - but not ~99%...