I was wondering today why Claude was rushing ahead, making decisions without asking me. Asking surfaced this snippet in the system prompt: "The user is not watching in real time and cannot answer questions."
I was a little floored because this is flat-out incorrect most of the time (for me at least) and sounds like it would encourage reckless behavior.
I get that some people want the model to behave more autonomously, but this should be something that is surfaced to the user, not casually placed in the system prompt.
IMO this is one of the worst system-prompt crimes since the "never write comments" line (that has since been removed, as far as I know).
This reinforces my suspicion that alignment and training in general is closer to being a pedagogical problem than anything else. Given a finite amount of training input, how do we elicit the desired model behavior? I’m not sure if asking educators is the right answer, but it’s one place to start.
It's a weird new thing. You might call it "AI psychology".
The problem with cribbing from education is that what "educators" do to humans doesn't apply to AIs cleanly. And it's not like "human alignment" is anywhere near a solved problem.
A big part of the bet USSR made was that human flaws like selfishness and greed could be educated out of population. The result was: a resounding failure. Even state-level efforts fail to robustly "align" human behavior.
With AI, we have a lot more control over behavior, but that control just isn't very human-shaped. A lot of the practical methods in play seem closer to esoterics than to math, but they're not the kind of methods that are used in human education. You can teach humans by talking to them. You can't teach humans through soul data self-distillation.
That's basically what the GOFAI field was for decades before the new neural net boom. Go read Minsky's Society of Mind, or the AGI Conference series papers.
you mean completely wrong, spread a problematic understanding psychology, and delay real progress for decades because smart people spend fruitless years trying to find a use for it.
...I think we might already have those people running AI companies.
I’ve also found that compiling large packages in GCC or similar tends to surface problems with the system’s RAM. Which probably means most typical software is resilient to a bit-flip; makes you wonder how many typos in actual documents might have been caused by bad R@M.
That's exactly how my bad RAM manifested itself. In fact, I was compiling Firefox, and gcc would get a segmentation fault at some random point during compilation. I'd have to clobber and restart the hour-long build. It was only when gcc started crashing while compiling other things that I even started considering the possibility of hardware failure. I'm a software developer, and based on what I produce myself, I just assume that all software is horribly buggy. ;-)
I’ve been doing this for a few months now (rolled my own setup with Claude Code) and it’s totally changed the way I manage my portfolio and retirement plan. I mean yes, this is something that could technically have been set up in Excel but who has the time and patience to sit around fiddling with formulas to make an accurate financial forecast?
The cherry on top is that, obviously, you can then ask Claude for thoughts on the resulting analyses and hopefully save yourself from making bad decisions.
Same! Not an Excel pro and also have zero interest in going deep at Excel at all here.
Seems like you've been running this setup for quite some time. Curious to know if your setup is similar to how I did it, or do you have a slighly different config? Do you also use json files for your data and just let Claude do all the magic?
Yes, my setup is similar, probably because that’s what Claude drifts towards by default, and in this case I didn’t want to impose my will on it much since it’s a simple problem that doesn’t need to be over-engineered.
I was a bit dubious until I read the gist. I've used a similar technique before to 'tame' GPT-3.5 and keep it following instructions and it worked well (though I had to ask the model to essentially repeat instructions after every 10 or so turns). I'm surprised you see that much drift though; older models were pretty bad with long contexts but I feel like the problem has mostly gone away with Claude Opus 4.6.
Glad it resonated! Yeah repeating instructions every N turns was the old approach — SCAN basically does the same thing but with ~20 tokens instead of the full prompt each time.
On drift being "mostly gone" — depends on prompt complexity. With a simple system prompt, sure, modern models hold up fine. But with a large instruction set (mine is ~4000 tokens, 25+ rules across 7
sections) the drift is very much still there, even on Opus. The more rules you have, the more they compete for attention, and the easier it is for specific ones to drop off mid-session.
Also worth noting — this isn't limited to coding agents. Any long-running LLM workflow with complex instructions has the same problem. Customer support bots that forget their tone policy, medical assistants
that stop citing sources, content moderation that gets lenient over time. If you have a system prompt with rules and a session longer than 20 minutes — the rules will decay.
Interesting. I've been coping by being very conservative about how many rules I introduce into the context, but if what you're saying is true, then something like SCAN actually helps the models break past the "total rule count" barrier by giving them something like "cognitive scaffolding". I'm eager to try this out. Thanks again for sharing!
That's a great way to put it — "cognitive scaffolding" is exactly what it is. And yeah, keeping rules minimal is smart, but at some point the project just needs 25 rules and you can't cut them down without
losing something important. SCAN lets you have a large instruction set without paying the full attention cost. Let me know how it goes!
Sounds interesting. What makes DeBERTA + RAG any better than detecting contradictions in the context than a frontier LLM, and why? I see that the NLI scorer itself was evaluated, but I’d love to see data about how the full system performs vs SotA if you have any on hand.
@soletta Great question — this is exactly why we built it this way.
*Short answer*: frontier LLMs are excellent at static self-critique, but terrible for *real-time token-by-token streaming guardrails* because of latency, cost, and lack of persistent custom memory.
*Why DeBERTa + RAG wins here*:
- *Latency*: DeBERTa-v3-base + Rust kernel scores every ~4 tokens in ~220 ms (AggreFact eval). A frontier LLM call (GPT-4o/Claude 3.5) is 400–2000 ms per check. You can’t do that mid-stream without killing UX.
- *Cost*: Frontier self-checking at scale = real money. This runs fully local/offline after the one-time model download.
- *Custom knowledge*: The 0.4× RAG weight pulls from your GroundTruthStore (ChromaDB). Frontier models don’t have a live, updatable external fact base unless you keep stuffing context (expensive + context-window limited).
- *Determinism & auditability*: Small fine-tunable NLI model + fixed vector DB = reproducible decisions. LLMs-as-judges are stochastic and hard to debug in prod.
We’re completely transparent: the NLI scorer alone is *not SOTA* (66.2% balanced acc on LLM-AggreFact 29k samples — see full table vs MiniCheck/Bespoke/HHEM in the repo). The value is the live system: NLI + user KB + actual streaming halt that no one else ships today.
Full end-to-end comparisons vs. LLM-as-judge in streaming setups are next on the roadmap (happy to run them on any dataset you care about).
Have you tried frontier self-critique in real streaming agents? What broke for you?
I should have been clearer. I'm not talking about making a separate call to the model to ask it to check itself. Any given model essentially is already watching for contradictions all the time as it is generating its output tokens. Frontier models like Claude Opus 4.6 are already exceptionally good at not contradicting themselves as they go. As for not having an external fact base - you could in principle insert content ephemerally into the context that is relevant to the task at hand, though doing this without killing modern prompt caching schemes is challenging.
I saw your benchmarks, what I was asking for is benchmarks of the full system (LLM + the NLI model) vs a frontier LLM on its own. Its fine if you didn't do them, but I think it hurts your case.
@soletta Got it — thanks for the extra clarity, that’s an important distinction.
You’re absolutely right: modern frontier models (Claude 3.5/Opus-class, GPT-4o, etc.) have become extremely good at maintaining internal consistency during autoregressive generation. They rarely contradict themselves within the same response anymore.
Where Director-AI adds unique value is *external grounding + hard enforcement* against a user-owned, persistent knowledge base:
- Your GroundTruthStore (ChromaDB) can be arbitrarily large, versioned, and updated without blowing up context windows or breaking prompt caching.
- The guardrail gives a *hard token-level halt* (Rust kernel severs the stream) instead of “hoping” the model self-corrects in the next few tokens.
- You get full audit logs: exact NLI score + which facts conflicted.
- It lets you pair strong-but-cheaper models (Llama-3.1-70B, Mixtral, local vLLM setups) with enterprise-grade factual reliability.
You’re also correct that we don’t have published head-to-head numbers yet for “frontier LLM alone vs. frontier LLM + Director-AI” on end-to-end hallucination rate in streaming scenarios. The current benchmarks focus on the guardrail component itself (66.2% balanced acc on LLM-AggreFact 29k samples, with full per-dataset breakdown and comparison table vs MiniCheck/Bespoke/HHEM — see README).
That full-system eval is literally next on the roadmap (we’re setting up the scripts this week). If you have a specific domain/dataset where you’d like to see the comparison run, I’d be genuinely happy to do it publicly and share the raw logs/results.
In the meantime, the repo is 100% open (AGPL) — feel free to fork and run your own tests. Would love to hear what you find.
In the same way we’re making a category error in defining prompt injection, the framing of “AI agents” as primarily “intelligent actors” misses the fact that many of them will be endowed with some form of memory, be it specific to that entity or shared, and they should no longer be thought about as simply ephemeral tools.
It is by the juice of Zig that binaries acquire speed, the allocators acquire ownership, the ownership becomes a warning. It is by typography alone I can now turbopuffer is written in zig.
The usual path an engineer takes is to take a complex and slow system and reengineer it into something simple, fast, and wrong. But as far as I can tell from the description in the blog though, it actually works at scale! This feels like a free lunch and I’m wondering what the tradeoff is.
It seems like this is an approach that trades off scale and performance for operational simplicity. They say they only have 1GB of records and they can use a single committer to handle all requests. Failover happens by missing a compare-and-set so there's probably a second of latency to become leader?
This is not to say it's a bad system, but it's very precisely tailored for their needs. If you look at the original Kafka implementation, for instance, it was also very simple and targeted. As you bolt on more use cases and features you lose the simplicity to try and become all things to all people.
> It seems like this is an approach that trades off scale and performance for operational simplicity.
Yes, this is exactly it. Given that turbopuffer itself is built on the idea of object storage + stateless cache, we're all very comfortable dealing with it operationally. This design is enough for our needs and is much easier to be oncall for than adding an entirely new dependency would have been.
> Failover happens by missing a compare-and-set so there's probably a second of latency to become leader?
Conceptually that makes sense. How complicated is it to implement this failover logic in a safe way? If there are two processes, competing for CAS wins, is there not a risk that both will think they're non-leaders and terminate themselves?
7. On failure due to CAS, fail active requests and terminate
The client should have a retry mechanism against the broker (which may include looking up the address again).
From the brokers PoV, it will never fail a CAS until another broker wins a CAS, at which point that other broker is the leader. If it does fail a CAS the client will retry with another broker, which will probably be the leader. The key insight is that the broker reads the file once, it doesn't compete to become leader by re-reading the data and this is OK because of the nature of the data. You could also say that brokers are set up to consider themselves "maybe the leader" until they find out they are not, and losing leadership doesn't lose data.
The mechanism to start brokers is only vaguely discussed, but if a host-unreachable also triggers a new broker there is a neat from-zero scaling property.
This is the hardest part because you can easily end up in a situation like you're describing, or having large portions of clients talking to a server just to have their writes rejected.
Further, this system (as described) scales best when writes are colocated (since it maximizes throughput via buffering). So even just by having a second writer you cut your throughput in ~half if one of them is basically dead.
If you split things up you can just do "merge manifests on conflict" since different writers would be writing to different files and the manifest is just an index, or you can do multiple manifests + compaction. DeltaLake does the latter, so you end up with a bunch of `0000.json`, `0001.json` and to reconstruct the full index you read all of them. You still have conflicts on allocating the json file but that's it, no wasted flushing. And then you can merge as you please. This all gets very complex at this stage I think, compaction becomes the "one writer only" bit, but you can serve reads and writes without compaction.
Everything I’ve experienced with working with models (from GPT-2 to Opus 4.6) broadly supports the claim that they learn a persona. It comes back to the point that haters love to harp on: they are fundamentally trained on completing the text, and most texts they are trained on have continuity of persona.
I was a little floored because this is flat-out incorrect most of the time (for me at least) and sounds like it would encourage reckless behavior.
I get that some people want the model to behave more autonomously, but this should be something that is surfaced to the user, not casually placed in the system prompt.
IMO this is one of the worst system-prompt crimes since the "never write comments" line (that has since been removed, as far as I know).