Installing the CA requires jumping through some hoops, but yes, intercepting traffic for apps that don’t use cert pinning isn’t that difficult on iOS.
Apps that do use cert pinning is a whole other matter, I’ve tried unsuccessfully a few times to inspect things like banking apps. Needs a rooted device at the minimum.
ANTI_DISTILLATION_CC
This is Anthropic's anti-distillation defence baked into Claude Code. When enabled, it injects anti_distillation: ['fake_tools'] into every API request, which causes the server to silently slip decoy tool definitions into the model's system prompt. The goal: if someone is scraping Claude Code's API traffic to train a competing model, the poisoned training data makes that distillation attempt less useful.
I was thinking just yesterday that the research that Anthropic was sharing regarding how it's easy to poison training was unlikely to be conducted out of goodness of the heart.
Haven’t looked at the code, but is the server providing the client with a system prompt that it can use, which would contain fake tool definitions when this is enabled? What enables it? And why is the client still functional when it’s giving the server back a system prompt with fake tool definitions? Is the LLM trained to ignore those definitions?
Wonder if they’re also poisoning Sonnet or Opus directly generating simulated agentic conversations.
Claude Code has a server-side anti-distillation opt-in called fake_tools, but the local code does not show the actual mechanism.
The client sometimes sends anti_distillation: ['fake_tools'] in the request body at services/api/claude.ts:301
The client still sends its normal real tools: allTools at services/api/claude.ts:1711
If the model emits a tool name the client does not actually have, the client turns that into No such tool available errors at services/tools/StreamingToolExecutor.ts:77 and services/tools/toolExecution.ts:369
If Anthropic were literally appending extra normal tool definitions to the live tool set, and Claude used them, that would be user-visible breakage.
That leaves a few more plausible possibilities:
Fake_tools is just the name of the server-side experiment, but the implementation is subtler than “append fake tools to the real tool list.”
or
The server may inject tool-looking text into hidden prompt context, with separate hidden instructions not to call it.
or
The server may use decoys only in an internal representation that is useful for poisoning traces/training data but not exposed as real executable tools.
We do know that Anthropic has the ability to detect when their models are being distilled, so there could be some backend mechanism that needs to be tripped to observe certain behaviour. Not possible to confirm though.
We can be used to refer to people in general, and we know because Anthropic published a post called "Detecting and preventing distillation attacks" a month ago, while calling out 3 AI labs for large scale distillation
It made me raise my eyebrows when everyone was rushing to jump to Claude because OpenAI agreed to work with the DoW. Both companies are just as shitty as each other and will resort to underhanded tactics to stay on top.
Go China to be honest. They're the most committed to open AI research and they have more interesting constraints to work under, like restricted access to NVIDIA hardware.
You're perfectly free to scrape the web yourself and train your own model. You're not free to let Anthropic do that work for you, because they don't want you to, because it cost them a lot of time and money and secret sauce presumably filtering it for quality and other stuff.
Stole? Courts have ruled it's transformative, and it very obviously is.
AI doomerism is exhausting, and I don't even use AI that much, it's just annoying to see people who want to find any reason they can to moan.
> Stole? Courts have ruled it's transformative, and it very obviously is.
The courts have ruled that AI outputs are not copyrightable. The courts have also ruled that scraping by itself is not illegal, only maybe against a Terms of Service. Therefore, Anthropic, OpenAI, Google, etc. have no legal claim to any proprietary protections of their model outputs.
So we have two things that are true:
1) Anthropic (certainly) violated numerous TOS by scraping all of the internet, not just public content.
2) Scraping Anthropic's model outputs is no different than what Anthropic already did. Only a TOS violation.
Nobody is saying they can't try to stop you themselves. That's where the Terms of Service violation part comes in. They can cancel your account, block your IP, etc. They just can't legally stop you by, for instance, compelling a judge to order you to stop.
The Supreme Court already ruled on this. Scraping public data, or data that you are authorized to access, is not a violation of the Computer Fraud and Abuse Act.
Now, if you try to get around attempts to block your access, then yes you could be in legal trouble. But that's not what is happening here. These are people/companies that have Claude accounts in good standing and are authorized by Anthropic to access the data.
Nobody is saying that Anthropic can't just block them though, and they are certainly trying.
> You're perfectly free to scrape the web yourself and train your own model.
Actually, not anymore as a result of OpenAI and Anthropic's scraping. For example, Reddit came down hard on access to their APIs as a response to ChatGPT's release and the news that LLMs were built atop of scraping the open web. Most of the web today is not as open as before as a result of scraping for LLM data. So, no, no one is perfectly free to scrape the web anymore because open access is dying.
Rich people aren't going to find themselves needing to sleep under a bridge, so the law really only exists as a constraint on the poor. Duh. The flex that "well a rich guy couldn't do it either" is A) at best a myopic misunderstanding perpetuated by out of touch people and B) hopelessly naive, because anny punishment for the rich guy actually sleeping under a bridge is so laughably small it may as well not even exist. Hence, the whole bit of "a legal system to keep these accountable, but not for me".
Okay, you explained what Anatole France meant, which is probably helpful for those few who didn't get it from the quote itself. Perhaps now you can explain what on earth this has to do with Anthropic not wanting to let other for-profit businesses mooch off its investment of time, brainpower and money?
You explained what “rich and poor are equally forbidden from sleeping under bridges” means, but not what this has to do with the statement that one is free to do their own scraping and training, which I’m pretty sure is what kspacewalk was asking.
Try this: If you want to train a model, you’re free to write your own books and websites to feed into it. You’re not free to let others do that work for you because they don’t want you to, because it cost them a lot of time and money and secret sauce presumably filtering it for quality and other stuff.
I introspect all the time. I just disagree with you so I have thin skin? Lol.
I think it's transformative. I also think that it's a net positive for society. I lastly think that using freely available, public information is totally fair game. Piracy not so much, but it's water under the bridge.
I hope you introspect some day, too, and realize it's acceptable for people to have different views than you. That's why I don't care; you aren't going to change my mind and I can't change yours either, so it's moot and I don't care to argue about it further.
You had appeared to scuttle off but alas I was wrong (and sorry to imply you are a crab of some sort) however your comment followup on not changing minds might be a tad shell-ish. I'm open minded actually on the issue and these are major issues of our time. I'm personally impacted by this and it does make me wonder "will I write X thing again" and it is a very hard question to answer frankly. When you see your works presented in summary on search and a major decline in traffic you really do think about that. It impacts my ability to make money as I once did prior to 2024 (when it really hit) without doubt. Edit/spelling
Your selective respect for work is a glaring double standard. The effort to produce the original content they scraped is order of magnitudes bigger than what it took to train the model, so if this wasn't enough to protect the authors from Anthropic it shouldn't be enough to protected Anthropic from people distillating their models.
Your legal argument is all over the place as well. What is more relevant here: what the courts ruled or what you consider obvious? How is distillation less transformative than scraping? How does courts ruling that scraping to train models is legal relate to distillation?
Nobody is scoring you on neutrality points for not using AI much and calling this doomerism is just a thought-terminating cliche that refuses to engage with the comment you're replying.
In fact, your comment is not engaging with anything at all, you're vaguely gesturing towards potentitial arguments without making them. If you find discussing this exhausting then don't but also don't flood the comments with low effort whining.
reminds me of `don't look up` a bit. there clearly is an imbalance in regards to licenses with model providers, not even talking about knowledge extraction (yes younger people don't learn properly now, older generations forget) shortly before the rug-pull happens in form of accessibility to not rich people
Let's talk ethics, not law. Why is it okay for these companies to pirate books and scrape the entire web and offer synthesized summaries of all of it, lowering traffic and revenue for countless websites and professions of experts, but it is not okay for others to try to do the same to an AI model?
Is the work of others less valid than the work of a model?
>Why is it okay for these companies to pirate books
Courts have ruled it's not, and I don't think anyone is arguing it's okay.
>but it is not okay for others to try to do the same to an AI model?
The steelman version is that it's okay to do it once you acquired the data somehow, but that doesn't mean anthropic can't set up roadblocks to frustrate you.
I don't think anyone's saying it's not okay - I think the point is that Anthropic has every right to create safeguards against it if they want to - just like the people publishing other information are free to do the same.
And everyone is free to consume all the free information.
Settled out of court does not mean the lawsuit never went to court. It means the settlement happened outside of court. Every lawsuit has to go to court, that's how you file a lawsuit. If it isn't sent to a court it's just words in a document.
It's not really paranoia if it's happening a lot. They wrote a blog post calling several major Chinese AI companies out for distillation.[0] Perhaps it is ironic, but it's within their rights to protect their business, like how they prohibit using Claude Code to make your own Claude Code.[1]
Their business shouldn't exist. It was predisposed on non-permissive IP theft. They may have found a judge willing to cop to it not being so, but the rest of the public knows the real score. And most problematically for them, that means the subset of hackerdom that lives by tit-for-tat. One should beware of pissing off gray-hats. Iit's a surefire way to find yourself heading for bad times.
I would say not all that ironic. Book publishers, Reddit, Stackoverflow, etc., tried their best to attract customers while not letting others steal their work. Now Anthropic is doing the same.
Unfortunately (for the publishers, at least) it didn't work to stop Anthropic and Anthropic's attempts to prevent others will not work either; there has been much distillation already.
The problem of letting humans read your work but not bots is just impossible to solve perfectly. The more you restrict bots, the more you end up restricting humans, and those humans will go use a competitor when they become pissed off.
It's really just tech culture like HN that obsesses over solving problems perfectly. From seat belts to DRM to deodorant, most of the world is satisfied with mitigating problems.
No, it's ethical people pointing out that if you toss aside ethics for success at all costs, you aren't going to find any sympathy when people start doing the same thing back to you. Live by the sword, die by the sword, as they say.
There is a reason we don't do things. That reason is it makes the world a worse place for everyone. If you are so incredibly out of touch with any semblance of ethics at all; mayhaps you are just a little bit part of the problem.
The funny thing about ethics is there is no absolute, which makes some people uncomfortable. Is it ethical to slice someone with a knife? Does it depend if you're a surgeon or not?
Absolutism + reductionism leads to this kind of nonsense. It is possible that people can disagree about (re)use of culture, including music and print. Therefore it is possible for nuance and context to matter.
Life is a lot easier if you subscribe to a "anyone who disagrees with me on any topic must have no ethics whatsoever and is a BAD person." But it's really not an especially mature worldview.
Categorical imperative and Golden Rule, or as you may know it from game theory "tit-for-tat" says "hi". The beautiful thing about ethics is that we philosophers intentionally teach it descriptively, but encourage one to choose their own based on context invariance. What this does is create an effective litmus test for detecting shitty people/behavior. You grasping on for dear life to "there's no absolutes" is an act of self-soothing on your own part as you're trying to rationalize your own behavior to provide an ego crumple zone. I, on the other hand, don't intend to leave you that option. That you're having to do it is a Neon sign of your own unethicality in this matter. We get to have nice things when people moderate themselves (we tolerate eventual free access to everything as long as the people who don't want to pay for it don't go and try to replace us economically at scale). When people abuse that, (scrape the Internet, try to sell work product in a way that jeopardizes the environment we create in) the nice thing starts going away, and you've made the world worse.
Welcome to life bucko. Stop being a shitty person and get with the program so we have something to leave behind that has a chance of not making us villains in the eyes of those we eventually leave behind. The trick is doing things the harder way because it's the right way to do it. Not doing it the wrong way because you're pretty sure you can get away with it.
But you're already ethically compromised, so I don't really expect this to do any good except to maybe make the part of you you pointedly ignore start to stir assuming you haven't completely given yourself up to a life of ne'er-do-wellry. Enjoy the enantidromia. Failing that, karma's a bitch.
Capitalism is always underpinned by a strong legal system which is why most criticism is about constraining growth in legislation, not killing off interference outright. Copyright law is a good example of a law that made sense in it's original form but turned into a monster with scope-creep.
Although, if we're being realpolitik, every time government interference grows in scope and corrupts markets, capitalism still gets blamed and people call for more government to fix it (see: housing). So the capitalism vs state capitalism distinction isn't very meaningful in practice.
I watched a talk from Bjarne Stroustrup at CppCon about safety and it was pretty second hand embarrassing watching him try to pretend C++ has always been safe and safety mattered all along to them before Rust came along.
Well, there has been a long campaign against manual memory management - well before Rust was a thing. And along with that, a push for less use of raw pointers, less index loops etc. - all measures which, when adopted, reduce memory safety hazards significantly. Following the Core Guideliness also helps, as does using span's. Compiler warnings has improved, as has static analysis, also in a long process preceding Rust.
Of course, this is not completely guaranteed safety - but safety has certainly mattered.
Yes, this what Stroustrup said and it makes me laugh. IIRC he phrased with a more of a 'we had safety before Rust' attitude. It also misses the point, safety shouldn't be opt-in or require memorising a rulebook. If safety is that easy in C++ why is everyone still sticking their hand in the shredder?
You're "moving the goal posts" of this thread. Safety has mattered - in C++ and in other languages as well, e.g. with MISRA C.
As for the Core Guidelines - most of them are not about safety; and - they are not to be memorized, but a resource to consult when relevant, and something to base static analysis on.
Because they don’t work with it. It’s a simple as that. I don’t trust people who don’t work with a terminal these days, the further they get from a terminal, the less grounded their views are. They rely on hearsay and CEO hype. To make matters worse, they say whatever they think will earn them a bonus/promotion, which leads to a cascade of BS down the chain.
I seriously doubt Satya Nadella is sitting down for hours a day to use Copilot to draft detailed documents. He's being fed fantastical stories by his lackeys telling him what he wants to hear.
I've seen this misconception so many times in open source projects - commits just bumping the version in go.mod to 'get the latest performance and security improvements.' Like no, that's not how it works, you just made your code compile with fewer compiler versions for no reason.
I think the directive could have been named better though, maybe something like min_version.
Yes, the general trend is the unprecedented growth that we've seen. Typically one would have some time in advance to re-engineer the systems to support the increased in traffic and users. But we're dealing with very compressed timelines and while most of the time we're able to fix the issues beforehand, sometimes we have to do them in production. Sorry for that.
The US under Trump is behaving exactly like a country with intentions of damaging the Western order and antagonising enemies to open new front lines. I think writing off Trump's actions as stupid is wrong, he's malicious.
GitHub, npm, PyPi, and other package registries should consider exposing a firehose to allow people to do realtime security analysis of events. There are definitely scanners that would have caught this attack immediately, they just need a way to be informed of updates.
It is not effective if it just takes a simple base64 encode to bypass. If Claude is trivially able to find that it is malicious then Pypi is being negligent.
The package in question was live for 46 minutes. It generally takes longer than that for security partners to scan and flag packages.
PyPI doesn't block package uploads awaiting security scanning - that would be a bad idea for a number of reasons, most notably (in my opinion) that it would be making promises that PyPI couldn't keep and lull people into a false sense of security.
It should not let people download unscanned dependencies without a warning and asking the user to override and use a potentially insecure package. If such security bug is critical enough to need to bypass this time (spoiler: realistically it is not actually that bad for a security fix to be delayed) they can work with the pypi security team to do a quicker manual review of the change.
The whole point is that this would give a false sense of security. Scanned dependencies aren't secure, they're just scanned by some tools which might catch some issues. If you care about security, you need to run those same scans on your side, perhaps with many more rules enabled, perhaps with multiple tools. PyPI, understandably, does NOT want to take any steps to make it seem like they promise their repo doesn't contain any malware. They make various best effort attempts to keep it that way, but the responsibility ultimately falls on you, not on them.
sadly I still worry about that. An install fails once, you you hard code the --force flag in all your CI/CD jobs and we are back in the same place again. I am not sure what the answer is, though problems...
Adding a hardcoded flag is not the same as asking the user if they want potential malware. If CI/CD is broken they should revert the change to pinned dependencies instead of trying to install a bleeding edge version of a new dependency that hasn't been scanned yet.
I don't understand why this would be an issue. Firstly, you could just pin your dependencies, but even if you don't, couldn't the default behaviour be to just install the newest scanned version?
Then it will be downloadable and then it's up to your own security scanners to catch it. If you find it, it should be reported to pypi and then the scanner should be improved to catch that kind of bypass the next time it comes around. In such a world I don't think pypi is acting negligent.
That's really not very different from what we have right now. PyPI works with scanners which catch a whole lot of malware and are getting better all the time.
I think PyPI suggesting that software is safe would be a step down from this because it make promises that PyPI can't keep, and would encourage a false sense of security.
It's less about suggesting that it's safe, and more about avoiding pushing out arbitrary code to thousands of people without checking if you are pushing out malicious code to all of those people. It is the responsible thing to do.
>That's really not very different from what we have right now.
What I'm advocating for is different enough to have stopped this malware from being pushed out to a bunch of people which at the very least would raise the bar of pulling off such an attack.
I realize this is controversial (and many Python folks would claim anti ethical). But I keep wondering if requiring a small payment for registering and updating packages would help. The money could go to maintaining pypix as well as automated AI analysis. Folks who really couldn't afford it could apply for sponsorship.
Very much not speaking for the PSF here, but my personal opinion on why that wouldn't work is that Python is a global language and collecting fees on a global basis is inherently difficult - and we don't want to discriminate against people in countries where the payment infrastructure is hard to support.
Plus a small fee wouldn't deter malware authors, who would likely have easy access to stolen credit cards - which would expose PyPI to the chargebacks and fraudulent transactions world as well!
If pypi charges money, python libraries will suddenly have a lot of "you can 'uv add git+https://github.com/project/library'" instead of 'uv add library'.
I also don't think it would stop this attack, where a token was stolen.
If someone's generating pypi package releases from CI, they're going to register a credit card on their account, make it so CI can automatically charge it, and when the CI token is stolen it can push an update on the real package owner's dime, not the attackers, so it's not a deterrent.
Also, the iOS app store is an okay counter example. It charges $100/year for a developer account, but still has its share of malware (certainly more than the totally free debian software repository).
Not speaking on behalf of PSF, but to me, it looks like a no-go, as some packages are maintained, legitimately, by people from sanctioned countries, with no way to pay any amount outside their country.
I don't see how this would help in the least, what kind of criminal would be dissuaded by paying a small fee to set an elaborate scheme such as this in motion? This is not a spamming attack where the sheer volume would be costly. It doesn't even help to get a credit card on file, since they can use stolen CC numbers.
It's far more likely that hobbyists will be hurt than someone that can just write off the cost as a small expense for their criminal scheme.
I suspect that for a nation-state type threat actor, this wouldn’t be much of a deterrent. Any type of reputation system like this would work to a point until motivated threat actors find a way to game it.
Would you happen to know where the latency comes from between upload and scanning? Would more resources for more security scanner runners to consume the scanner queue faster solve this? Trying to understand if there are inherent process limitations or if a donation for this compute would solve this gap.
(software supply chain security is a component of my work)
He said, "pypi doesn't block upload on scanning"; that's part of where the latency comes from. The other part is simply the sheer mass of uploads, and that there's not money in doing it super quickly.
I agree that's a bad idea to do so since security scanning is inherently a cat and mouse game.
Let's hypothetically say pypi did block upload on passing a security scan. The attacker now simply creates their own pypi test package ahead of time, uploads sample malicious payloads with additional layers of obfuscation until one passes the scan, and then uses that payload in the real attack.
Pypi would also probably open source any security scanning code it adds as part of upload (as it should), so the attacker could even just do it locally.
I suppose my argument is that pypi could offer the option to block downloads to package owners until a security scan is complete (if scanning will always take ~45-60 minutes), and if money is a problem, money can solve the scanning latency. Our org scans all packages ingested into artifact storage and requires dependency pinning, and would continue to do so, but more options (when cheap) are sometimes better imho. Also, not everyone has enterprise resources for managing this risk. I agree it is "cat and mouse" or "whack-a-mole", and always will be (ie building and maintaining systems of risk mitigation and reduction). We don't not do security scanning simply because adversaries are always improving, right? We collectively slow attackers down, when possible.
So I've been thinking about this a lot since it happened. I've already added dependency cooldowns https://nesbitt.io/2026/03/04/package-managers-need-to-cool-... to every part of our monorepo. The obvious next thought is "am I just dumping the responsibility onto the next person along"? But as you point out it just needs to give automated scanners enough time to pick up on obvious signs like the .pth file in this case.
It is in a sense dumping responsibility, but there’s a legion of security companies out there scanning for attacks all the time now to prove their products. They’re kind of doing a public service and you’re giving them a chance to catch attacks first. This is why I think dep cooldowns are great.
I feel like they should be legally responsible for providing scanning infrastructure for this sort of thing. The potential economic damage can be catastrophic. I don't think this is the end of the litellm story either, given that 47k+ people were infected.
It's not about expectation of work (well, there's some entitled people sure.)
It's about throwing away the effort the reporter put into filing the issue. Stale bots disincentivise good quality issues, makes them less discoverable, and creates the burden of having to collate discussion across N previously raised issues about the same thing.
Bug reports and FRs are also a form of work. They might have a selfish motive, but they're still raised with the intention of enriching the software in some way.
Apps that do use cert pinning is a whole other matter, I’ve tried unsuccessfully a few times to inspect things like banking apps. Needs a rooted device at the minimum.
reply