Understanding distillation attacks as a technique is one thing. Seeing them operate at industrial scale, in coordinated campaigns, with the explicit goal of extracting frontier AI capabilities, is something else entirely. That’s what several public disclosures over the past month have described.
Key Takeaways#
- Three Chinese AI labs — DeepSeek, Moonshot AI, and MiniMax — ran coordinated campaigns generating over 16 million exchanges through ~24,000 fraudulent Anthropic accounts; Anthropic attributed each with high confidence
- Distillation copies capabilities but not values: a model trained on well-aligned outputs can achieve frontier performance while retaining none of the original safety constraints
- Google’s Threat Intelligence Group detected a campaign of 100,000+ prompts targeting Gemini’s multilingual reasoning — in real time, while it was happening
- The actual threat model isn’t IP theft — it’s authoritarian governments using distilled frontier AI, stripped of safety alignment, for offensive cyber operations and mass surveillance
- If only the most cautious providers invest in defenses, attackers move to the least-protected target; no single organization can solve this unilaterally
What Has Actually Happened#
On February 23rd, Anthropic disclosed that three Chinese AI laboratories had been running systematic campaigns to extract capabilities from its Claude model. The labs named were DeepSeek, Moonshot AI, and MiniMax. Between them, they generated over 16 million exchanges through roughly 24,000 fraudulent accounts, in deliberate violation of Anthropic’s terms of service and its restrictions on access from China.
The campaigns weren’t opportunistic. Each lab targeted specific capabilities. Moonshot focused on agentic reasoning, tool use, and computer vision, accounting for over 3.4 million exchanges. MiniMax, which accounted for 13 million of those exchanges, concentrated on agentic coding and tool orchestration. When Anthropic launched a new Claude version, MiniMax redirected nearly half its traffic to the updated model within 24 hours. That kind of operational agility doesn’t happen by accident.
Anthropic said it attributed each campaign to a specific company with high confidence, using IP address correlation, request metadata, and infrastructure indicators. The prompts themselves were distinctive: high volume, narrow focus, repetitive structure, directly mapped onto capabilities valuable for AI training.
OpenAI had raised similar concerns earlier in the month, in a memo to the House Select Committee on China dated February 12th. The company stated it had observed accounts associated with DeepSeek employees working to circumvent API access restrictions through obfuscated third-party routers. It also noted that DeepSeek employees had developed code specifically to access and extract outputs from US AI models programmatically.
This followed the controversy that erupted in January 2025 when DeepSeek released its R1 model. The model matched frontier performance on several benchmarks at a fraction of the reported training cost. OpenAI told the Financial Times it had found evidence of distillation. Early testers had noticed something more anecdotal but harder to dismiss: when asked about its origins, DeepSeek’s model had, in some instances, identified itself as ChatGPT. DeepSeek later denied that R1 was trained on OpenAI outputs, and the peer-reviewed version of their R1 paper, published in Nature in September 2025, maintained that position. The question has not been resolved to everyone’s satisfaction.
Google’s Threat Intelligence Group, meanwhile, reported detecting campaigns targeting its Gemini models. One involved over 100,000 prompts designed to replicate Gemini’s reasoning ability in non-English languages across a wide range of tasks. Google said it detected the probe in real time and protected its internal reasoning traces, though it acknowledged that distillation remains extremely difficult to eliminate as a threat category.
The Part That Goes Beyond IP#
When these stories get covered in the business press, the framing tends to be about intellectual property: who built what, who is free-riding on whose research investment, what the legal exposure is. That framing isn’t wrong, but it misses what the security community finds most concerning.
Frontier AI models aren’t just capable. They’re constrained. The safety properties built into models like Claude or GPT are the result of years of deliberate alignment work: mechanisms designed to prevent misuse for bioweapons development, large-scale cyberattacks, targeted disinformation, and similar threats. These guardrails aren’t bolted on at the end. They’re embedded throughout training, and they represent a significant portion of the total development effort.
Distillation copies behavior, not values. A model trained on outputs extracted from a well-aligned system may learn to code, reason, and use tools at a frontier level, while retaining none of the safety properties that governed the original. The student inherits the teacher’s capabilities, but the teacher’s judgment doesn’t transfer.
This matters because the distilled models don’t stay in a research lab. They get integrated into other systems. Some of those systems are commercial. Some aren’t. Anthropic’s disclosure explicitly named military, intelligence, and surveillance applications as the concern: authoritarian governments using frontier AI capabilities, stripped of safety constraints, for offensive cyber operations, disinformation at scale, and mass surveillance infrastructure.
This is the threat model that justifies treating distillation attacks as a national security issue rather than a terms-of-service violation with policy implications.
The Deeper Parallel#
There’s something worth sitting with here for anyone interested in how security failures actually happen.
These attacks didn’t work because someone found a vulnerability. They worked because the systems involved behaved perfectly. The models answered questions. The accounts looked like users. The queries, taken individually, were indistinguishable from legitimate use. The attack was visible only in aggregate, and even then, only if you were watching closely enough to see the pattern.
This is the same structure as many of the social engineering attacks that cause the most damage in traditional security. The most effective approaches don’t trick a system into doing something it should not do. They convince a person, or in this case a model, to do exactly what it is supposed to do, just many more times, with carefully chosen inputs, at the direction of someone the system has no way to identify as an adversary.
Defending against this kind of threat requires a different orientation. You’re not looking for something broken. You are looking for normal behavior that has been weaponized, and that means your detection has to be sensitive to context and pattern rather than to individual events.
What You Can Actually Do About It#
A note before we get into this: distillation attacks are an emerging and fast-moving threat. The defensive landscape is evolving in real time, and guidance that holds today may be outdated or incomplete within weeks. Treat what follows as a starting framework, not a checklist, and revisit it often.
The honest answer is that no single measure stops a determined, well-resourced attacker. But that’s true of most serious threats, and it’s not a reason to do nothing. It’s a reason to think in layers.
💡 If you are exposing an AI model through an API, the first priority is visibility. You can’t defend what you can’t see. Log everything: query volume per account, prompt structure, the domains queries cluster around, the timing and pacing of requests. Distillation campaigns have a fingerprint, and you can only recognize it if you have the telemetry to look for it. Aggregate anomaly detection matters more here than per-request filtering.
💡 Rate limiting is necessary but not sufficient on its own. Pair it with graduated access tiers, where high-volume API usage requires additional verification and carries higher cost. The economics of distillation attacks are sensitive to friction. Making bulk extraction meaningfully expensive without penalizing ordinary users is a design challenge, but it’s a tractable one.
💡 For the accounts themselves, strengthen verification at onboarding for use cases that have legitimate reasons to generate high query volumes, like research institutions and enterprise customers. Flag and investigate accounts that show signs of coordinated behavior: identical request patterns across multiple accounts, traffic routed through the same proxy infrastructure, prompt structures that vary in superficial ways but probe the same narrow capability domain.
💡 At the model level, consider output perturbation for high-sensitivity capabilities. The goal is to degrade the utility of harvested outputs for training without affecting the experience for legitimate users. This is an active area of research and not a solved problem, but some approaches show promise for specific capability domains.
💡 If you are building proprietary capabilities on top of a foundation model, your attack surface is broader than you may have considered. Your fine-tuned model’s outputs are as exposed as the base model’s. Access controls, usage monitoring, and rate limiting apply to your layer of the stack just as much as to the provider’s. Treat your model’s outputs as intellectual property and design your access controls accordingly from the start.
💡 If you are buying AI capabilities, ask your vendors how they detect and respond to distillation attempts. Ask what their incident history looks like. A vendor who has detected and disclosed attacks, as Anthropic and Google have recently done, isn’t a vendor to avoid. It’s a vendor that knows what is happening to their systems. Opacity on this question is the more concerning signal.
💡 Finally, and this is the point that tends to get overlooked: treat this as an ecosystem problem, not just a product problem. OpenAI’s memo to Congress put it plainly. If only the most cautious providers invest in defenses, attackers will move to whichever provider is least protected. Sharing threat intelligence across providers, engaging with cloud infrastructure partners, and supporting policy frameworks that raise the baseline across the industry are all part of the defense. No organization can solve this unilaterally, and the ones that act as if they can are making the problem worse for everyone else.
What This Means in Practice#
For organizations building or buying AI capabilities, a few things follow from all of this.
Proprietary AI capabilities are a target. If you have built something valuable on top of a foundation model, or if you are building your own, the outputs of that system can be used to replicate it. Access controls and usage monitoring aren’t optional considerations for later. They belong in the architecture from the beginning.
The safety properties of the AI you use are part of your security posture. A capable model isn’t the same as a safe one. If you’re evaluating AI vendors, provenance and alignment practices belong in your assessment alongside performance benchmarks.
Competitive intelligence on AI has become unreliable in a new way. When a company releases a model that appears to represent a sudden capability leap, that leap may reflect genuine independent research. It may also reflect systematic extraction from a competitor’s system. The two are genuinely difficult to distinguish from the outside, and the policy response to each is quite different.
The pattern of these attacks, coordinated, sustained, operationally sophisticated, and targeted specifically at the capabilities most valuable for training, should inform how you think about the threat landscape for AI systems more broadly. This isn’t a hypothetical risk. It’s an active one, and the organizations caught flat-footed tend to be the ones who assumed it was someone else’s problem.
At Greymantle Risk Advisory, we help organizations understand and navigate emerging threats at the intersection of AI and security. If you’re thinking through your AI security posture, we’d be glad to talk.