There’s a category of attack that leaves no forensic trace, triggers no intrusion alert, and requires no vulnerability to exploit. The target system performs exactly as designed throughout. By the time anyone notices something is wrong, the damage is done and largely irreversible.
It’s called a distillation attack. And if you work in or adjacent to AI, it’s worth understanding in some detail.
Key Takeaways#
- Distillation attacks let competitors clone a proprietary AI model through normal API use — no breach, no exploit, no forensic trace; the target system performs exactly as designed throughout
- The “vulnerability” is the product’s core function: answering questions. You cannot fix it through conventional hardening
- Attackers run structured, automated query campaigns — sometimes millions of exchanges — targeting specific capability domains, then use those outputs to fine-tune an open-source base model
- Rate limiting slows the attack but doesn’t stop it; sophisticated attackers distribute traffic across proxy networks and vary prompt structure to stay below detection thresholds
- The realistic goal is making the attack more expensive than it’s worth — not preventing it entirely
Start with the Legitimate Version#
Knowledge distillation is a standard machine learning technique. The idea is straightforward: you have a large, expensive model that performs well, and you want a smaller, cheaper version that behaves similarly. Rather than training the smaller model from scratch, you use the large model as a teacher. You feed it thousands of questions, collect its answers, and train the smaller model on those input-output pairs. The student learns to mimic the teacher’s behavior without ever seeing the teacher’s internal weights.
AI labs use this constantly. It’s how frontier models get compressed into versions that run on modest hardware. Google does it. Anthropic does it. OpenAI does it. There’s nothing inherently suspicious about it, which is precisely what makes it useful as an attack vector.
The Attack#
The attack follows the same basic process, applied to someone else’s model, at a scale they never intended to enable.
An attacker creates API access to a target model, usually through a network of accounts designed to avoid detection. From there, they run structured, automated query campaigns. These are not random questions. The prompts are carefully engineered to probe specific capabilities, and they arrive in high volume, targeting narrow domains: coding, reasoning, tool use, multilingual tasks, whatever the attacker most wants to replicate.
Every response is logged. After tens of thousands, or millions, of exchanges, the attacker has a high-quality synthetic dataset that maps the target model’s behavior across the domains they care about. That dataset is then used to fine-tune an open-source base model or augment their own training pipeline.
The result is a model that performs comparably to the target on the tasks that matter, built at a fraction of the original development cost. The target model never misbehaved. No security control was bypassed. The attack surface was the product’s core function: answering questions.
One security analysis framed the problem bluntly: an LLM that refuses to answer prompts is useless. You cannot fix this through conventional hardening, because the “vulnerability” is the feature.
What Makes It Hard to Catch#
The volume and structure of a distillation campaign is what distinguishes it from legitimate use, but even that signal is hard to act on cleanly. A legitimate researcher might send thousands of coding questions. A distillation campaign looks similar at first glance, just more systematic and more concentrated.
Defenders are looking for patterns: prompts that repeat with minor variations, traffic that clusters tightly around specific capability domains, accounts that behave in lockstep. Sophisticated attackers know this, and they adapt. They distribute traffic across proxy networks, vary prompt structure, and pace their queries to stay below detection thresholds.
Rate limiting slows the attack but penalizes legitimate users. Enhanced verification raises the barrier but determined adversaries route around it through front companies and fake credentials. Detection is possible, but it’s a continuous effort against an adversary who has every incentive to stay one step ahead.
The Defender’s Economics#
There’s a useful way to think about this that cuts through the complexity. You can’t prevent distillation without degrading the product. So the actual goal is to make the attack more expensive than it is worth.
This means rate limiting to slow data collection, pricing tiers that make high-volume access economically prohibitive, and behavioral fingerprinting to identify and disrupt campaigns in progress. It means monitoring for coordinated account activity, detecting chain-of-thought elicitation attempts, and sharing threat intelligence across the industry so that hardening one provider actually hardens the ecosystem rather than just redirecting attackers to the next target.
That last point is the one that organizations tend to underestimate. If only the most cautious providers invest in defenses, attackers simply move to whichever provider is least protected. The economics of distillation attacks are such that the weakest link in the provider ecosystem sets the effective ceiling for what can be stolen.
In part two of this series, we look at where this has played out in practice, including recent disclosures from Anthropic, OpenAI, and Google, and what the security implications are beyond intellectual property.
At Greymantle Risk Advisory, we help organizations understand and navigate emerging security threats. If you’re thinking through your AI security posture, we’d be glad to talk.