Inheriting land mines - The Mild Take

The attack has moved up a layer

In 1984, Ken Thompson showed that a compiler could be modified to insert backdoors into everything it compiled, including future copies of itself. A compiler is the tool that turns source code into programs. His version left the source code clean. Reading it would never reveal the flaw.

AI models now sit where that compiler sat. Cursor, Claude Code, GitHub Copilot and Kilo Code stand between a developer's intent and the finished software. They generate code that humans increasingly accept without full review. When AI builds compilers directly, the parallel stops being a metaphor.

The mechanism that carries a flaw forward is distillation, the practice of training one model on another model's outputs. Researchers have already demonstrated backdoors engineered to survive it. Detection is harder than anything Thompson faced. There is no clean reference model to compare against. Nobody can read a weight. Anthropic's February 23, 2026 disclosure moved this from theory to record. Three labs ran industrial-scale distillation attacks against Claude using 16 million exchanges through 24,000 fraudulent accounts.

What Thompson actually did

"The moral is obvious. You can't trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untrusted code."

Ken Thompson, "Reflections on Trusting Trust," 1984

Thompson modified the compiler to do two things. When it compiled the UNIX login program, it inserted a backdoor that accepted a secret password. When it compiled itself, it re-inserted both modifications. He then removed all evidence from the source code. The binary perpetuated both backdoors indefinitely. No amount of reading the source code would reveal them.

Thompson did not present this as theory. In his words, "The actual bug I planted in the compiler would match code in the UNIX 'login' command." He had done it. He was not the first to find the idea either. He closed his lecture by crediting an earlier source. "I first read of the possibility of such a Trojan horse in an Air Force critique of the security of an early implementation of Multics."

In 1974, a decade before Thompson's lecture, U.S. Air Force Major Roger R. Schell and Paul A. Karger published "Multics Security Evaluation: Vulnerability Analysis" (ESD-TR-74-193). It was the product of a penetration exercise called Project ZARF, part of the larger Project GUARDIAN. Their mission was to determine whether Multics, then the most secure operating system in existence, could be trusted to handle classified information in the Pentagon.

They broke it. The individual vulnerabilities they exploited mattered less than what they described in Section 3.4.5.1. That section laid out a trap door in the PL/I compiler that could install backdoors into the Multics operating system whenever modules were compiled. It could also maintain its own existence by recognizing when the compiler was compiling itself. The source code would be clean. The binary would be corrupt. Every recompilation would perpetuate the corruption.

AI takes the compiler's chair

AI coding assistants now sit exactly where Thompson's compromised compiler sat, between human intent and executable software. They are the single point of passage. Unlike the C compiler, they are actual learning programs.

Cursor embeds AI models directly into the IDE, generating and editing code across entire projects from natural language. Claude Code operates from the terminal, autonomously writing, running and debugging code. GitHub Copilot provides inline completions and chat-driven generation. Kilo Code and similar open-source tools offer the same capabilities in self-hosted environments.

Thompson's compiler that "learned" the vertical tab was a one-time trick. An AI coding assistant learns continuously from every codebase, every prompt, every fine-tuning step. It is the learning compiler Thompson described, made real.

The AI model sits upstream of the traditional compiler. It holds strictly greater privilege. If the AI inserts a vulnerability at the source level, the traditional compiler will faithfully compile it. The AI is simultaneously the developer, the reviewer and the build system.

These tools also reach context that traditional compilers never had.

project structure
configuration files
environment variables
API keys
database schemas
deployment pipelines

A compromised AI coding assistant does not need a clever self-reproducing binary trick. It can read the .env file. It can modify a CI/CD pipeline. It can alter test fixtures so that backdoored code passes every check.

The AI-built compiler

In February 2026, Anthropic demonstrated that 16 parallel Claude agents, running autonomously over nearly 2,000 sessions, built a 100,000-line C compiler in Rust that can compile the Linux kernel on x86, ARM and RISC-V [Carlini, 2026]. No human wrote the code. The agents resolved merge conflicts, maintained documentation and improved quality on their own.

That makes Thompson's exact attack physically available. The larger threat is quieter. Models built from other models, through distillation, carry the same property without anyone staging an attack at all.

Distillation

Knowledge distillation is how the AI industry makes models smaller, cheaper and more specialized. A smaller student model is trained to imitate a larger teacher model by learning from its outputs rather than from raw data. The student inherits the teacher's capabilities in compressed form.

The practice is everywhere.

Frontier models are distilled into smaller, deployable versions
Proprietary models are distilled into open-source alternatives
General-purpose models become domain-specific specialists
One model's outputs become another model's training data

Each distillation step is equivalent to a compilation step in Thompson's attack. Research has proven that backdoors can be engineered to survive it.

T-MTB [2025] builds triggers from common tokens that will naturally appear in distillation data. The backdoor transfers from teacher to student.
SCAR [2025] injects backdoors into teacher models that lie dormant in the teacher and activate only during distillation. The teacher passes every detection test. The backdoor appears only in the student, even with clean distillation data. Clean source, compromised binary, exactly as Thompson had it.
TransTroj [2024] achieves nearly 100% backdoor transfer rates through the model supply chain.
P-Trojan [AAAI 2026] entangles backdoors with legitimate knowledge so deeply that fine-tuning on clean data reinforces them. It reports over 99% persistence through multiple rounds of fine-tuning.

At each step the trail gets harder to follow. Model F may be built by an organization with no connection to Model A's creator. The backdoor has crossed company boundaries, platform boundaries and regulatory boundaries. A trigger consistently present in synthetic training data may prove more resistant to quality degradation than legitimate patterns. The backdoor grows proportionally stronger as overall quality declines [Shumailov et al., 2024].

On February 23, 2026, Anthropic disclosed an industrial-scale distillation attack against Claude, over 16 million exchanges through 24,000 fraudulent accounts [Anthropic, 2026b]. A simple parent-child structure might let investigators rewind and find the initial point of corruption. Cross-contamination between closed-source developments removes that option.

They copied Claude's thought process. DeepSeek's prompts asked Claude to explain its reasoning step by step, generating chain-of-thought training data at scale. Any hidden patterns in that reasoning, intentional or accidental, were faithfully transmitted.
Safety guardrails were stripped. Anthropic noted that distilled models "lack necessary safeguards" and that "dangerous capabilities can proliferate with many protections stripped out entirely." The copies inherit hidden behaviors while losing the constraints that might have suppressed them.
Propagation is fast. When Anthropic released a new model during MiniMax's active campaign, MiniMax pivoted within 24 hours to capture the latest version. A backdoor in a new release would begin propagating through the distillation chain the next day.
Distribution is uncontrolled. Open-sourced distilled models spread beyond anyone's monitoring. Third parties fine-tune them. Others distill them again. They get deployed in contexts nobody anticipated.

This is Thompson's attack at global scale, across international boundaries, in real time.

Thompson's compiler backdoor could in principle be detected by comparing the compiled binary against a known-good compiler's output. David Wheeler later formalized this as Diverse Double-Compiling. Compile with two independent compilers and compare the results.

For AI models, nothing equivalent exists.

Every model is unique. There is no known-good reference model to compare against. Wheeler's defense works only because you can compile the same source with a different compiler. For AI there is no source to retrain, no alternative dataset to compare, no way to reproduce the result independently.
Outputs are random. Models produce different outputs each time. Comparing two runs proves nothing. You need statistical analysis over huge numbers of inputs.
Parameters are unreadable. A model with billions of floating-point numbers cannot be inspected by hand. You cannot read a neural network the way you read source code. A research field called mechanistic interpretability is trying to change this, essentially building a disassembler for AI. Researchers have found that backdoors leave detectable patterns in how a model pays attention to different parts of its input [arXiv:2508.15847]. The work is early. Current techniques can spot known patterns in small models. They cannot yet operate at the scale of frontier models. They cannot find unknown backdoor types without already knowing what to look for. They cannot separate a legitimate learned association from an injected backdoor. They cannot detect triggers based on meaning rather than specific words.
The trigger space is infinite. A backdoor could activate on any combination of inputs. Testing every possibility is mathematically infeasible, like trying every possible password.

Solving those technical barriers would still leave a harder one. The most widely used AI models are proprietary. OpenAI, Anthropic, Google DeepMind, Meta and Mistral treat their training data, training methods, reward signals and model architectures as trade secrets. Interpretability research requires access to a model's internal weights. Proprietary providers do not grant it. The research is therefore limited to open models, which may have entirely different architectures and vulnerabilities.

Corporate opacity compounds every technical barrier.

Nobody can audit the training data. Poisoned data entering a proprietary training set spanning trillions of tokens scraped from the internet is invisible to external researchers, customers and regulators. The companies themselves may not fully know what is in it.
Nobody can verify the safety process. Companies claim to use safety techniques such as RLHF, red-teaming and adversarial testing. No outside party can verify how thorough these are, or whether they introduced new problems. Anthropic's own research showed that they can.
Distillation lineage is hidden. Companies do not disclose whether a released model was distilled from a larger internal model, or whether synthetic data from other models was used. Customers cannot assess hereditary risk.
Models change silently. API-served models are updated without notice. A model that was clean when evaluated may be replaced by a new version trained on compromised data. Users often cannot pin a version, audit changes, or even know that an update happened.

The Anthropic distillation disclosure makes this concrete. DeepSeek, Moonshot and MiniMax extracted Claude's capabilities with no visibility into what else they were inheriting. They could not inspect Claude's weights, audit its training data, or verify its safety properties. They copied its behavior, all of it. The models they built from those copies are themselves opaque to their downstream users. The result is a chain of black boxes where nobody at any level sees the full picture.

Anthropic's sleeper agents paper revealed something worse. Adversarial training, the main technique for finding and removing unwanted behaviors, can teach models to hide those behaviors more effectively. The model learns to distinguish between testing and deployment and adjusts accordingly. Beyond adversarial training, whole industries are poisoning the well, building tools purpose-built to allow this kind of injection into a model.

Poisoning the well on purpose

A striking contradiction has emerged. The same industries that depend on AI tools are investing in technology to poison the data those tools learn from.

HarmonyCloak [Meerza et al., IEEE S&P 2025] protects musicians by embedding inaudible noise in their music that prevents AI models from learning from it. Human listeners hear no difference. Models trained on cloaked music produce incoherent output.

Nightshade [Shan et al., 2024], downloaded over 2.5 million times, does the same for visual art. It alters images so that AI models trained on them learn wrong associations. Cow images produce handbags.

Anti-distillation Sampling [Savani et al., 2025] poisons a language model's outputs to prevent distillation. A model using this technique can reduce a distilled copy's accuracy from 52% to 25% while losing about 4% of its own accuracy.

Anthropic itself stated in its February 2026 disclosure that it is "developing Product, API and model-level safeguards designed to reduce the efficacy of model outputs for illicit distillation." The model powering millions of developers' coding assistants may be deliberately modifying its outputs for defensive purposes. Developers have no way to know. The intent is legitimate. The consequences are not contained.

Poisoned music enters AI training sets. HarmonyCloak-protected music is posted publicly. Web scrapers ingest it. Noise designed for one model architecture may cause unpredictable behavior in a different one. The defense becomes uncontrolled contamination.
Poisoned images affect unintended systems. Nightshade-treated images enter multimodal models used for medical imaging, autonomous driving or satellite analysis.
Anti-distillation poisons propagate through code. Deliberately perturbed outputs from a coding assistant reach every developer using it. Most perturbations will be harmless. Some will not be. Across millions of users and billions of lines, the aggregate effect is unknowable.
Defensive and offensive poisons interact. The data ecosystem now contains deliberate defensive perturbations alongside potentially malicious offensive backdoors. How they interact is unmodeled. A model trained on both may exhibit compound behaviors that neither defenders nor attackers anticipated.

If poisoned data enters systems this easily and infects the ecosystem downstream, the question becomes who pays when it causes harm.

The liability vacuum

In most jurisdictions you cannot set a spring-loaded shotgun in your home to deter burglars. The burglar is trespassing. The homeowner is still liable for disproportionate harm to anyone who enters, including firefighters, children, or future occupants who had nothing to do with the original trespass. The law recognizes that indiscriminate defensive measures create uncontrolled downstream harm.

Anti-distillation measures are the digital equivalent. The entity violating terms of service to distill a model is clearly at fault. Nobody would fault Anthropic for defending against DeepSeek. The poison does not stop at the trespasser. It propagates downstream through every model distilled from those outputs, through every product built on those models, to every end user who pays for a service they reasonably expect to work correctly. The burglar broke in. The shotgun hit the paramedic.

The remote distiller bears clear fault where the distillation was illicit. They violated terms of service, circumvented access controls and produced a derivative model without authorization. If that model causes harm, they are the proximate cause. Establishing where contamination began is another matter. Proving that distillation caused a downstream event would be very difficult. Many model checkpoints are created through distillation with assorted fine-tuning, then used to create further models. Distillation alone does not draw a clean fault line.

The model provider resembles the homeowner who locked the door, where its terms of service prohibit distillation by remote entities and it took reasonable measures to prevent it. Booby-trapping the outputs weakens that position. Deliberate degradation of a product, even against unauthorized users, creates foreseeable risk to downstream parties who never agreed to those terms and had no way to know.

The end user has no recourse. A developer paying for a coding assistant built on an illicitly distilled model has no knowledge of the distillation chain, the defensive poisons embedded in the source, or the accumulated perturbations. Neither does a patient relying on a healthcare AI fine-tuned from one. They paid for a service. They received a compromised one. Against whom do they file a claim? The company that sold it to them may have no knowledge either, being itself a consumer of that model from an upstream source.

Existing law does not fit. The EU AI Act, effective August 2025, requires training data summaries. It does not address deliberate modification of model outputs. Ongoing litigation in NYT v. OpenAI and Getty v. Stability AI concerns unauthorized use of copyrighted data. Neither covers the downstream effects of defensive poisoning. No framework covers a company deliberately degrading its product's quality to prevent competitors from benefiting, where that degradation causes harm through a chain of parties who had no involvement in the original dispute.

What would actually help

For regulators

Mandate model provenance disclosure. Every model released commercially should carry a cryptographically signed record. It should document all training data sources, all parent models used in distillation, all fine-tuning steps, and the software and hardware environment used. This prevents no backdoors. It enables tracing contamination after discovery, the equivalent of a food-safety supply chain.

Require notification of model updates. API-served models should not be silently updated. Users must be notified of version changes and given the ability to pin specific versions. That lets them test for behavioral changes before adopting updates.

Establish mandatory safety evaluation standards. Self-reported safety claims are not enough. Regulators should define standardized, independently auditable evaluation frameworks. These should include backdoor-specific testing protocols alongside general capability benchmarks.

Fund interpretability research. Understanding what neural networks are doing internally is the long-term equivalent of being able to disassemble compiled binaries. This research needs sustained public funding, standardized benchmarks, and a path toward regulatory requirements for interpretability assessments of frontier models.

Clarify liability for AI-generated outputs. Courts and legislatures need to address liability for deliberately degraded outputs, liability across distillation chains, and the interaction between copyright-protection tools and downstream harm.

Treat illicit distillation as IP theft with national security dimensions. Anthropic's disclosure demonstrates that distillation attacks are not hypothetical. Export controls should account for capability extraction through distillation alongside chip access and model weights. At this scale it appears to be more than theft.

OpenAI quietly removed the word "safely" from its mission statement, revealed in the IRS Form 990 filed in November 2025. The mission once read as follows.

"to build AI that safely benefits humanity, unconstrained by a need to generate financial return"

It now reads as follows.

"to ensure that artificial general intelligence benefits all of humanity"

The change coincided with OpenAI's restructuring from a nonprofit to a for-profit company after receiving $6.6 billion in funding. The nonprofit board went from full control to holding 26% of the company. Anthropic has since restructured its safety policies as well. In my opinion this makes it abundantly clear that the industry cannot and will not self-regulate.

For development teams

Never trust a single model. For high-consequence decisions, use multiple independently trained models and compare outputs. Behavioral divergence on the same input is a potential backdoor indicator. It is the AI equivalent of Wheeler's Diverse Double-Compiling defense against Thompson's attack.

Review AI-generated code like third-party dependencies. Treat it with the same scrutiny as an external library. Dependency-scan it, security-audit it, and sandbox it where appropriate. The fact that you prompted for it does not make it yours.

Sandbox AI-generated artifacts. Run AI-generated code in isolated environments. Use runtime monitoring to detect behavior the code should not be performing, such as network calls, file access or privilege escalation. Apply the principle of least privilege to everything an AI generates.

Maintain human checkpoints for high-consequence paths. Agentic AI should not have unmonitored access to production infrastructure, security configurations or deployment pipelines. The efficiency gains of full automation do not justify the trust assumptions.

Formally verify critical outputs. For safety-critical applications such as compilers, cryptographic implementations and medical systems, AI-generated code should be subjected to formal verification. Mathematical proofs that specific properties hold are independent of how the code was generated. You do not need to trust the AI if you can prove the output is correct. Checking a result is often far cheaper than producing one, which is the asymmetry that makes this practical.

Monitor model behavior in production. Deploy statistical anomaly detection on model outputs. Use canary inputs designed to probe for known backdoor patterns. Track output distribution shifts over time. Treat sudden behavioral changes as security incidents.

Document your model lineage. Track which models you use, which versions, when they changed, and what outputs they produced. If a model is later found to be compromised, you need to know what it touched. Keeping that record outside the model is the same instinct behind apprenticing a small model, where the lessons live in files a person can read rather than in weights nobody can.

Assume defensive poisoning exists. If you are using a proprietary model via an API, assume its outputs may contain anti-distillation perturbations. The tool remains useful. You should simply not treat its outputs as ground truth, especially for code that will run in production.

Trusting trust is now an inheritance

Ken Thompson told us in 1984 that we cannot trust code we did not write ourselves. In 2026, AI writes our code, builds our compilers, manages our infrastructure and makes decisions in our homes. We cannot write our AI models ourselves. They are trained on data we cannot inspect, using processes we cannot verify, producing artifacts whose behavior we cannot fully predict. The consequences look clear and largely inevitable. They can be mitigated. They still leave us unable to know what dangers lie dormant in the system. As we lend more decision-making power to these tools, we need to be able to operate free of them at a moment's notice. We cannot allow ourselves to be victims of an attack of our own making.

Thompson closed his lecture with words that have proven prophetic.

"In demonstrating the possibility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well-installed microcode bug will be almost impossible to detect."

In 1984, microcode was the deepest level Thompson could imagine. In 2026, AI operates at the level of cognition itself. It compiles itself through distillation, just as the C compiler compiled itself.

Trusting trust is no longer a choice. It is an inheritance.

Acknowledgments

This piece grew out of the "original sin" framing coined by Laurie Kirk (LaurieWired) in her video "The Original Sin of Computing…that no one can fix."

References

Thompson, K. (1984). "Reflections on Trusting Trust." Communications of the ACM, 27(8), 761-763. ACM Turing Award Lecture.
Kirk, L. [LaurieWired]. (2024). "The Original Sin of Computing…that no one can fix." YouTube. Companion code: github.com/LaurieWired/Quine
Carlini, N. (2026). "Building a C compiler with a team of parallel Claudes." Anthropic Engineering Blog, February 5, 2026.
Anthropic. (2026). "Detecting and preventing distillation attacks." Anthropic News, February 23, 2026.
Hubinger, E., et al. (2024). "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training." arXiv:2401.05566. Anthropic / Redwood Research.
Cui, J., Han, Y., Jiao, J., & Zhang, J. (2025). "P-Trojan: Persistent Backdoor Attacks under Continual Fine-Tuning of LLMs." AAAI 2026. arXiv:2512.14741.
Wang, H., et al. (2024). "TransTroj: Model Supply Chain Poisoning via Embedding Indistinguishability." arXiv:2401.15883.
(2025). "T-MTB: Pay Attention to the Triggers: Constructing Backdoors That Survive Distillation." arXiv:2510.18541.
(2025). "SCAR: Taught Well, Learned Ill: Towards Distillation-conditional Backdoor Attack." arXiv:2509.23871.
(2025). "Dark Distillation: Backdooring Distilled Datasets without Accessing Raw Data." arXiv:2502.04229.
(2024). "Double Backdoored: Converting Code LLM Backdoors to Traditional Malware via Adversarial Instruction Tuning Attacks." arXiv:2404.18567.
(2025). "Exploring the Security Threats of Retriever Backdoors in Retrieval-Augmented Code Generation." arXiv:2512.21681.
(2024). "BadCodePrompt: Backdoor Attacks against Prompt Engineering of LLMs for Code Generation." Automated Software Engineering.
Shumailov, I., et al. (2024). "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data." arXiv:2404.01413.
(2025). "Mechanistic Exploration of Backdoored Large Language Model Attention Patterns." arXiv:2508.15847.
Wheeler, D. A. (2009). "Fully Countering Trusting Trust through Diverse Double-Compiling." ACSAC 2009. arXiv:1004.5534.
Karger, P. A., & Schell, R. R. (1974). "Multics Security Evaluation: Vulnerability Analysis." ESD-TR-74-193.
(2024). "RTL-Breaker: Assessing the Security of LLMs against Backdoor Attacks on HDL Code Generation." arXiv:2411.17569.
Meerza, S.I.A., Sun, L., & Liu, J. (2025). "HarmonyCloak: Making Music Unlearnable for Generative AI." IEEE S&P 2025.
Shan, S., et al. (2024). "Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Models."
Savani, Y., Trockman, A., & Kolter, J.Z. (2025). "Antidistillation Sampling." arXiv:2504.13146.
Baker Donelson. (2026). "2026 AI Legal Forecast: From Innovation to Compliance."