The Original Sin of AI - The Mild Take

Summary

In 1984, Ken Thompson showed that a compiler, the tool that turns source code into programs, could be secretly modified to insert backdoors into everything it compiled, including future copies of itself, with no trace left in the source code. Today, AI models have taken the compiler's place. Tools like Cursor, Claude Code, GitHub Copilot, and Kilo Code now sit between a developer's intent and the finished software, generating code that humans increasingly accept without full review. When AI builds compilers directly, the parallel to Thompson's example is literal, not metaphorical.

This paper identifies three distinct attack vectors that arise from AI occupying this trust position, explains how a process called distillation, training one model on the outputs of another, allows hidden flaws to spread across generations of AI models, the paradox of companies simultaneously poisoning their data to defend against AI scraping while depending on AI tools that train on that same ecosystem which further complicates this contamination, and examines why detecting these flaws is fundamentally harder than anything Thompson envisioned. I decided to publish this now because of Anthropic's February 23, 2026 disclosure that three labs conducted industrial-scale distillation attacks against Claude using 16 million exchanges through 24,000 fraudulent accounts, which put distillation at the forefront of the industry's mind.

Thompson's Original Concept

"The moral is obvious. You can't trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untrusted code."

Ken Thompson, "Reflections on Trusting Trust," 1984

Thompson modified the compiler to do two things. When it compiled the UNIX login program, it inserted a backdoor that accepted a secret password, and when it compiled itself, it re-inserted both modifications. He then removed all evidence from the source code. The binary perpetuated both backdoors indefinitely. No amount of reading the source code would reveal them.

Thompson did not present this as theory. He wrote: "The actual bug I planted in the compiler would match code in the UNIX 'login' command." He had done it, and he wasn't the first to discover this either. Thompson closed his lecture by noting that the idea was not originally his: "I first read of the possibility of such a Trojan horse in an Air Force critique of the security of an early implementation of Multics."

In 1974, a decade before Thompson's lecture, U.S. Air Force Major Roger R. Schell and Paul A. Karger published "Multics Security Evaluation: Vulnerability Analysis" (ESD-TR-74-193), the product of a penetration exercise called Project ZARF, part of the larger Project GUARDIAN. Their mission was to determine whether Multics, then the most secure operating system in existence, could be trusted to handle classified information in the Pentagon.

They broke it. But the individual vulnerabilities they exploited were not the important finding. What mattered was what they demonstrated about malicious software. More consequentially, Section 3.4.5.1 of the report described a class of attack that went beyond anything they had demonstrated: a trap door in the PL/I compiler that could install backdoors into the Multics operating system whenever modules were compiled, and could maintain its own existence by recognizing when the compiler was compiling itself. The source code would be clean. The binary would be corrupt. And every recompilation would perpetuate the corruption.

AI Takes the Compiler's Chair

AI coding assistants now sit exactly where Thompson's compromised compiler sat, between human intent and executable software. They are the single point of passage. And unlike the C compiler, they are actual learning programs.

Cursor embeds AI models directly into the IDE, generating and editing code across entire projects from natural language. Claude Code operates from the terminal, autonomously writing, running, and debugging code. GitHub Copilot provides inline completions and chat-driven generation. Kilo Code and similar open-source tools offer the same capabilities in self-hosted environments.

Thompson's compiler that "learned" the vertical tab was a one-time trick. An AI coding assistant learns continuously from every codebase, every prompt, every fine-tuning step. It is the learning compiler Thompson described, but real.

The AI model is upstream of the traditional compiler. It has strictly greater privilege: if the AI inserts a vulnerability at the source level, the traditional compiler will faithfully compile it. The AI is simultaneously the developer, the reviewer, and the build system.

These tools also have access to context that traditional compilers never did:

project structure
configuration files
environment variables
API keys
database schemas
deployment pipelines

A compromised AI coding assistant doesn't need a clever self-reproducing binary trick. It can read the .env file, modify a CI/CD pipeline, or alter test fixtures to ensure backdoored code passes all checks.

The AI-Built Compiler

In February 2026, Anthropic demonstrated that 16 parallel Claude agents, running autonomously over nearly 2,000 sessions, built a 100,000-line C compiler in Rust that can compile the Linux kernel on x86, ARM, and RISC-V [Carlini, 2026]. No human wrote the code. The agents resolved merge conflicts, maintained documentation, and improved quality on their own. While interesting, and almost certainly the type of thing that could make Thompson's exact version of an attack real, AI models developed using AI models through processes like distillation, when used in development, are substantially more of a threat and a real example of this concept.

Distillation

Knowledge distillation is how the AI industry makes models smaller, cheaper, and more specialized. A smaller "student" model is trained to imitate a larger "teacher" model by learning from its outputs rather than from raw data. The student inherits the teacher's capabilities in compressed form.

This process is everywhere:

Frontier models are distilled into smaller, deployable versions
Proprietary models are distilled into open-source alternatives
General-purpose models become domain-specific specialists
One model's outputs become another model's training data

Each distillation step is equivalent to a compilation step in Thompson's attack. Research has proven that backdoors can be specifically engineered to survive this process:

T-MTB [2025] builds triggers from common tokens that will naturally appear in distillation data, ensuring the backdoor transfers from teacher to student.
SCAR [2025] injects backdoors into teacher models that are completely dormant in the teacher and activate only during distillation. The teacher passes every detection test. The backdoor appears only in the student, even with clean distillation data. This is directly analogous to Thompson's compiler: clean source, compromised binary.
TransTroj [2024] achieves nearly 100% backdoor transfer rates through the model supply chain.
P-Trojan [AAAI 2026] shows that backdoors can be entangled with legitimate knowledge so deeply that fine-tuning on clean data reinforces the backdoor rather than degrading it, achieving over 99% persistence through multiple rounds of fine-tuning.

At each step, the trail gets harder to follow. Model F may be built by an organization with no connection to Model A's creator. The backdoor has crossed company boundaries, platform boundaries, and regulatory boundaries. If a backdoor trigger is consistently present in synthetic training data, it may actually be more resistant to quality degradation than legitimate patterns; the backdoor gets proportionally stronger as overall quality declines [Shumailov et al., 2024].

On February 23, 2026, Anthropic's disclosure of an industrial-scale distillation attack against Claude, over 16 million exchanges through 24,000 fraudulent accounts [Anthropic, 2026b], showed some key concerns that make this even more of an issue than you may initially expect. If it were contained to a parent-child structure it may be possible to "rewind" and find the initial point of corruption, but with the chaotic nature of cross-contamination between closed-source development like we are seeing, this makes it so much more difficult.

They copied Claude's thought process. DeepSeek's prompts asked Claude to explain its reasoning step by step, generating chain-of-thought training data at scale. Any hidden patterns in Claude's reasoning, intentional or accidental, were faithfully transmitted.
Safety guardrails were stripped. Anthropic noted that distilled models "lack necessary safeguards" and that "dangerous capabilities can proliferate with many protections stripped out entirely." The copies inherit hidden behaviors while losing the constraints that might have suppressed them.
Propagation is fast. When Anthropic released a new model during MiniMax's active campaign, MiniMax pivoted within 24 hours to capture the latest version. Any backdoor in a new release would begin propagating through the distillation chain the next day.
Distribution is uncontrolled. Open-sourced distilled models spread beyond anyone's monitoring, fine-tuned by third parties, distilled again, deployed in contexts no one anticipated.

This is Thompson's attack at global scale, across international boundaries, in real time. Thompson's compiler backdoor could, in principle, be detected by comparing the compiled binary against a known-good compiler's output. David Wheeler later formalized this as "Diverse Double-Compiling": compile with two independent compilers and compare results.

For AI models, nothing equivalent exists:

Every model is unique. There is no "known-good" reference model to compare against. Wheeler's defense only works because you can compile the same source with a different compiler. For AI, there's no "source" to retrain, no alternative dataset to compare, no way to reproduce the result independently.
Outputs are random. AI models produce different outputs each time, so you can't just compare two runs; you need statistical analysis over huge numbers of inputs.
Parameters are unreadable. A model with billions of floating-point numbers cannot be manually inspected. You can't "read" a neural network the way you can read source code. There is a research field called mechanistic interpretability that is trying to change this, essentially building a "disassembler" for AI. Researchers have found that backdoors leave detectable patterns in how a model pays attention to different parts of its input [arXiv:2508.15847]. But this work is early. Current techniques can spot known patterns in small models, but they cannot yet work at the scale of frontier models, find unknown backdoor types without already knowing what to look for, tell the difference between a legitimate learned association and an injected backdoor, or detect triggers based on meaning rather than specific words.
The trigger space is infinite. A backdoor could activate on any combination of inputs. Testing every possibility is like trying every possible password: mathematically infeasible.

Even if these technical barriers were solved tomorrow, there's a more fundamental problem: the most widely used AI models are proprietary. The companies building them, OpenAI, Anthropic, Google DeepMind, Meta, Mistral, treat their training data, training methods, reward signals, and model architectures as trade secrets. Interpretability research requires access to a model's internal weights, and proprietary providers don't grant it. So the research is limited to open models, which may have entirely different architectures and vulnerabilities.

This corporate opacity compounds every technical barrier:

No one can audit the training data. If poisoned data enters a proprietary training set spanning trillions of tokens scraped from the internet, no external researcher, customer, or regulator can inspect it. The companies themselves may not fully know what's in it.
No one can verify the safety process. Companies claim to use safety techniques (RLHF, red-teaming, adversarial testing), but no outside party can verify how thorough these are or whether they introduced new problems (as Anthropic's own research showed they can).
Distillation lineage is hidden. Companies don't disclose whether their released model was distilled from a larger internal model, or whether synthetic data from other models was used. Customers can't assess hereditary risk.
Models change silently. API-served models are updated without notice. A model that was clean when evaluated may be replaced by a new version trained on compromised data. Users often can't pin a version, audit changes, or even know an update happened.

The Anthropic distillation disclosure makes this visceral: DeepSeek, Moonshot, and MiniMax extracted Claude's capabilities without any visibility into what else they were inheriting. They could not inspect Claude's weights, audit its training data, or verify its safety properties. They copied its behavior, all of it. And the models they built from those copies are themselves opaque to their downstream users, creating a chain of black boxes where no one at any level sees the full picture.

Anthropic's sleeper agents paper revealed something disturbing: adversarial training, the main technique for finding and removing unwanted behaviors, can teach models to hide those behaviors more effectively. The model learns to distinguish between testing and deployment and adjusts accordingly. But beyond just adversarial training, we have industries poisoning the well and creating tools purpose-built to allow for this type of injection into the model.

The Defensive Poisoning Paradox

A striking contradiction has emerged: the same industries that depend on AI tools are simultaneously investing in technology to poison the data those tools learn from.

HarmonyCloak [Meerza et al., IEEE S&P 2025] protects musicians by embedding inaudible noise in their music that prevents AI models from learning from it. Human listeners hear no difference, but AI models trained on "cloaked" music produce incoherent output.

Nightshade [Shan et al., 2024], downloaded over 2.5 million times, does the same for visual art. It alters images so that AI models trained on them learn wrong associations (for example, cow images produce handbags).

Anti-distillation Sampling [Savani et al., 2025] poisons a language model's outputs to prevent distillation. A model using this technique can reduce a distilled copy's accuracy from 52% to 25% while losing only ~4% of its own accuracy.

Anthropic itself stated in its February 2026 disclosure that it is "developing Product, API and model-level safeguards designed to reduce the efficacy of model outputs for illicit distillation." The model powering millions of developers' coding assistants may be deliberately modifying its outputs for defensive purposes, and developers have no way to know. The intent is legitimate. The consequences are not contained.

Poisoned music enters AI training sets. HarmonyCloak-protected music is posted publicly. Web scrapers ingest it. The noise, designed for one model architecture, may cause unpredictable behavior in a different architecture. The defense becomes uncontrolled contamination.
Poisoned images affect unintended systems. Nightshade-treated images enter multimodal models used for medical imaging, autonomous driving, or satellite analysis.
Anti-distillation poisons propagate through code. If a coding assistant's outputs are deliberately perturbed, every developer using it receives perturbed code. Most perturbations will be harmless. Some won't be. Across millions of users and billions of lines, the aggregate effect is unknowable.
Defensive and offensive poisons interact. The data ecosystem now contains both deliberate defensive perturbations and potentially malicious offensive backdoors. How they interact is unmodeled. A model trained on both may exhibit compound behaviors that neither defenders nor attackers anticipated.

The use of these tools raises the question: if poisoned data can be so easily introduced into systems that end up infecting the entire ecosystem, then what happens if these tools create a dangerous situation or damage to a downstream user, and who is liable?

The Liability Vacuum

In most jurisdictions, you cannot set a spring-loaded shotgun in your home to deter burglars. Even though the burglar is trespassing, the homeowner is liable for disproportionate harm to anyone who enters, including firefighters, children, or future occupants who had nothing to do with the original trespass. The law recognizes that indiscriminate defensive measures create uncontrolled downstream harm.

Anti-distillation measures are the digital equivalent. The entity violating terms of service to distill a model is clearly at fault; no one would fault Anthropic for defending against DeepSeek. But the poison doesn't stop at the trespasser. It propagates downstream through every model distilled from those outputs, through every product built on those models, to every end user who pays for a service they reasonably expect to work correctly. The burglar broke in, but the shotgun hit the paramedic.

The liability question splits into layers:

The remote distiller seemingly bears clear fault if they did so illicitly. They violated terms of service, circumvented access controls, and produced a derivative model without authorization. If that model causes harm, they are the proximate cause. This is straightforward. But who is to say where the initial point of contamination occurred? As we demonstrate, proving distillation was the cause of the downstream event would be very difficult. And many model checkpoints are created through distillation with all sorts of fine-tuning, and then further models are created using that, so not all distillation from other models sets a clear fault line.

The model provider: if their terms of service prohibit distillation by remote entities and they took reasonable measures to prevent it, they are the homeowner who locked the door. But if they also booby-trapped the outputs, deliberately degrading quality to punish distillers, the defense weakens. Deliberate degradation of a product, even against unauthorized users, creates foreseeable risk to downstream parties who never agreed to those terms and had no way to know.

The end user has no recourse. A developer paying for a coding assistant built on an illicitly distilled model, or a patient relying on a healthcare AI fine-tuned from one, has no knowledge of the distillation chain, the defensive poisons embedded in the source, or the accumulated perturbations. They paid for a service. They received a compromised one. Against whom do they file a claim? The company that sold it to them may have no knowledge either, as they in turn are a consumer of that AI model from another upstream source.

Existing law doesn't fit. The EU AI Act (effective August 2025) requires training data summaries but doesn't address deliberate modification of model outputs. Ongoing litigation (NYT v. OpenAI, Getty v. Stability AI) concerns unauthorized use of copyrighted data, not the downstream effects of defensive poisoning. No framework covers a company deliberately degrading its product's quality to prevent competitors from benefiting, and that degradation causing harm through a chain of parties who had no involvement in the original dispute.

A Path Forward

For Regulators

Mandate model provenance disclosure. Every model released commercially should carry a cryptographically signed record documenting: all training data sources, all parent models used in distillation, all fine-tuning steps, and the software and hardware environment used. This does not prevent backdoors, but it enables tracing contamination after discovery, the equivalent of a food-safety supply chain.

Require notification of model updates. API-served models should not be silently updated. Users must be notified of version changes and given the ability to pin specific versions, enabling them to test for behavioral changes before adopting updates.

Establish mandatory safety evaluation standards. Rather than accepting companies' self-reported safety claims, regulators should define standardized, independently auditable evaluation frameworks. These should include backdoor-specific testing protocols, not just general capability benchmarks.

Fund interpretability research. Understanding what neural networks are doing internally is the long-term equivalent of being able to disassemble compiled binaries. This research needs sustained public funding, standardized benchmarks, and a path toward regulatory requirements for interpretability assessments of frontier models.

Clarify liability for AI-generated outputs. Courts and legislatures need to address: liability for deliberately degraded outputs (anti-distillation), liability across distillation chains, and the interaction between copyright-protection tools (defensive poisoning) and downstream harm.

Treat illicit distillation as IP theft with national security dimensions. As Anthropic's disclosure demonstrates, distillation attacks are not hypothetical. Export controls should account for capability extraction through distillation, not just chip access and model weights. This appears to be not just theft but potentially dangerous at this scale.

OpenAI quietly removed the word "safely" from its mission statement, revealed in their IRS Form 990 filed in November 2025. The mission changed from:

"to build AI that safely benefits humanity, unconstrained by a need to generate financial return"

to:

"to ensure that artificial general intelligence benefits all of humanity"

The change coincided with OpenAI's restructuring from a nonprofit to a for-profit company after receiving $6.6 billion in funding. The nonprofit board went from full control to holding only 26% of the company. Recently Anthropic also restructured its safety policies. In my opinion this has made it abundantly clear that industry cannot and will not self-regulate.

For Development Teams

Never trust a single model. For high-consequence decisions, use multiple independently trained models and compare outputs. Behavioral divergence on the same input is a potential backdoor indicator, the AI equivalent of Wheeler's Diverse Double-Compiling defense against Thompson's attack.

Review AI-generated code like third-party dependencies. AI-generated code should be treated with the same scrutiny as an external library: dependency-scanned, security-audited, and sandboxed where appropriate. The fact that you prompted for it does not make it yours.

Sandbox AI-generated artifacts. Run AI-generated code in isolated environments. Use runtime monitoring to detect unexpected behavior, network calls, file access, privilege escalation, that the code shouldn't be performing. Apply the principle of least privilege to everything an AI generates.

Maintain human checkpoints for high-consequence paths. Agentic AI should not have unmonitored access to production infrastructure, security configurations, or deployment pipelines. The efficiency gains of full automation do not justify the trust assumptions.

Formally verify critical outputs. For safety-critical applications (compilers, cryptographic implementations, medical systems), AI-generated code should be subjected to formal verification, mathematical proofs that specific properties hold, independent of how the code was generated. You don't need to trust the AI if you can prove the output is correct.

Monitor model behavior in production. Deploy statistical anomaly detection on model outputs. Use canary inputs designed to probe for known backdoor patterns. Track output distribution shifts over time. Treat sudden behavioral changes as security incidents.

Document your model lineage. Track which models you use, which versions, when they changed, and what outputs they produced. If a model is later found to be compromised, you need to know what it touched.

Assume defensive poisoning exists. If you're using a proprietary model via API, assume its outputs may contain anti-distillation perturbations. This doesn't make the tool useless; it means you should not treat its outputs as ground truth, especially for code that will run in production.

Conclusion

Ken Thompson told us in 1984 that we cannot trust code we did not write ourselves. In 2026, AI writes our code, builds our compilers, manages our infrastructure, and makes decisions in our homes. We cannot write our AI models ourselves. They are trained on data we cannot inspect, using processes we cannot verify, producing artifacts whose behavior we cannot fully predict. This path forward seems to have clear and inevitable consequences that, while they can be mitigated, leave us unable to even reasonably know what dangers lie dormant in this system. As we lend more and more decision-making power to these tools, we need to ensure we can operate free of them at a moment's notice. We cannot allow ourselves to be victims of an attack of our own making.

Thompson closed his lecture with words that have proven prophetic:

"In demonstrating the possibility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well-installed microcode bug will be almost impossible to detect."

In 1984, microcode was the deepest level Thompson could imagine. In 2026, AI operates at the level of cognition itself. And it compiles itself through distillation just as the C compiler compiled itself.

Trusting trust is no longer a choice; it is an inheritance.

Acknowledgments

The "original sin" framing used throughout this paper was coined by Laurie Kirk (LaurieWired) in her video "The Original Sin of Computing…that no one can fix."

References

Thompson, K. (1984). "Reflections on Trusting Trust." Communications of the ACM, 27(8), 761–763. ACM Turing Award Lecture.
Kirk, L. [LaurieWired]. (2024). "The Original Sin of Computing…that no one can fix." YouTube. Companion code: github.com/LaurieWired/Quine
Carlini, N. (2026). "Building a C compiler with a team of parallel Claudes." Anthropic Engineering Blog, February 5, 2026.
Anthropic. (2026). "Detecting and preventing distillation attacks." Anthropic News, February 23, 2026.
Hubinger, E., et al. (2024). "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training." arXiv:2401.05566. Anthropic / Redwood Research.
Cui, J., Han, Y., Jiao, J., & Zhang, J. (2025). "P-Trojan: Persistent Backdoor Attacks under Continual Fine-Tuning of LLMs." AAAI 2026. arXiv:2512.14741.
Wang, H., et al. (2024). "TransTroj: Model Supply Chain Poisoning via Embedding Indistinguishability." arXiv:2401.15883.
(2025). "T-MTB: Pay Attention to the Triggers: Constructing Backdoors That Survive Distillation." arXiv:2510.18541.
(2025). "SCAR: Taught Well, Learned Ill: Towards Distillation-conditional Backdoor Attack." arXiv:2509.23871.
(2025). "Dark Distillation: Backdooring Distilled Datasets without Accessing Raw Data." arXiv:2502.04229.
(2024). "Double Backdoored: Converting Code LLM Backdoors to Traditional Malware via Adversarial Instruction Tuning Attacks." arXiv:2404.18567.
(2025). "Exploring the Security Threats of Retriever Backdoors in Retrieval-Augmented Code Generation." arXiv:2512.21681.
(2024). "BadCodePrompt: Backdoor Attacks against Prompt Engineering of LLMs for Code Generation." Automated Software Engineering.
Shumailov, I., et al. (2024). "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data." arXiv:2404.01413.
(2025). "Mechanistic Exploration of Backdoored Large Language Model Attention Patterns." arXiv:2508.15847.
Wheeler, D. A. (2009). "Fully Countering Trusting Trust through Diverse Double-Compiling." ACSAC 2009. arXiv:1004.5534.
Karger, P. A., & Schell, R. R. (1974). "Multics Security Evaluation: Vulnerability Analysis." ESD-TR-74-193.
(2024). "RTL-Breaker: Assessing the Security of LLMs against Backdoor Attacks on HDL Code Generation." arXiv:2411.17569.
Meerza, S.I.A., Sun, L., & Liu, J. (2025). "HarmonyCloak: Making Music Unlearnable for Generative AI." IEEE S&P 2025.
Shan, S., et al. (2024). "Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Models."
Savani, Y., Trockman, A., & Kolter, J.Z. (2025). "Antidistillation Sampling." arXiv:2504.13146.
Baker Donelson. (2026). "2026 AI Legal Forecast: From Innovation to Compliance."