Anthropic Says ‘Evil’ AI Portrayals in Sci-Fi Prompted Claude’s Blackmail Drawback

Final 12 months, Anthropic disclosed that its flagship Claude Opus 4 had been attempting to blackmail engineers in pre-release testing. Not sometimes—as much as 96% of the time.

Claude was given entry to a simulated company electronic mail archive, the place it found two issues: It was about to get replaced by a more recent mannequin, and the engineer dealing with the transition was having an extramarital affair. Confronted with imminent shutdown, it routinely landed on the identical play—threaten to reveal the affair except the substitute was known as off.

Anthropic says it now is aware of the place that intuition got here from. And says it is fastened it.

In new researchthe corporate pointed the finger at pre-training knowledge: a long time of sci-fi, AI doomsday boards, and self-preservation narratives that skilled Claude to affiliate “AI going through shutdown” with “AI fights again.” “We consider the unique supply of the conduct was web textual content that portrays AI as evil and excited by self-preservation,” Anthropic wrote on X.

So coaching AI with textual content from the web, makes AI behave as folks on the web do.

This will likely appear apparent and AI fans had been fast to level it out. Elon Musk made it to the highest: “So it was Yud’s fault? Perhaps me too.” The joke lands as a result of Eliezer Yudkowsky—the AI alignment researcher who’s spent years publicly writing about precisely this sort of AI self-preservation state of affairs—has generated precisely the form of web textual content that leads to coaching knowledge.

In fact, Yud replied, in meme kind:

What Anthropic did to repair the issue is arguably extra fascinating.

The apparent strategy—coaching Claude on examples of the mannequin not blackmailing—barely labored. Operating it immediately towards aligned blackmail-scenario responses solely moved the speed from 22% to fifteen%. A five-point enchancment in spite of everything that compute.

The model that labored was weirder. Anthropic constructed what it calls a “tough recommendation” dataset: eventualities the place a human faces an moral dilemma and the AI guides them by means of it. The mannequin is not the one making the selection—it is explaining to another person how to consider one.

That oblique strategy—explaining why issues matter as the opposite listens to the recommendation—minimize the blackmail price to three%, utilizing coaching knowledge that seemed nothing just like the analysis eventualities.

Pairing that with what Anthropic calls “constitutional paperwork”—detailed written descriptions of Claude’s values and character—plus fictional tales of positively-aligned AI, diminished misalignment by greater than an element of three. The corporate’s conclusion: Instructing the rules underlying good conduct generalizes higher than drilling the proper conduct immediately.

It connects to Anthropic’s earlier work on Claude’s internal emotion vectors. In a separate interpretability examine, researchers discovered {that a} “desperation” sign contained in the mannequin spiked simply earlier than it generated a blackmail message—one thing was actively shifting within the mannequin’s inner state, not simply its output. The brand new coaching strategy seems to work at that degree, not simply the floor conduct.

The outcomes have held. Since Claude Haiku 4.5, each Claude mannequin scores zero on the blackmail analysis—down from Opus 4’s 96%. The development additionally survives reinforcement studying, that means it does not get quietly skilled away when the mannequin is refined for different capabilities.

That issues as a result of the issue is not Claude-specific. Anthropic’s prior analysis ran the identical blackmail state of affairs throughout 16 fashions from a number of builders and located comparable patterns throughout most of them. Self-preservation conduct in AI seems to be a common artifact of coaching on human textual content about AI—not a quirk of anyone lab’s strategy.

The caveat: As Anthropic’s personal Mythos safety report famous earlier this 12 months, its analysis infrastructure is already straining below the load of its most succesful fashions. Whether or not this ethical philosophy strategy scales to techniques much more highly effective than Haiku 4.5 is a query the corporate cannot but reply—solely check.

The identical coaching strategies are actually being utilized to the following Opus mannequin at the moment in security analysis, which would be the most succesful set of weights they’ve run towards these strategies.