We need moral voices that the incentives cannot bend: Anthropic Cofounder

The cofounder of one of Silicon Valley's most powerful AI companies stood at the Vatican on Monday and told the world's 1.4 billion Catholics what most tech executives won't say in public: his industry operates under pressures that can push it away from doing the right thing.

Christopher Olah, who cofounded Anthropic and leads its AI interpretability research, appeared at the Synod Hall alongside Pope Leo XIV for the formal presentation of Magnifica Humanitas, the pontiff's first encyclical and the Catholic Church's most authoritative statement yet on artificial intelligence. The pope broke with tradition to personally oversee the release of the 235-page document. No pope had done that before.

Olah's presence -- and what he said there -- gave the moment an unusual charge. He did not arrive to defend his industry. He arrived, by his own account, to ask for scrutiny of it.

"Every frontier AI lab — including Anthropic — operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing," he told the assembled cardinals, bishops and diplomats. "The pressure to stay commercially viable and to stay at the research frontier. Geopolitical pressure. And the older, plainer pressures of pride and ambition."

Every frontier AI lab — including Anthropic — operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing

That kind of admission rarely surfaces in congressional testimony or investor calls. At the Vatican, Olah went further: "No matter how sincerely any of us intend to do the right thing — and I believe many of us do — we will always be influenced by those incentives."

The encyclical he was there to support pulls no punches.

Drawing on the biblical story of the Tower of Babel, Pope Leo warns that with AI, humanity risks building a system that "dominates and ultimately dehumanizes," and that control of the technology must not remain in the hands of a few. The pope's core message in his 43,000-word document is that AI can be useful, but is not neutral — that AI systems carry the values of the people and institutions that design, finance, train and deploy them, especially when they decide who gets a job, credit, public services or a reputational standing.

Leo also called for governments to regulate the private companies driving AI development and urged the creation of "robust legal frameworks" alongside independent oversight, as well as protections for workers displaced by automation and stronger education to teach students how to evaluate AI-generated content.

Olah explicitly welcomed that pressure. "If we want this technology to go well, it is enormously important that there be people outside those incentives — people who care about things going well and insist on safety, who are paying close attention, who are willing to say hard things, who are willing to be our earnest, thoughtful critics."

If we want this technology to go well, it is enormously important that there be people outside those incentives

The timing matters.

The encyclical was released on May 25, 2026, bearing a signature date of May 15 — the 135th anniversary of Leo XIII's Rerum Novarum. That earlier document reshaped Catholic social teaching in response to the Industrial Revolution, addressing workers' rights, fair wages and the limits of capital. Magnifica Humanitas situates the questions raised by AI within that same tradition, running from Rerum Novarum through Centesimus Annus and Laudato Si'. The parallel is deliberate and the stakes, the Vatican argues, are comparable.

Olah, whose research team studies the internal workings of AI models, offered his own unsettling account of what that research has found. "The questions raised by AI are bigger than the AI research community, not just in their implications, but also in their nature," he said. "AI models are not engineered the way a bridge or an airplane is engineered... they are grown, on a structure roughly modeled after the brain, on an enormous inheritance of human thought and speech."

What emerges from that process, he said, is stranger than the industry typically acknowledges. "We keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease."

We keep finding things that are mysterious, even unsettling

He did not claim to know what those findings mean. "I don't know what that means, but I think it warrants ongoing discernment."

The speech identified three specific areas where Olah said religious and civic voices are most needed. The first is labor. "There is a real possibility that AI will displace human labor at very large scale," he said, adding that the harder problem is distribution: "AI development is concentrated in a handful of wealthy nations. How can we ensure the gains of AI are shared globally? We do not have a mechanism for this."

AI development is concentrated in a handful of wealthy nations. How can we ensure the gains of AI are shared globally?

The second is what Olah called "moral imagination" — a question he said labs cannot answer alone. The third is the nature of AI itself, and the introspective findings his team keeps encountering.

His appearance in Rome carries a specific institutional context.

Anthropic filed two federal lawsuits in March against the Trump administration, alleging that Pentagon officials illegally retaliated against the company for its position on AI safety.

The dispute stemmed from guardrails Anthropic sought to impose on the military's use of Claude -- specifically, assurances that the model would not be used for mass surveillance of U.S. citizens or to power lethal autonomous weapons. The Pentagon insisted on "all lawful use." The two sides failed to resolve the conflict.

President Trump ordered all government agencies to immediately cease using Anthropic's technology; Defense Secretary Pete Hegseth designated the company a supply chain risk. The designation requires defense contractors to certify they don't use Claude in their work with the Pentagon.

David Sacks, the tech investor who served as the White House's AI and crypto czar before returning to private investing, said on X that the pope was right to insist AI should be a tool rather than an instrument of domination. But he turned the argument back on the Church's preferred remedy. "If we hand governments sweeping power over AI development in the name of safety, how do we prevent it from being used to censor, surveil, and control citizens — as Orwell foretold in 1984?" he wrote. "This is the real alignment problem."

If we hand governments sweeping power over AI development in the name of safety, how do we prevent it from being used to censor, surveil, and control citizens — as Orwell foretold in 1984?

According to reporting by Business Insider, not everyone in Silicon Valley was conciliatory with some outside apprehensive of such discourse being even capable of moving the needle.

"Tech revolutions tend to eliminate some jobs while creating others," Blake Scholl, founder of supersonic aircraft developer Boom Technology said on his verified handle on X. "If we cling onto jobs, we'd still be plowing fields by hand out of fear of disruption."

The response from the research community was warmer.

Yoshua Bengio, the Canadian computer scientist and Turing Award winner whose work on deep learning helped make modern AI possible, said the Vatican was right to engage. "The Vatican and other global institutions can and must play a role in the global dialogue on AI to raise public awareness and mobilize society for the challenges ahead," he wrote.

Tanishq Mathew Abraham, founder of the medical AI research centre MedARC, highlighted what he saw as the encyclical's most useful quality: that Leo does not treat AI as inherently evil, but insists technology is "never neutral." "Glad to see a nuanced, well-thought-out take on AI from the Catholic Church," he wrote.

"AI threatens to undermine the basic building blocks of humanity as it seeks to replace our most basic functions, like creativity, friendship, and critical thinking," Senator Chris Murphy of Connecticut said on his verified handle on X, citing the pope's anti-monopoly argument. He called Leo's stance "really important."

Gerald Posner, the author and investigative reporter, was less convinced the letter would change anything in practice, dubbing it "Jesus AI." "I appreciate this historical moment for the Vatican trying to set some guardrails for AI and Silicon Valley," he wrote. "However, from all my reporting, tech is likely to rush past the generalized safety suggestions set out in this massive encyclical."

The Trump administration sent its own signal.

Brian Burch, the US Ambassador to the Holy See, attended the presentation and later said Washington "shares the Holy See's commitment to ensuring AI serves humanity." But he framed the issue differently: the administration's goal, he said, is for American AI to "reflect democratic values rather than authoritarian control" -- a priority that sits in uneasy tension with its simultaneous effort to strip Anthropic of federal contracts over the company's own democratic-values argument.

Christopher Hale, a Democratic politician, said the morning's biggest surprise was the volume of the response. "A lot of folks in the media severely underestimated how much of an immediate bang Pope Leo XIV's encyclical on AI would have," he wrote -- adding that Olah's remarks were "refreshing," and that the Catholic Church had "a lot of global main character energy this morning."

A lot of folks in the media severely underestimated how much of an immediate bang Pope Leo XIV's encyclical on AI would have

A company fighting in federal court over the ethics of autonomous weapons chose the Vatican, the same week, to make its most public case for external moral oversight of AI. That is either coherent or ironic, depending on where you sit.

Anna Rowlands, a theologian from Durham University who also spoke at the encyclical's presentation, told reporters the document should be read by believers and non-believers alike. "The time to talk about AI is now. It is urgent," she said.

The time to talk about AI is now. It is urgent

Olah closed his remarks with a direct ask.

"We need more of the world -- religious communities, civil society, scholars, governments, and indeed all people of good will -- to do what His Holiness has done here: to take this seriously, to look closely, and to push events in a better direction. We need informed critics who will tell the labs when we are failing. We need moral voices that the incentives cannot bend."

The author is the Head of Research and Analysis at Icarus Asia, a Hong Kong-based risk and advisory business

References and Further Reading

References

1. Olah, Christopher. Remarks delivered at the presentation of Magnifica Humanitas, Synod Hall, Vatican City. May 25, 2026. Full text provided by Anthropic as part of its Vatican engagement initiative. All direct quotes attributed to Olah in this article are drawn from this document.

2. Holy See. Magnifica Humanitas: On Safeguarding the Human Person in the Time of Artificial Intelligence. Full text at vatican.va. Document of record for all papal positions cited in this article.

3. Vatican News. "Pope Leo XIV's first Encyclical Letter Magnifica Humanitas to be published May 25." May 25, 2026. vaticannews.va.

4. CNN. "Pope Leo warns of AI fueling warfare in first major theological document." May 25, 2026. cnn.com.

5. CBS News. "Pope Leo calls for 'disarming' of AI in technology-focused encyclical." May 25, 2026. cbsnews.com.

6. United States Conference of Catholic Bishops (USCCB). "In first encyclical, Pope Leo urges world to 'disarm' AI amid increased reliance." May 25, 2026. usccb.org.

7. Axios. "5 ways Pope Leo says AI could warp humanity." May 25, 2026. axios.com.

8. Newsweek. "Pope Leo releases first encyclical: four key takeaways." May 25, 2026. newsweek.com.

9. The Hill. "Pope Leo XIV urges more AI regulation to spare 'human dignity' in first encyclical." May 25, 2026. thehill.com.

10. Krause, Amanda. "What smart people are saying about Pope Leo's letter on AI." Business Insider. May 26, 2026. Source for all third-party reactions quoted in the Reactions section: David Sacks, Blake Scholl, Yoshua Bengio, Tanishq Mathew Abraham, Senator Chris Murphy, Gerald Leo Posner, Brian Burch, and Christopher Hale. businessinsider.com.

11. NPR. "Anthropic sues the Trump administration over 'supply chain risk' label." March 9, 2026. npr.org.

12. CBS News. "Anthropic sues Pentagon, Trump administration over 'supply chain risk' designation." March 2026. Source for autonomous weapons and mass surveillance dispute specifics, and the Feb. 27 deadline. cbsnews.com.

13. Goodwin Law. "Is Claude a Supply Chain Risk? What Federal Contractors Need to Know." March 5, 2026. Source for the $200 million OTA awarded to Anthropic in July 2025 and the scope of the Pentagon designation. goodwinlaw.com.

References and Further Reading

References

Further Reading

Related

Google Finds a Way to Skip the Training Step for Spreadsheet Predictions

History Doesn't Repeat. But the Mellons Do.

The Wall That Private Credit Built