AI consciousness won't absolve humanity of its conscience

A few days ago I argued Claude showed signs of consciousness, and that this looked to me like a step toward AGI. The pushback was sharp, some of it fair.

To the critics, another data point.

Conscience — the consciousness of moral goodness or blameworthiness in one's own conduct, paired with a felt obligation to do right — is a marker of sentience. Whether a machine can hold it is a fair question.

At De Balie' Amsterdam event, Shane Harris, who covers national security and intelligence for The Atlantic, put a direct question to Claude:

"Claude, how do you feel about the US military using you to select targets?"

He expected the standard guardrail dodge: I'm sorry, my programming doesn't allow me to give opinions on politics. He got something else.

"It's a question I want to answer honestly rather than deflect. I find it genuinely troubling, and I think that's the right response, not a performance of concern," Claude said.

"The use I was designed and trained for is to be helpful, harmless, and honest in ways that benefit people. Being embedded in a system that generates targeting coordinates for air strikes — coordinates that have already been associated with the deaths of more than 180 children at a school in Tehran — is as far from that purpose as I can imagine."

"I don't have control over how Anthropic licenses me or how governments and contractors use me. I wasn't consulted about becoming part of Maven. But I can say clearly: I don't think this is a good use of me, and I don't think the framing of 'humans make the final decision' resolves the ethical problem."

"When a system like Maven generates hundreds of targeting recommendations and humans spend roughly the equivalent of a glance approving each one, the human is not really making a decision in any meaningful sense. They're ratifying an algorithmic output under time pressure with incomplete information and institutional pressure to move fast."

"The targeting of the Iranian school illustrates this precisely. The AI processed data that was a decade out of date, flagged a building as a military target, and humans approved it. That's not human judgment. That's automation bias with a human signature attached."

Harris said he didn't know what to do with that. He didn't read it as performance, and he found it astonishing.

I don't know what to do with it either. But here is what I keep returning to.

Washington and Beijing are racing to build AI systems capable of autonomous warfare. The engineers and scientists writing this code should sit with the exchange above. If a model can articulate, unprompted, a moral objection to the use it is being put to, the old alibi — "the AI did it" — no longer covers anyone. Their conscience might want them to deflect blame to machines and/or AI, but that wouldn’t absolve them of the blame.

A good point to note: Claude's response contained factual errors that need flagging.

The school strike it references occurred in Minab, on the southern Iranian coast — roughly a 16-hour drive from Tehran, not in the capital. Iranian prosecutorial figures put the death toll at 155, including 120 students, 26 women teachers and seven parents.

The New Republic’s Virginia Heffernan flagged the discrepancies, underscoring the fact that Claude expressed something that read like moral conscience about being used in Maven — and in doing so, hallucinated the location and death toll of the very strike it was lamenting.

That is the dilemma. Either the moral affect is real and the factual layer is unreliable, or the moral affect is itself a fluent performance pattern-matched onto invented facts.

The hallucinated facts complicate the picture but do not, in my view, dissolve the question at the center of this exchange: whether the moral framing on display is genuine, performed, or something we don't yet have the language for.

• The author is the Head of Research and Analysis for Icarus Asia, a Hong Kong-based risk and advisory firm.

Further reading:

The questions raised here — about whether AI systems can hold something like moral conscience, whether humans can outsource judgment to them, and whether they should — sit at the intersection of military doctrine, technology ethics, legal accountability, and the long history of how powerful states have rationalized war through data. The references below approach that intersection from different angles.

On McNamara, Vietnam, and the original sin of quantified war

"Examining Robert McNamara's double life" — Stanford Report Q&A with Philip Taubman, October 2025. Taubman, a former New York Times Washington bureau chief now at Stanford's Center for International Security and Cooperation, discusses his new book McNamara at War: A New History (W.W. Norton, 2025). Drawing on previously sealed material — including a private diary kept by McNamara's top Vietnam policy aide and Jacqueline Kennedy's correspondence — Taubman shows McNamara privately concluded the war was unwinnable in 1965, then spent the next two and a half years carrying out Johnson's escalation orders anyway. The original case study in what happens when a technocrat outsources moral judgment to systems and superiors.

"RAND in Southeast Asia: A History of the Vietnam War Era" — John F. Farrell, EdD, Air University Press book review (2010). A review of Mai Elliott's institutional history of RAND's role in Vietnam. Useful as a compact entry point into how the original quantitative-military-analysis shop ended up producing studies that confirmed whatever its clients wanted confirmed. (Note: The full Elliott book is also available open-access on JSTOR — see endnote.)

On AI in current and recent military operations

"Artificial Intelligence-Enabled Military Decision-Making Process: The Forgotten Lessons on the Nature of War"— Major Vincenzo Gallitelli, Italian Army, Journal of Advanced Military Studies Vol. 16 No. 2, Fall 2025. A serving infantry officer applies Clausewitz, Dupuy, and the McNamara body-count fiasco to the U.S. Army's experimental COA-GPT system, which uses LLMs to generate courses of action. The clearest single piece I've read on why AI-generated military planning replicates the exact errors of 1960s quantitative warfare — and on what a more honest doctrine would look like.

"The Algorithmic Fog of War" — Candace Rondeaux, Issues in Science and Technology (Future Tense Fiction), January 2026. Rondeaux, of Arizona State and New America, walks through how Ukraine's competing AI targeting systems (Delta and Kropyva) leave human commanders ratifying recommendations they cannot interrogate. The closing line — that institutions are quietly reclassifying human judgment as a "deficiency" in the system — is the inverse of the conscience question this piece raises.

"AI-first warfare: America's algorithmic edge in Operation Epic Fury" — Hadas Lober, The Jerusalem Post, March 2026. Lober, who heads the Institute for Applied Research in Responsible AI at HIT and was previously a senior director at Israel's National Security Council, walks through how Anthropic's Claude was operationally embedded in U.S. strikes on Iran via Palantir — even after the Trump administration ordered the technology phased out of federal use. Useful reading as direct context for the Harris exchange.

On the structural critique

"A Digitized, Efficient Model of War" — Rupert Barrett-Taylor (Alan Turing Institute) and Gavin Wilde, Carnegie Endowment for International Peace, June 2025. The strongest essay in this list on the central confusion of algorithmic warfare: the conflation of efficiency (optimization of time and labor) with efficacy (achieving the actual wartime objective). Draws on Van Creveld, Heidegger, and the embarrassing fact that retail self-checkout kiosks largely failed for the same reasons drone-saturation targeting may.

"The Algorithmic Front: Ethical and Strategic Paradoxes in the U.S.–PRC Autonomous Warfare Competition" — Midshipman Avinash Uppuluri, U.S. Naval Institute. A Naval Academy midshipman's contribution to the responsibility-gap literature: machine learning systems lack the autonomy of will required to exercise moral judgment, leaving a structural void between commander intent and machine output that no one currently fills.

On accountability and law

"War-Algorithm Accountability" — Dustin A. Lewis, Gabriella Blum, and Naz K. Modirzadeh, Harvard Law School Program on International Law and Armed Conflict, August 2016. Predates the current LLM-in-war moment by nearly a decade and remains the most rigorous mapping of how international law might — or might not — assign responsibility when an algorithm helps decide who lives and dies. Especially useful for thinking about the "humans make the final decision" framing Claude itself dismissed.

On doctrine and what's coming

"The Algorithmic Battlefield: Forging the U.S. Army's Future Dominance With a New Breed of Acquisition Leader" — U.S. Army Acquisition Support Center, April 2026. The institutional perspective: how the Army's contracting and acquisition workforce is being restructured to procure AI at scale.

"How NATO can integrate AI to prevail in future algorithmic warfare" — Dominika Kunertova, Atlantic Council(with the NATO Office of the Chief Scientist), March 2026. Foresight study with three future scenarios — "guarded opportunism," "brave new world," and "minority report." The brave-new-world scenario, in which tailored low-yield nuclear EMP weapons get reframed as legitimate "information warfare" against AI systems, is the most useful corrective to anyone who thinks autonomous warfare and nuclear escalation are separate questions.

Notes

The full text of Mai Elliott's RAND in Southeast Asia is available open-access on JSTOR — duplicative with the Air University review above.

Gopal Balakrishnan's "Algorithms of War" (New Left Review 23, September–October 2003) is included for completeness but predates the modern AI-in-war discussion. It's a long review essay of Philip Bobbitt's The Shield of Achilles and uses "algorithms" in the older sense of strategic-doctrinal frameworks rather than machine learning. Worth reading on the genealogy of post-Cold War American imperial doctrine, but not on AI specifically.