A quieter race

The call to slow AI deserves support. The harder question is how to make restraint work when the incentive to continue does not disappear.

Corporations have already crossed the Rubicon in the race to commercialise AI. Whether they fully understood the consequences or not, they helped turn a technology into a competition for markets, infrastructure and strategic advantage. That competition is no longer theirs alone to control.

Dario Amodei’s We Must Pace the Frontier deserves serious support. The proposal is to give safety work time to catch up with growing capabilities, bring independent evaluators inside laboratories, and pursue coordination between companies and governments. It goes beyond delaying public releases. The essay also explicitly recognises the difficulty of verifying agreements when participants could develop models secretly, including for military use.

That difficulty is not a reason to dismiss the proposal. It is the central problem that any attempt to implement it will have to face.

A company can decide to proceed more carefully. Several companies can agree to do the same. Governments can negotiate common rules. None of those actions, by itself, removes the advantage available to someone who continues.

Even sincere agreement does not eliminate the incentive to break it quietly.

The race has more than one purpose

A company can genuinely care about safety and still fear losing its market. A government can genuinely want stability and still fear that restraint will leave it dependent on a rival. Neither needs to be dishonest for the incentives to conflict.

At Davos in January 2023, Palantir’s Alex Karp said the country advancing fastest in AI capabilities was “going to define the law of the land.” (Reuters)

That is a different motivation from selling subscriptions.

Once leadership is understood in those terms, the calculation changes. The question becomes not only whether a technology is dangerous, but whether the other side will possess it first. America, China, another state, another corporation: the same reasoning is available to all of them. It does not require a conspiracy, only a valuable advantage and uncertainty about what everyone else is doing.

There are two incentives worth separating. One is to develop AI that can be sold. The other is to develop AI whose value comes from an advantage that others do not have.

These are not separate industries. The same laboratory, funding source or model can serve both purposes. But they respond differently to public restrictions. A commercial product needs users and revenue. A capability developed for exclusive strategic use does not need a public launch, a leaderboard or applause. Keeping it unavailable to others may be part of its value.

Figure 1. Two incentives. The same laboratory can feed both.

This creates a possible failure mode for regulation. If restrictions are effective mainly against visible commercial releases, some investment could move towards less visible work. Programmes justified by national security or internal strategic value could become more attractive precisely because they do not depend on public distribution.

That shift is not inevitable. It is a risk that good regulation should anticipate.

The public experience of AI could become much quieter. Fewer launches. Fewer demonstrations. Fewer posts saying, “This model came out two days ago, and look what people are already doing with it.”

Less hype would be welcome. But fewer announcements would not establish that development had slowed. The most consequential work could continue with more concentrated resources and fewer opportunities for outside scrutiny.

The race might become less visible before it becomes less dangerous.

Manhattan did not need an audience

The Manhattan Project is a reminder that public attention is not a prerequisite for extraordinary technological progress. It was a large, secret wartime mobilisation of science, engineering, industry and military resources to produce an atomic weapon. Its work led to the Trinity test in July 1945 and the weapons subsequently used against Hiroshima and Nagasaki. (National Park Service)

Now consider a counterfactual: access to modern AI assistance during that research.

How much faster might parts of the work have moved? What would a weapon available months earlier have meant? What if another country had gained that advantage first? The possible consequences concern the course of a war and the balance of power after it, not a better quarterly result.

AI would not have removed the need for factories, materials and physical testing. There is no defensible number for how much time it would have saved. But even the prospect of a meaningful acceleration helps explain why a government might be unwilling to stop while remaining uncertain about its rivals.

Whether an institution is called a Ministry of Defence or a Ministry of War does not resolve that problem. The name describes an intention. It does not guarantee how the capability will be used, or remove the pressure to acquire it before someone else does.

This is the concern behind a quieter AI race. Public restraint could coexist with private urgency. A government could support a safety agreement while concluding that one particular programme is too important to delay.

A credible framework has to account for that possibility without treating every participant as an enemy. Trust has value. So does the ability to check what is happening.

A nuclear reactor in every home

There is another danger, almost the opposite of a secret state programme: distributing powerful capabilities so widely that access grows much faster than the ability to use them responsibly.

Imagine a nuclear reactor in every home.

The attraction is obvious: enormous useful power. The problem is what happens when that power arrives without the expertise, containment, maintenance and accountability needed to manage it.

The analogy is not about treating every user as malicious. It is about the gap between being able to operate something and understanding the consequences of operating it.

An AI system could make a complicated action easy to request without making its implications equally easy to judge. Reducing the expertise needed to perform a task does not automatically reduce the expertise needed to recognise when it should not be performed.

That creates a different policy problem from the existence of advanced research. It concerns who can access a capability, what they can do with it, and what safeguards stand between an instruction and its consequences.

Neither risk cancels the other. Dangerous capabilities could spread widely while the strongest systems remain concentrated inside institutions with limited external oversight.

Preventing one should not become an excuse to ignore the other.

The same acceleration can improve lives

Now move from the Manhattan Project to NASA and Apollo.

Imagine giving those engineers access to the AI assistance now reachable through a phone: help navigating technical knowledge, translating ideas into code, checking calculations and exploring alternative designs. Think about the years of analytical work and repeated engineering iterations, and how much shorter some of those cycles might have become.

This does not require imagining an entire space programme completed in a week. It requires imagining parts of the intellectual workload compressed from months into days or weeks, while physical construction and validation continue to demand their own time.

There are already narrower, concrete examples. In 2023, NASA described specialised AI-assisted engineering that could produce structural designs in hours and take certain prototype parts through design, analysis and fabrication in as little as a week. Human judgement and established validation processes remained part of the work. (NASA)

Then add substantial funding, dedicated research teams and laboratories. The potential is not only to build more powerful weapons. It is to improve engineering, advance science and solve problems that are too expensive or time-consuming to address today.

CRISPR-Cas9 offers a useful parallel. The medical potential of genome editing is a reason to continue research. It is not a reason to make unrestricted human experimentation available to anybody willing to try. The World Health Organization’s recommendations explicitly combine the potential to treat disease with oversight of research and clinical applications, including mechanisms for addressing unsafe or unregistered work. (World Health Organization)

Supporting that oversight does not mean opposing medical progress. Supporting research does not mean accepting every use of its results.

The same distinction matters for AI. Progress includes treatment, protection, reliability and more comfortable lives. A safety framework should preserve those possibilities while limiting unacceptable risks.

A pause is not the only possible intervention. Unrestricted release is not the only way for research to deliver value.

Less public does not have to mean less accountable

Three questions need to remain separate: what is publicly announced, who can use the technology, and who can independently inspect it.

Figure 2. A launch, or the absence of one, does not tell the safety story.

A system can remain confidential while being subject to serious external evaluation. A public product can be widely discussed without its development being meaningfully scrutinised. Neither a launch event nor the absence of one tells the whole safety story.

This is why embedded independent evaluation is a valuable direction. Its purpose should be to make scrutiny possible without requiring dangerous capabilities, sensitive information or commercially valuable research to be exposed to everybody.

The important distinction is between confidentiality and freedom from accountability.

For that distinction to hold, oversight must follow consequential capabilities and activities, not only products prepared for sale. A powerful system should not become less accountable because it is labelled internal research or reserved for strategic use.

Evaluators would need enough access to discover problems, not merely review selected demonstrations. Findings would need a route to authorities capable of requiring changes. Refusal of access would need consequences. Otherwise, an inspection process could exist formally while leaving the most important questions unanswered.

There is a relevant principle in nuclear non-proliferation. IAEA safeguards use independent verification to assess whether states are meeting their obligations concerning the peaceful use of nuclear material and technology. A declaration is not treated as the whole process. (IAEA)

AI will require its own verification methods. But the question to borrow from nuclear arms agreements and climate commitments is straightforward: what turns a promise into something that can be checked and acted upon?

Who reports? Who verifies? What happens when access is denied or a commitment is broken? Why would a participant find cooperation preferable to gaining an advantage outside the agreement?

Those questions are not administrative details to settle after a declaration. They determine whether the declaration can change behaviour.

There is also a problem of participation. A framework intended to reduce common danger must offer something to parties that disagree about who should lead. A government that believes an agreement merely preserves another country’s permanent advantage will have a strong reason to resist it.

This does not make coordination impossible. It makes reciprocal assurance, credible verification and the distribution of benefits central to the work.

After the box is opened

There is no need to choose between believing that the technology is irreversible and believing that its development can be made safer.

The decision to build carefully can still matter. Independent oversight can still matter. Narrow agreements, enforceable restrictions and better safeguards can still reduce risk even when universal cooperation is unavailable.

But success should be measured by changes in dangerous capabilities, unsafe uses and institutional accountability. A decline in release frequency or public excitement is not enough.

The call for restraint deserves support precisely because the incentives to continue are so strong. The challenge is to turn that call into arrangements that can survive commercial pressure, geopolitical rivalry and incomplete trust.

Pandora’s box is already open. That does not mean nothing can be done.

It means managing the consequences can no longer depend only on the goodwill of those who opened it.

A shorter version of this argument was published on X.