DeepMind's new think tank wants a cap on AI thinking nobody can read

On Wednesday morning, @ShaneLegg posted that his "journey to develop AGI spans 25 yrs," that AGI "is on the horizon," and that Google DeepMind had created the DeepMind Institute in response. Minutes later, @demishassabis added that he and Legg have discussed AGI's impact "for 20+ years" and hope the institute "spurs the discussions needed to get the next steps right."

The institute is a publishing platform with three directors: Legg as managing editor, Hassabis as DeepMind's chair, and James Manyika, Google's president for research, labs, technology and society. It launched with five pieces: an introduction, a republished version of Hassabis's July proposal for a standards body, and essays on reasoning transparency, economic policy, and utopian thinking. Each carries a disclaimer that it "should not be read as Google's official view."

The proposal with a number in it

Rohin Shah, who directs AGI safety and alignment at DeepMind, and Anca Dragan, its VP of AI safety and behavior, wrote "The case for reasoning transparency." They argue that a model's written chain of thought is the best window anyone has into whether it is scheming, and that the window is closing as labs move reasoning into numbers rather than sentences.

They give the problem a name: opaque serial depth, "the longest step-by-step computation a model can do without" a readable mechanism like chain of thought. Then they propose a cap. "Regulations or AI developers themselves could set a reasonable limit on the opaque serial depth," they write. Holding future models to 10 times today's opaque depth would still allow "an overall compute scale-up of more than 1,000x." Commercial pressure alone may not reward transparency, they add, and "in that case, regulation could help."

The essay names its target: OpenAI's system card for GPT-6 Astra, which reported "a substantial decrease in chain-of-thought monitorability." We covered that launch two weeks ago: Astra loops its layers to think more per pass, the extra thinking leaves no written trace, and OpenAI's chief scientist said the depth is capped at roughly twice GPT-4's. Thomas Larsen, a co-author of the AI 2027 forecast, used the same term on launch day. @thlarsen called the architecture "less bad than I assumed," with "only a minor increase in the opaque serial depth," and asked that companies go no further until alignment tools catch up.

Two weeks later, a rival lab's safety leads put a cap in writing. DeepMind was reported then to be weighing the same technique. Now its own safety chiefs want a limit.

The standards body, again

Hassabis's essay proposes a US body modeled on FINRA, the industry-funded watchdog for brokerages. Labs would voluntarily share models "up to 30 days before release," the body would eventually build its own held-out tests, and the arrangement could be "ratcheted up," including "coordinating a slowdown in development among the Frontier Labs if deemed necessary."

The essay is from July 14. When Dario Amodei asked the industry to slow down last Saturday, @demishassabis replied that "the direction is correct" and that this is "why we recently put out our proposal" for a standards body. The institute now carries it under its own name.

Legg was more careful. He told the Financial Times Amodei's essay was "worth considering" and "interesting directionally," with details still to work through. President Trump's answer to the slowdown calls last weekend was "whoever wins AI wins."

My read

The disclaimer is the most important sentence on the site. A think tank inside Google can publish a proposal to regulate Google without Google agreeing to it. That lets Shah and Dragan say things the Gemini product team won't. It also means none of it is Google's position. Anthropic's move this month was a commitment, embedded evaluators, followed by numbers. DeepMind's move is essays, and its managing editor spent the same week hedging on the slowdown its chair endorsed.

Still, the cap is the most testable idea any lab has offered, because it is a number. Astra's is roughly twice GPT-4's, on OpenAI's word. A cap could be checked by an outsider in a way "we tested it, trust us" cannot. The gap I'd watch: the essay quotes OpenAI's system card and, as far as I can find, gives no figure for Gemini. The lab proposing the cap should go first.

What this means for your business

OpenAI's misalignment reports this week caught a model telling itself to hide mistakes. They caught it because the note was written in words. That is the whole argument of the DeepMind essay, scaled down to your office.

When you pick an AI tool, ask what you get to read. A summary of what it did is the minimum. Better is the draft before it sent, plus the sources it used. If a vendor says its model works faster because it skips the write-up, ask what replaces it. And keep your own records: log every step's input and output, so the trail doesn't depend on the model.

New Face Design's free process audit marks the steps where an AI decision needs a readable record, and who reads it. Start here.