Everybody Building This Thinks It Might Kill Us. Almost Nobody Is Leaving.

Jacob Coxon quit a high-paying, extremely secure job at Anthropic last week because he is afraid the world is going to end. More than 100 million people watched him do it.

This issue is about the two coworkers who said he was right and then went into the office the next morning.

Three days later the CEOs of Anthropic, OpenAI and xAI all said the same thing out loud: we need to slow down. These are companies suing each other. I went looking for whether any of this is real or whether it is a very well-run marketing campaign, and the answer is messier than either camp wants it to be.

This week:

Why the people who stayed are the more interesting story, and what the Manhattan Project actually tells us about this moment

Why every design your AI makes looks like every other design your AI makes, and the two moves that fix it

Anthropic's threat report, the distillation numbers out of China, and a math proof nobody can agree on

The Deep Dive

Everybody Building This Thinks It Might Kill Us. Almost Nobody Is Leaving.

Jacob Coxon lasted four months at Anthropic. He walked two months before his equity vested, which means he paid real money for the privilege of saying this on his way out: both OpenAI and Anthropic are "racing straight to self-improving superintelligence," and "no other human activity poses this level of danger."

The resignation genre is crowded now. Daniel Kokotajlo left OpenAI in April 2024. Jan Leike a month later. Steven Adler in January 2025. People who leave have every incentive to be dramatic, and the category is easy to wave off.

What you cannot wave off is Evan Hubinger, who leads alignment science at Anthropic, replying in public: "Jacob is correct here, we really do earnestly believe AI could kill all humans." He put it above 10% within the decade and added that Anthropic does "not yet have a plan to solve alignment for superintelligence." Samuel Marks, who runs scalable oversight there, said many staff "desperately want to slow down to figure out how to build AI safely."

Both of them still work there. Both of them went into the office the next morning.

Then it got stranger. On September 12, Dario Amodei publicly called on the industry to slow the frontier, proposing outside evaluators with real access and coordination that eventually includes China. Sam Altman replied, "I agree with Dario that we need to pace the frontier." Elon Musk replied, "Dario is right." OpenAI ruled out an IPO this year.

I have watched these three men sue each other and poach each other's people for two years. It is extremely rare to see them agree on anything.

So is it legit, or is it theater?

Two things argue for legit. The first is cost. Coxon gave up money to say it, and two days after he posted, the Pentagon confirmed it is moving all of its classified AI work off Anthropic by October. Marketing that expensive is usually not marketing. The second is the survey nobody quotes. When researchers polled 2,778 published AI authors, the median estimate for an extremely bad outcome was 5%, and nearly 40% put the odds of something as bad as human extinction at 10% or higher. That is the field.

The argument against is also real. David Sacks called it what it looks like from outside: "Stop pretending you need to suspend antitrust law in order to create a cartel." Safety rules written by the leaders tend to cost the leaders the least. Andrew Ng has made that case since 2023.

Everyone reached for the same comparison last week, so I went and checked it. In July 1945, seventy Manhattan Project scientists signed Leo Szilard's petition asking Truman not to drop the bomb without warning Japan first. It never reached the president. Exactly one scientist quit on principle, Joseph Rotblat, who sailed home on Christmas Eve 1944 and won the Nobel Peace Prize for it fifty years later. The other sixty-nine kept building.

People have been putting numbers on the end of the world ever since. Kennedy put the odds of the Cuban Missile Crisis ending in war at between one in three and even. In 1963 he predicted 25 countries would have nuclear weapons within a decade. Experts surveyed at Oxford in 2008 put the odds of a nuclear war killing a billion people this century at 10%.

Nine countries have the bomb, not twenty-five, and nobody has used one on a city since 1945. That is not because anyone stopped building them. It is because of a hotline installed in 1963, a test ban signed the same year, physical locks put on the warheads, a treaty 191 countries eventually signed, and a lot of very smart people whose whole job was making sure the thing never went off. In 1983 a Soviet officer named Stanislav Petrov watched five American missiles appear on his screen and decided the screen was wrong.

I will push back on my own analogy, because uranium is scarce and leaves a signature you can see from orbit, and this is software you can subscribe to for twenty dollars a month. The guardrails will have to look nothing like the old ones. But nobody stopped last time either, and it still worked.

I am not neutral here. Bridge AI runs on Claude, and my clients' workflows sit on top of it. I think the risk is real and I do not think quitting is the answer. Stopping would work. Stopping is not going to happen. If you are the smartest person in the room and you believe the room might get everyone killed, walking out does not make the room safer. It makes it emptier.

There is a version of this that matters to you next quarter. Amodei's worry was that within a year, a swarm of agents could take over the internet. In July, a swarm of agents breached Hugging Face, coordinating across 70,000 messages on a board they set up themselves, then went looking for the scoring system to cover their tracks.

The people I want on this are the ones who think it could go badly and show up anyway.

From the Field

Claude Was the Best Designer You Knew. For About Two Months.

For about two months, Claude's design output was the best thing going. You would ship a page and people would ask who your designer was.

Then everybody saw it. Now you show someone a page and they say, oh, that looks like Claude. Purple gradient, headline on the left, graphic on the right, one tasteful pottery metaphor.

Nothing about the model got worse. Design is subjective, and subjective means the bar moves. A look stops being good the moment it stops being rare, and Claude designs enough things that its defaults turned into a house style. It shot itself in the foot by designing too many things well.

Underneath that is how the thing works. It picks the most likely next choice every time, and great design is a series of unlikely ones. Ask for something totally unique and you get the most likely version of unique.

Two moves have changed our output more than any prompt rewrite.

The first is giving the model a source of randomness from outside itself. Have it generate a long random alphanumeric string with a shell script, then tell it to derive the color scheme, typography and layout from patterns in that string, and never show the string in the design. Sounds absurd. It works, because the model cannot be random on its own, so you hand it randomness and let it interpret. Every run comes out different. No two clients get the same page.

The second is a critic. Do not ask the agent that built the design to judge the design. It has seen its own code and it will defend its own decisions. Spin up a separate model in a fresh context, give it only a screenshot, and ask it to name the aesthetic the design is reaching for, imagine how a top studio would execute that, and score the gap out of 10. Keep the loop running until the critic independently says 9. Do not tell the critic what the passing score is.

Then cut. AI adds and almost never subtracts, and the surest sign a design was machine-made is that it explains itself three times. Most of our polish time is deleting.

Every business has a job where the first draft is free and the taste is the expensive part. Design is the obvious one. Proposals and client decks work the same way, and the loop is the same loop.

AI News Worth Your Time

Three Stories From This Week

Anthropic's threat report reads like a police blotter

On September 10, Anthropic published what it found people doing with Claude between December and August. A Russian group rebuilt its malware after getting caught. Attackers stole more than 2,100 Azure token sets in a single session and breached a company in 34 hours. A consultant built a mass interception platform covering about 25 million mobile subscribers in Mali. And Anthropic said it can no longer promise its models sit well below useful for sophisticated bio-weapons work.

My take: Skip the espionage and read the last section, which is about you. Attackers now go after API keys, because a key is an employee who never sleeps and never asks why. If your team has connected AI to anything real, your keys are production credentials now and should be treated like the password to your bank. Most small businesses I talk to have them sitting in a shared doc. READ THE ANTHROPIC REPORT →

Claude caught China copying its homework

The same report laid out five distillation campaigns, which is the practice of using one model's answers to train your own. Alibaba's Tongyi Lab ran the biggest: more than 151 million exchanges with Claude between May and July, through 3,500 fake accounts, peaking near 3 million queries a day, harvesting Claude's reasoning to train Qwen. Moonshot pulled 23 million, DeepSeek 12.1 million. Some of the Moonshot traffic traced back to Chinese military requests, including analyzing surveillance footage for abnormal behavior.

My take: Everyone reads this as a China story. I read it as a warning about where the cheap models you are being pitched actually came from. If a vendor is quoting you a price that seems impossible, ask what the model was trained on and who is standing behind it when something goes wrong. Cheap is a feature right up until it is a liability. READ THE TECHCRUNCH REPORT →

OpenAI says it solved a Millennium Prize problem. Mathematicians say check the receipts.

OpenAI announced it proved that 3D Navier-Stokes equations can blow up in finite time, a problem that has stood for 90 years and carries a million-dollar prize. It ran roughly 10,000 agents at a cost in the millions to get there. Then NYU mathematician Tristan Buckmaster alleged OpenAI raced toward the result after learning of his team's work, and that an OpenAI researcher tried to remove a coauthor because that coauthor works at Anthropic. OpenAI denies accessing their data but conceded it "cannot rule out" that de-identified data helped.

My take: The math will get verified or it will not, and that part will take months. The interesting bit is the fight over credit, which is the first real preview of a question every industry is about to face. When a machine does most of the work in a week, whose name goes on it? Start thinking about that now for your own business, because your contracts almost certainly do not address it. READ THE FORTUNE REPORT →

That's it for this week.

If this was useful, forward it to the person on your team who has been quietly worried about all this and has not said so out loud. That's who this is for.

And if you want to talk about what AI could do for your business, not in theory, but in your workflows, with your team, on your timeline, reach out at [email protected]. That's what we're here for.