Ms. Frizzle Takes On the Black Box of AI

Seatbelts, everyone. This week we are shrinking the bus.

I have spent years explaining AI, and the phrase I hear most is "it's a black box." Which is true. Nobody, including the people who build these models, can fully explain why a given answer came out the way it did. But "black box" has turned into a reason to stop asking questions, and that bugs me. So this week I want to do what Ms. Frizzle would do. Shrink down and look around.

If you have no idea what I am talking about, I cannot recommend strongly enough that you redeem yourself by watching one of the three great shrinking episodes of The Magic School Bus. Inside Ralphie, where the class shrinks the bus to find out what is making Ralphie sick and ends up in a fight with his white blood cells. For Lunch, where the bus gets swallowed by Arnold and digestion becomes a personal tour. And Goes Cellular, where the bus shrinks to the size of a cell and Arnold turns orange (long story). Watch one tonight. You will feel better about everything.

Also this week: Claude can now send real emails from your real inbox, and I have opinions about what that should and should not be used for. Plus a hacker got into OpenAI through a picture, the President weighed in on AI guardrails, and an AI-written intelligence report almost sent US aircraft after the wrong ship.

This week:

The deep dive: we shrink down and ride through the black box, one stop at a time

From the field: Claude as your email buddy, and where to draw the line

Three stories: hacking OpenAI with an image, Trump on guardrails, and the false report that "almost started a war"

The Deep Dive

Ms. Frizzle Takes On the Black Box of AI

Take it away Ms. Frizzle.

Stop one: the search bar. The bus shrinks, the world goes white, and we land in a big empty rectangle with a blinking cursor. On the other side of the screen, someone starts typing a question. With every tap of their keyboard, another letter appears over our heads, until the whole sentence stretches across the box: “Is it important to wear a seatbelt?” Ms. Frizzle hops off the bus and rolls up her sleeves. She grabs the sentence and starts chopping. Not into words exactly. Into chunks called tokens. Some are whole words. Others are pieces of words, or punctuation.

“How do you know where to cut?” She pulls out a guide labeled Tokenizer. “I’m following a pattern. This tokenizer already has a list of chunks, built by looking for patterns in lots of text. Common words often stay in one piece. Other words get split into smaller pieces. Even spaces can come along for the ride.” She holds the guide against the sentence. “A different tokenizer might cut this very sentence differently.”

Then Ms. Frizzle pulls out a sheet of numbered stickers and puts one on each chunk. “Every token has its own number,” she says. “Like a name tag, but with digits instead of letters. There’s already a list that tells us which number belongs to which token.” She points to one sticker. “Imagine this chunk’s number is 42. Whenever this exact chunk appears, this tokenizer uses 42. That’s all the number means: this particular piece of text.” She turns the chunks over so only the stickers show. Our sentence is now a row of numbers, still in the same order. “Same question,” she says. “Just written in a language the model can work with.”

Stop two: the map room. Ms. Frizzle leads us into a room with no walls, just tiny floating lights. Billions of them. “Each dot is one token. Like a city on a map, except this map is made of pieces of text.” She finds our “seatbelt” dot by its sticker number, then lifts a long strip of numbers hanging off it. “If the sticker is like the city’s name. Then this strip is its coordinates. Thousands of them, and this is what the model actually works with.” Sure enough, “seatbelt” sits near “car” and “crash” and “safety,” and nowhere near “giraffe.”

Arnold asks who decided where the dots go. “Nobody,” she says, and presses a button marked Training. A page of text appears. She covers the next word with her hand, the model guesses, she lifts her hand, and a few numbers on our strip change. The dot slides a little. Then the pages start flipping, millions of times, and the dots slowly rearrange themselves. “Nobody told it to put seatbelt next to car. It just kept guessing the next word, and the map fell into place.” She hands us the strips for our whole question, in order, and we head for the elevator.

Stop three: the stack. She points up. Dozens of floors, and we ride the elevator through every one of them. “Each floor is a layer of the model.” The doors open and our tokens line up in sentence order. “Watch the strips. They are about to change. Right now ‘wear’ has the same strip whether you wear a seatbelt or wear someone out. The model has to work out what these tokens mean together, in this question.”

Beams of light stretch between the tokens, some bright, some barely glowing. “This is attention. Each token pulls in information from the tokens before it, more from some, less from others.” “Important” glows hard toward “seatbelt.” Numbers slide along the beams, mix in, and every strip updates. Ding. Next floor, same thing, starting from the updated strips. “Nothing in here is looking or deciding,” she says. “It is arithmetic. The lights just show you where the weight goes.” By the top floor the strips carry the whole question.

Keesha looks from one strip to the next. “So where exactly is the understanding?” There is no drawer marked understanding. Ms. Frizzle smiles and does not answer. Hold that thought.

Stop four: the exit. At the top there is a single door marked “exit”, and beside it a board covered in tokens, each with a bar underneath. Ms. Frizzle feeds our last strip into a slot and the bars rise and fall. “Each bar is how likely the model thinks that token comes next.” The tallest one reads “Yes.” “So that is the answer?” Ralphie asks. “The beginning of one. We pick one token.” She slides “Yes” through the door, and on the other side of the screen the person who typed the question watches it appear.

Then she clips “Yes” onto the end of our sentence and we head back to the elevator. Another lap. Another token. Then another. A 200-word answer is more than 200 trips through this building, and it keeps going until the model produces a special token that means stop. Ralphie asks if the whole answer was in here somewhere and we just found it. “Nope. Nobody is pulling an essay out of a drawer. One token at a time, and every token is a bet on what most plausibly comes next.” She taps the board. “And likely is not the same as true. Hold onto that one too.”

Everything above is the plumbing. It explains how the numbers move. It does not explain how those movements add up to a useful answer. The mystery is Keesha’s question on stop three. We gave our tokens stickers so we could follow them around, but the billions of learned numbers inside the model did not arrive with little notes explaining their jobs. Nobody, Ms. Frizzle included, can point to the exact floor where “seatbelts save lives” lives.

Researchers are now doing the Frizzle thing for real. Anthropic's interpretability team built what they literally call an AI microscope, and they used it to find individual patterns inside Claude that light up for specific ideas. In 2024 they found one for the Golden Gate Bridge, cranked it up, and Claude started bringing up the bridge in every conversation, including ones about pasta. It was ridiculous and also the clearest proof I have seen that the box is not empty. There are real, findable circuits in there. We just could not see them before.

From the Field

Claude Can Send Your Email Now. Two Things Before You Let It.

As of this summer, Claude can send real emails from both Gmail and Outlook. Not draft them. Send them, to real people, on your behalf.

Two things before you do anything with that.

First, extra caution. A draft that sounds a little off costs you thirty seconds. A sent email that sounds a little off costs you a client. Leave the approval step on. Read what it wrote. On Team and Enterprise plans an admin can turn the approval off entirely, and I would think very hard before doing that for anyone on your team.

Second, be honest with yourself about where 90 percent of your work actually happens. It is email. A dashboard inside Claude or on a website is nice. I have built plenty. But a scheduled task that emails you a daily dashboard at 7 am, a daily recap at 5 pm, a standup prep before your Tuesday meeting, an hourly alert when a number crosses a line, any of that just makes more sense as an email. It lands where you already are. And every bit of it is possible directly in Claude today, no developer required.

Where to draw the line: the email to your boss where you scrutinize every word, this is not for that. The check-in on a project? The email to 14 different apartment buildings asking for more information? That has Claude in your voice all over it. A human still reads the output before it goes anywhere that matters. But the volume stuff, the stuff you have been putting off because it is boring, that is the stuff to hand over.

Why this travels: every business runs on email, and every business has a pile of it that is low-stakes and still not getting done.

AI News Worth Your Time

Three Stories From This Week

A picture hacked OpenAI. The bounty was $6,500.

Security researchers at Hacktron found a way into OpenAI's internal systems in July, and it started with an image upload. They uploaded a specially built HEIC photo to OpenAI's public community forum, which runs on Discourse. A bug in the image library that processes those files let them run their own code on the server, and a misconfiguration in OpenAI's single sign-on setup let them turn that into access to employee accounts. They proved it by getting into an employee's Codex account and submitting a pull request to OpenAI's internal code. OpenAI patched it in about 14 hours and paid $6,500. The researchers used Claude to help write the exploit.

My take: forget the OpenAI part. The lesson for a 40-person company is that the front door was a forum nobody thought of as important, running a library nobody at OpenAI wrote, tied to a login system that trusted it too much. Every business has a version of that. The question to ask this week is "what do we run that we did not build, and what does it have access to?" READ THE HACKTRON REPORT →

Trump: the only guardrail AI needs is a "High IQ" President

Last week I wrote about the CEOs of Anthropic, OpenAI, and xAI all calling for the industry to slow down. On Monday the President responded. In a social media post he wrote that the only control or "guardrails" AI needs is "a STRONG AND SMART (High IQ!) PRESIDENT," and that there is "a SICK conspiracy going on against AI and Data Centers." The data center line is about the growing local opposition to new builds, over power bills and water, which has become a real issue heading into the midterms.

My take: as the experts shout caution, Washington goes the opposite direction. What's new. For a business owner the practical read is that there will be no federal rulebook to hide behind any time soon. Whatever guardrails your company has around AI are the ones you build yourself. READ THE AP REPORT →

A false AI report almost sent the US after the wrong ship

CNN reported that during the Iran conflict, an analyst at US Special Operations Command used an AI chatbot to combine open-source information with classified signals intelligence. The chatbot produced a report saying a Chinese vessel was carrying components tied to a nuclear weapons program. The analyst then used AI again to format it into a standard intelligence report, which made it look official. Armed personnel were preparing to board and aircraft were already airborne before someone went back to the underlying intelligence and found the cargo had been misidentified. One source called the report "entirely false" and said it "almost started a war."

My take: read the sequence again. The AI got it wrong, then a second pass through AI made the wrong answer look polished, and the polish is what got it through the room. That is the copy-paste problem with a warship attached. A well-formatted answer is not a checked answer. Whoever is reviewing AI output at your company needs to be checking the source, not the font. READ THE CNN REPORT →

That's it for this week.

If this was useful, forward it to the person at your company who says "it's a black box" every time AI comes up. That's who this is for.

And if you want to talk about what AI could do for your business, not in theory, but in your workflows, with your team, on your timeline, reach out at [email protected]. That's what we're here for.