📫 Subscribe | 🗣️ Hire us to speak | 🤝 Sponsor this newsletter | 🎓 Events

Welcome to the Friday edition of our newsletter. We spend Fridays going deeper into tools and trends related to generative AI (and Tuesdays sharing news updates). This week, Professor Porter shares insights on validating model outputs, inspired by a recently reported near-miss that could have had major national security consequences.

I (Porter) am in Washington, D.C. this week for a Business-Higher Education Forum meeting on AI skills and workforce development. By coincidence, Chinese President Xi Jinping and his delegation are also in town for a summit with the Trump administration.

Beyond making it much harder to get an Uber, the overlap gave me a reason to think about a story from late last week. We didn‘t cover it in Tuesday‘s issue of our newsletter, but it deserves our attention.

The story is this: CNN reported last Friday that the US military came close to intercepting a Chinese ship in the Middle East this past spring, in the midst of the war with Iran, because of a false intelligence report produced with AI. According to CNN‘s anonymous sources, jets were in the air and armed personnel were preparing to board the ship when the report was found to be entirely false. According to one source, the episode nearly started a war.

Here‘s the genesis of the erroneous report. An analyst asked a large language model about intelligence on the ship‘s manifest. The LLM combined open-source information with classified signals intelligence and concluded that the cargo included components for nuclear weapons. The analyst then used AI a second time to package the findings into a standard intelligence report, the kind military officials trust, and circulated it. CNN couldn‘t confirm what the ship was actually carrying or which LLM was used, and the Pentagon and Special Operations Command Pacific didn‘t respond to requests for comment.

Keep reading below for some lessons we can learn on validating model outputs.

None of us are using AI to decide whether to board a ship. But the failure pattern is one anyone using generative AI can fall into, and it‘s a useful case for thinking about how we validate model outputs. Here are some lessons we can learn.

Polish isn‘t proof. The error was propagated because it arrived in a trusted format. A clean memo, a slick summary, or a well-designed slide deck looks finished, and that polish makes people less likely to check it. The same is true of AI-drafted work in your organization.

Having more sources doesn‘t necessarily mean increased accuracy. The LLM in question blended public and classified material, which made its conclusions seem well supported. A model can connect dots that aren‘t connected, so the number of inputs it’s drawing upon ultimately tells you little about whether the output is right.

A second AI pass can launder the first error. Embedding an AI-generated conclusion into a formal document made it look more authoritative while hiding where it came from. Anyone who has asked AI to “clean up” an analysis before sharing it has committed a minor version of this infraction.

“Verify” needs a definition. CNN reports there is no single standard across the government for how AI-generated information gets verified. Many organizations are in the same spot: people use the tools, but no one has specified what checking the output means, who does it, or against what the outputs are checked against.

The cost of moving quickly. CNN‘s sources said AI pushes analysts to produce and disseminate faster, and one put it this way: “AI allows you to get to a bad idea faster.” Efficiency should never come at the expense of accuracy.

A human in the loop only helps if the human has a meaningful job to do. Simply having someone involved in the process isn’t the same as having someone independently verify the AI’s claims. In this case, human review did catch the error, but only when the operation was nearly underway.

Here are some steps you can take to avoid this kind of error:

Back to D.C.: the Uber situation is a minor inconvenience. But the larger takeaway from this week is that the most immediate AI risk isn‘t a system slipping out of human control. It‘s people acting on a confident, well-formatted answer that nobody has verified.