The problem with "official sources say"
The problem with "official sources say"
AI doomerism, the drone incident that wasn't, and reflections on the importance of informed skepticism.
While I’m going to briefly cover the latest AI doomer story—and how a few incredulous reporters latched onto it despite it being both wrong and meaningless—my real focus here is on the practice of blindly reporting what officials say. Because sometimes officials are wrong. Or lying. Or just parroting something they read somewhere without any real understanding of it. And by sometimes I mean a lot.
What happens when popular coverage doesn’t account for these scenarios?
We reward corrections. See something wrong, misleading, or unfair? Use our anonymous Typeform or drop a comment in this post’s dispute doc.
Paperclips and progress
One nice thing about the potential rise of Skynet-like AI is that during our great sunset approximately two billion people will meet as one to share a final human experience together. Except instead of joining hands to sing Que Sera Sera, they’ll be bullrushing the internet’s town square in a mad race to be the first to tweet out “ah yeah just as I predicted”, even though virtually all of their predictions were useless in the actual business of advancing AI without creating Skynet.
This isn’t to say that AI safety isn’t important! It certainly is! I deeply appreciate all the competent and passionate professionals working to understand and control for the possible dangers of what a rogue AI could do. But most AI doomer takes aren’t about AI safety really, in that they’re almost never about controlling for any specific risk, nor are they generally attached to any greater proposal than just “shut it all down”. Most are just halfhearted restatements of Nick Bostrom’s famous thought experiment from 20 years ago:
Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans.
Scary! As was Jurassic Park when I was 8. But the moral I take away from the latter as an adult isn’t “let’s never push the boundaries of DNA science”. It’s more just “sure, yes, Frankenstein godzillas the size of the Eiffel Tower should prolly have a kill switch”. While there may be some dark sciences where the risks of exploration outweigh the rewards1, AI risks mostly remain distant and theoretical while the rewards are present and meaningful. We of course can and should eg. air gap critical infrastructure and take other reasonable precautions as we press forward. But the point of AI safety is progressing safely, not refusing to progress out of broad fear. To do otherwise would have left us with faster horses as less timid nations got to the moon.
Terror in the skies
A bunch of tech reporters got duped this week. Excerpting from VICE’s original take:
An AI-enabled drone killed its human operator in a simulated test conducted by the U.S. Air Force in order to override a possible "no" order stopping it from completing its mission, the USAF's Chief of AI Test and Operations revealed at a recent conference.
At [a summit…] held in London between May 23 and 24, Col Tucker ‘Cinco’ Hamilton … held a presentation that shared the pros and cons of an autonomous weapon system with a human in the loop giving the final "yes/no" order on an attack. As relayed … in a blog post for the host organization, the Royal Aeronautical Society, Hamilton said that AI created “highly unexpected strategies to achieve its goal,” including attacking U.S. personnel and infrastructure.
“We were training it in simulation to identify and target a Surface-to-air missile (SAM) threat. And then the operator would say yes, kill that threat. The system started realizing that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.” …
It then came out that, no, this is not what happened. Per a later update, it was just:
a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation
Mind you, I don’t fault the reporters for Colonel Tucker’s fuckup here. They didn’t misreport what he said. They did indeed copy and paste correctly from the source blog. The issue is that they took it uncritically without even beginning to contextualize why this story, even if true, was almost entirely meaningless.
Put another way, the story was silly because even if this was the result of a real simulation it would only be evidence that the simulation wasn’t based on conditions you’d find in the actual world. It’s equivalent to saying “I ran a simulation where Lewis Hamilton downed eight shots of tequila and took two massive bong rips and then crashed his car on the first lap”. Sure, that very well might be the result! But it’s also irrelevant in that it would never happen2. Because safety controls exist. And, sure, sometimes they aren’t quite good enough. Any reporting that thoughtfully exposes faulty controls and advocates for better ones is vital and laudable! But we don’t have here any detail about real safety controls. And the embedded presumption is that the drone engineers had just kinda…never heard of the paperclip problem (which they very, very, much all have). Instead of meaningful commentary we got:
Outside of the military, relying on AI for high-stakes purposes has already resulted in severe consequences. Most recently, a judge was caught using ChatGPT for a federal court filing after the chatbot included a number of made-up cases as evidence. In another instance, a man took his own life after talking to a chatbot that encouraged him to do so. These instances of AI going rogue reveal that AI models are nowhere near perfect and can go off the rails and bring harm to users.
First, it was a lawyer and not a judge. Also the AI didn’t go rogue. The lawyer did. Second, the man took his life because he was profoundly mentally ill, not because of AI3. But in general, sure, taking medications in a manner other than described on the label can harm you! That doesn’t mean it’s a bad idea to manufacture medicine or painkillers! Doomerism isn’t just needlessly pessimistic; it’s anti-progress and lazy. Want to work on AI safety to ensure things go well? Godpseed! Want to critique specific practices or blind spots! Please do! But if your addition is just “ah man Jurassic Park ended badly”, what value are you adding exactly?
Why expertise matters
What a reporter’s job is in any precise sense is a source of much debate, and is too subjective for me to say much about here one way or the other. What we can establish though is the utility of different types of reporting.
In rough terms:
Reporter A attends press conferences to (maybe) proffer a question and (mostly) accurately write down what was said.
Reporter B writes down the same things, then contextualizes said statements with informed commentary as to whether the quotes are actually bullshit.
While most reporting work falls somewhere in the middle, it seems trivially true to me that the second part of A’s job is basically just…retweeting?
Like:
If you can ask questions—and especially if you can demand answers—that’s great! That’s the first step in holding truth to power etc. (Huge props to local reporters who do this well!)
But most questions on national/global stories are lame, and most answers go unchallenged. And—as was true in this situation—sometimes the story is printed before any follow-up questions are actually answered.
So even allowing that A can sometimes be useful, it’s just mechanically true that B is always more useful. Or at least it is if done well. But doing it well requires a level of expertise that can’t be delegated to third parties. While a million outside experts are happy to supply quotes in either direction on these kinds of stories, if you don’t understand AI well enough to understand that the scenario described—true or not—just wasn’t news, how could you possibly judge which experts to quote? And how can you possibly add the right gloss to guide the reader towards the right takeaways?
PS - While it can be tempting to point out all the outlets that passed on this story, I’d counter with “ok and how many of them published a good take on why the story was meaningless either way?” While I of all people can intimately appreciate why calling out bad journalism is not the smoothest possible social experience, ceding the news cycle to those who get it wrong isn’t at all the same as being right.
(Standard warning against paid subscriptions. Output here is high effort but very scattered. Support is appreciated, but free subscriptions are also great support.)
While I remain skeptical of the lab leak theory as a matter of history, safety protocols certainly have their limits and we should be ultra-cautious of eg. bringing back smallpox out of the hubris of thinking that Level 4 safety measures will always be followed as outlined. Viruses are very small and you can’t attach a kill switch. But it’s not obvious to me that many areas of scientific development are like this.
Though I’ll allow that there’s a case where simulating this could be useful so far as understanding what might happen if some incompetent rogue actor (say a low-tech terror cell) reprogrammed a drone to inflict some idea of maximum chaos. But that’s not really a point about AI safety. It’s a point about terrorism.
At the point you “[appear] to view [a] bot as human”, even perfect AI controls aren’t going to solve the underlying issue. And for anyone who has ever been on the internet, it’s not like humans on the internet are consistent fountains of optimism and factual clarity! If we want to scapegoat AI here, we should at least present the data in comparative terms. This man may have killed himself after talking to a chatbot. But how often has this happened? And how does it rank against how many people kill themselves after talking to other people? An obvious analogue here is self-driving cars. Sure they sometimes kill people. Is that a good reason to shut down their development? To make that case you at the very least need to consider their fatalities caused vs human drivers, and the improvement curve. Let’s improve chatbot safety, sure. But let’s also be reasonable and less sensational about it.