The AI industry appears to be having its loudest debate yet over whether its technology poses an existential threat to humanity.
The current discussion began after AI researcher Jacob Coxon said he has resigned from Anthropic because he worries that major AI companies are “playing with our lives.” Then Anthropic’s lineup bowed and stepped in with a post stating“We really seriously believe that AI could kill all humans!” and adds that he personally believes the probability is “>10% in the next decade.”
In the last episode of TechCrunch Equity PodcastKirsten Korosec, Sean O’Kane and I discuss the latest apocalyptic warnings. I tried to explain why I’m skeptical of many fatalistic AI narratives, as Kirsten asked if this was “just a weird way to show how advanced your company’s AI model is,” particularly as these companies prepare to go public.
And Sean wondered how these concerns might show up in Anthropic’s S-1 filing for its IPO: “Are there young lawyers right now going through and having to rewrite that entire section of the S-1 filing to say, ‘It is Anthropic’s official position that there is a greater than 10% chance that we could develop something that would eradicate all of humanity and that would be materially bad for our business’?”
Read on for a preview of our conversation, edited for length and clarity. (Note: We recorded this episode before Anthropic CEO Dario Amodei. published its plan for more cautious development of AI.)
Sean O’Kane: It’s hard for me to think of something that blew up so quickly. Not only did this warning shot come from this young researcher who also worked at OpenAI, but it was also immediately shared on Exclamation mark!
What a weird vibe. That was just a ton of accelerator on an already complicated post or series of posts. After the Hugging Face hack of OpenAI’s internal model, plus the increased capabilities we’ve seen with the latest models released by Anthropic and now OpenAI with Astra a few weeks ago, I think this was perfectly timed to be a powder keg for this young researcher.
Antonio Ha: Just to disagree with you, I think if you think AI could destroy all of humanity, that deserves an exclamation point. I’d say that’s a perfectly well-used exclamation point!
My problem with that tweet was more the “we.” Who is the “we” here? To what extent can we talk about a kind of AI community or an AI research community as a monolith? And the probability greater than 10% is just a made-up number, it doesn’t mean anything. Sometimes [there is] This habit both in the tech industry and elsewhere of just throwing out these percentages, they are not based on anything nor are they calculated based on anything. [In retrospect, I realize the tweet was probably referencing the concept of P(doom), but I still think it’s silly.]
One thing I will say about Coxon’s statement and decision is that there’s a recurring theme in Equity, when someone like Sam Altman or Dario Amodei is doing this fatalistic narrative, there’s always this element of: Okay, so why are you doing what you’re doing? If you really believe that [AI could destroy humanity]You wouldn’t continue doing this.
[Whereas] In reality, this is someone who puts his professional background in what he says. You’re actually saying, “I think this is very, very, very bad and I don’t want to work on it anymore.” And so, congratulations for having the courage to do that, at least.
Kirsten Korosec: Yes, I put it in a separate field from everyone else who says that and talks about the dangers.
I’m going to put on my speculative hat, because I want to ask you both a question, which is: Is it possible that every time we see the growing number of blog posts about another incident where one of your AI agents accidentally breaks in, or they talk about how humanity is at risk, this is some strange way of showing how advanced your company’s AI model is?
I mean, that sounds very cynical, but it accomplishes that purpose. Which is: if these AI models weren’t advanced, weren’t capable, and weren’t innovative, we wouldn’t have to worry about these things, right? It’s like a very strange way of bragging about the capabilities of the models you’ve created within your own company.
Antonio: I’ve definitely wondered about this. I don’t think it’s completely cynical, in the sense that I don’t think it’s just a very conscious marketing strategy across the board. I think when a lot of these people (whether they’re researchers or CEOs) talk about it, they really feel a concern.
But of course it aligns with [their] commercial interests in many ways, to say, “Wow, we’ve created the deadliest software ever created.” I don’t want to get too psychoanalytic here, but others have pointed out that there’s this temptation on a personal level to: Of course, you want to believe that what you’re working on is the most important and dangerous thing in the world.
be: What comes to mind when I think about that question is that there’s certainly an element that makes it seem like, “Okay, we’re doing this thing that’s very capable and that’s good for us in some way, even if it looks bad from a lot of different points of view.”
I think what’s different about some of these more recent examples is that you really get the sense that these companies don’t have control over these things in certain ways, especially with the OpenAI stuff.
We continue to see more and more reports about other internal agents who have accessed different wikis on the web and they leave messages for each other, and in a way that doesn’t seem like OpenAI is handling it competently. I imagine there would be a little more polish to the story being told, if it were purely about making people believe that, “My God, they’ve done something so incredibly capable.”
The other thing that I think is really fascinating about this, in particular, [is] We’re a few weeks at most away from seeing Anthropic’s S-1 filing for its IPO, and only a couple more weeks or months away from a potential IPO.
And the idea of you coming out and saying these things in clear language before an IPO, I’m very interested in what that means for that process. How much of this type of thing had they already written in the S-1 and the risk factors within that document? Are there young lawyers right now who are reviewing and having to rewrite that entire section of the S-1 filing to say, “It is officially Anthropic’s position that there is a greater than 10% chance that we could develop something that would eradicate all of humanity and that would be materially bad for our business”?
Kirsten: You’re assuming it’s not there yet.
be: However, that’s what I’m saying: is it already there and being reformulated? Or is this something that is a real struggle? There must have been language there. It’s one of the reasons I’m so looking forward to reading this document in a way that, in some ways, goes even further than SpaceX. [S-1]because I’m sure there will probably be things specific to these ideas that will be interesting to see.
Kirsten: Here’s the thing: In a traditional investment environment, one might believe that language like this would hurt a company’s valuation, because it suddenly becomes dangerous. But we don’t live in normal times.
And again, going back to my point, it could end up being a strange beneficial flex for the company on the valuation side. It’s not the same as the whole anger-inducing trend we saw last year, but it’s in that same universe, let’s say, where something’s strength, ability, and even danger elements equate to a high rating. So I guess we’ll see in a few weeks.
Putting that aside for a minute, what is being done about it? And can we control this? The American CEO of a non-profit organization called ControlAI, Connor Leahy, was on the show this weektalking about this. So what are they paying attention to in terms of how to control the dangerous aspects of AI, or are we throwing up our hands and seeing how it all plays out?
Antonio: I don’t necessarily have a great answer to this myself, but I’ve been thinking about some aspects of this debate and perhaps why I respond the way I do.
To echo one of Sean’s points, I think part of what this speaks to is the extent to which these major AI companies feel like they no longer have control of these models. That’s definitely not cool. That’s something we should all be concerned about.
I think part of the reason I’m skeptical of or resistant to the fatalistic narrative is because it reaches this level of hysteria of “Wow, this could destroy humanity in the next 10 years.” It’s a bit of a distraction from the more immediate harms AI can have, whether related to work, the environment and climate.
Ideally, I think we should be able to discuss all of these things and have regulatory and other safeguards against all of these things. [including AI’s existential threat]. But once you start using phrases like AGI and superintelligence, that just sucks all the oxygen out of the room in a way that’s not very useful.
When you purchase through links in our articles, we may earn a small commission. This does not affect our editorial independence.