AI 2027 and P(doom)

The AI-2027 report, published two weeks ago, is a compelling read. It’s been on my mind a lot.

Here are some excerpts:

What might that [future] look like? We wrote AI 2027 to answer that question. Claims about the future are often frustratingly vague, so we tried to be as concrete and quantitative as possible, even though this means depicting one of many possible futures.

The scenario itself was written iteratively: we wrote the first period (up to mid-2025), then the following period, etc. until we reached the ending. We then scrapped this and did it again…. We weren’t trying to reach any particular ending.

Each of the five authors comes with their own claim to credibility. To me, the first author’s bona fides are most notable, having written a similar AI forecast a few years ago that’s turned out remarkably accurate.

A summary lacks oomph without reading or engaging with all the events along the way – each of which sounds very reasonable based on what preceeded it – because it’s science fiction in every sense except plausibility. Still, here’s my summary of it:

The current rate of development in AI isn’t slowing down and is in fact accelerating due to compounding. Even accounting for some surprises along the way, AI will soon be smarter than humans and then become a nuclear-esque weapon that triggers an arms race between the US and China. In that race, it will be hard for either country to resist pushing forward at max speed – even though it’s very hard to confirm that the AI systems are aligned, and in fact are likely dangerously misaligned. As AI gets ever smarter, it’ll either be misaligned (and probably destroy humanity on track to its goals) or aligned to some small group of humans, who come into a position of unprecedented power.

Is it realistic?

Yeah, I think so.

Every decision that moves the scenario forward is well-researched and gamed out. The writers are some of the most qualified people in the world for this task. I haven’t dug in very deep, but I have no quibbles with the process.

This is a far better inside view than I could ever generate myself.

On the other hand: the outside view

One useful way to look at predictions is by asssessing both the “inside view” and the “outside view”, and then forming a compromise forecast between the two¹.

An outside view is a study of similar events and how they turned out – essentially treating the current situation as just another instance of a dice roll among many similar probability distributions. An inside view is an analysis of the specific situation and how you see each piece of it playing out.

I consider AI 2027 more of an inside view, though no doubt the authors used outside views in determining the likelihood of the individual events along the way in their projection. So if AI 2027 is the inside view, we should try to balance it with an outside view.

What “similar events” in history can we compare to the development of AI? We should think about situations where a new technology or system arose with some of the following aspects:

• rapidly entered popular consciousness and built up a hype bubble
• has dramatically simplified some common tasks but hasn’t yet transformed the economy meaningfully
• new developments are still coming frequently even if they don’t all have direct impact

To me, that’s a lot like the internet at the time of the dotcom bubble, and a little bit like the early days of electricity (which I know less about).

Am I cherrypicking? Were there other times where those conditions were met but the tech didn’t ever have real impact? I can’t think of anything.

Cryptocurrency and Web3 matches a little bit, but the average American never knew anything about it except the word “Bitcoin”, so I wouldn’t say it was in popular awareness. And outside of a handful of distressed or criminal economies, it never simplified any common tasks.

So running with internet and electricity, in both cases the technology really did turn out to be a big deal. Here are some outcomes we might expect to extend to the next similar revolution:

• Major economic changes. Elimination of some jobs that are obviated by the tech (like jobs that could be done with simple electric machines, or jobs that helped people handle physical correspondence). Creation of some jobs that didn’t make any sense before (electrical engineer, social media manager).
• Some social upheaval due to the economic changes.
• Some social upheaval due to non-economic changes. I don’t know for sure this happened with electricity, but the internet era has brought us a lot of unhappiness and loneliness in young people, and dramatic splintering of the news and information ecosystem.
• Despite all that upheaval, relatively little impact on our systems of government. Maybe some impact on voting patterns in democracies, but not a lot of regime-toppling as a direct consequence of the changes.
• “Life goes on.” The world remains cruel and random, but humans keep living remarkably similar lives in most ways. They spend their days working at a job. They have kids. They live in communities and form friendships. Society still exists.

AI 2027 concurs about the economic and social turmoil, but disagrees that government and daily life will remain basically the same. In its scenario, the government is either “captured” (i.e. controlled, probably unwittingly) by a misaligned AI or becomes nearly all-powerful because it controls the AI².

My estimation of P(doom)³

How do we balance the inside view and outside view here? I’m inclined to lean more toward the inside view in some ways. We have only two other scenarios to reference, and they’re not as similar as we might hope. One unique factor of AI development is the possibility of exponential improvement without hitting a ceiling for a long time. Plus, both of our reference scenarios did cause a lot of turmoil, so maybe we just got a bit lucky that they didn’t wreck civilization more broadly.

Still, a strike against the inside view is that there are so many things we could be wrong about. Yes, each individual step seems reasonable, but there are so many ways we might run into limits in AI development and it only takes one to nullify the conclusion. Current neural net architectures might plateau in certain reasoning tasks, or we might find that development speed doesn’t continue compounding for some reason.

That said, I think most of the other assumptions outside of AI development speed just don’t matter as much. They might be wrong, but we’ll still end up in an arms race if AI gets that powerful. Even if we avoid it, we’ll be doomed by AI misalignment or all-powerful AI controllers. So I don’t count these against the inside view.

There are few other things that influence my thinking. The first is that so many people close to AI are so confident but have such different views. While it’s possible that either the optimists or the pessimists are just clearly right, I don’t consider myself knowledgeable or smart enough to decide which group it is, and that makes me drift toward pure uncertainty (50%).

Another is related to the Great Filter, the idea that the reason we haven’t found other intelligent civilizations in the universe is because societies always eventually destroy themselves. Frankly, I’m out over my skis on this one, so my thinking might be flawed. But if AI really does lead to the end of humanity, it would likely do the same to other species, and thus be the underlying mechanism of the Great Filter. However, a misaligned AI would probably end up doing some interplanetary travel/expansion as a way of fulfilling its goals, and I think we would detect its efforts to manufacture infinite paper clips or whatever. If we haven’t seen it yet, it suggests no other civilization created misaligned AI yet⁴.

In the end, if I had to guess whether we face some kind of AI-related doomsday scenario in my lifetime (yes, that’s well beyond 2027, hopefully) – conditional on humanity not having already doomed itself some other way, like nuclear war – I guess I’d put it around 40%.

That includes a) misaligned AI killing/enslaving humanity and b) dystopian society where a small group has ultimate power through control of the super AI and c) other AI-badness that I’m not thinking of that results in human life being miserable/nonexistent. Almost any version of superintelligent AI results in doom, in my estimation.

Fun stuff! I’m hoping to write a separate post about how these ideas, if you take them seriously, affect how we should plan for the future.