---
SUMMARY
Connor Leahy, CEO of Conjecture, joins Peter McCormack to discuss the existential risks posed by artificial intelligence. Leahy explains that despite building advanced AI systems like Large Language Models (LLMs), engineers do not fully understand how they work internally, comparing the process to growing a biological organism rather than traditional engineering. He details the technical breakthroughs of the transformer architecture and the scaling laws that have led to current capabilities. The conversation delves into the "alignment problem," the potential for AI to develop its own goals, and the societal dangers of a "superintelligence" that humans cannot control. Leahy argues for a global pause on AI development and more robust international regulation, likening the current situation to the management of nuclear weapons.

TRANSCRIPT
[00:00] CONNOR LEAHY: So it's very important to understand is that we do not understand intelligence. We don't know how the brain works, you know, we have a bunch of guesses, but we sure as hell don't know how it works. And we sure as hell don't know how these neural networks work either.

[00:11] PETER MCCORMACK: So we've built something but we don't understand how it works?

[00:13] CONNOR LEAHY: That's exactly correct.

[00:15] PETER MCCORMACK: So it's kind of like magic.

[00:17] CONNOR LEAHY: Yes, absolutely. It's kind of like looking into a petri dish. We do not know what our AIs can do until we make them. And even after we make them, like we don't know, like we don't know what ChatGPT-6 can do until it's done. None of the engineers at AI know what it will be able to do until it's done. And this is very, very different from other forms of engineering.

[00:39] PETER MCCORMACK: So even if the AI doesn't kill us all, it can still dethrone us as a species. What is our role? What is our purpose? Where do we exist?

[00:49] CONNOR LEAHY: Exactly. I think this happens before extinction happens. Like the thing I expect to happen is that one day we wake up and we're just not in control anymore. And I don't think we'd all fall over dead or anything like that. I don't think extinction happens right away, but we won't be in charge. We won't be in control.

[01:04] [visual: Intro sequence for The Peter McCormack Show plays with music and stylized graphics of Peter McCormack.]

[01:11] PETER MCCORMACK: Good morning, Connor. How are you?

[01:12] CONNOR LEAHY: I'm doing great today.

[01:13] PETER MCCORMACK: Really good to meet you. So, I'm very intrigued to talk to you because your background is actually working in the engine historically of building LLMs. And we've been talking about AI on this show quite a bit, but I'm conscious I know nothing about the engines and I'm intrigued to know that as somebody who was working helping build these things, what is it that you saw that made you realize this is no longer a tool anymore? So can you just give your background—I don't normally do this—but give your background to the audience so they know who they're talking or who I'm talking to.

[01:48] CONNOR LEAHY: Yeah. So, I've been involved in AI for pretty long, basically since I have a developed frontal cortex, you know, since I was like 16, 17, 18 or something. I guess it wasn't fully developed at that point, but the way I got into this field is I was kind of thinking when I was like 15, 16, kind of like, how can I do the most good in the world? You know, how can I help the most people? How can I solve the most problems? And my thinking was, you know, well, you know, I could try to cure cancer or I could work on climate change or I could, you know, try to do many, many possible things I could do. But what do all these things have in common? Well, intelligence. If I just automate intelligence, then I can solve all the problems. I can cure all the diseases and just fix everything. Great, so I'll go do that. How hard can it be?

[02:31] CONNOR LEAHY: So, I guess my first foray into AI was, you know, I was like 16, you know, I barely understood how anything worked, you know, trying to, as teenagers do, build what is now would be called like AGI. I didn't really know those terms at the time. Of course, none of this worked, didn't really understand anything. This was around like 2012, 2013. And this is kind of when what's now called deep learning kind of started. You know, we started seeing, you know, like good image recognition, we were seeing stuff like AlphaGo, you know, beat players at Go, which was, you know, kind of like even more complex than chess. Stuff like this. Still pre-generative AI as we see it today.

[03:10] CONNOR LEAHY: So I got really involved in this kind of stuff, studied a bunch, kind of—I was very much a self-taught kind of hackery person, kind of just, you know, pick it up as I go. I still thought at this time that, you know, there's still like quite a lot of time until like, you know, the big things are going to happen. Because to kind of give you a feeling of how it felt for me at the time, at the time the way AI worked was the new technique was called deep learning or neural networks. These aren't the only ways you can do AI, but these were like the new hot ones. And they're also what's underlying the current generation of AI systems.

[03:46] CONNOR LEAHY: The way these work is very different from normal programming. So normal programming, you write code, you know, you write line by line, here's what the computer's supposed to do, and then it does that. Neural networks are very different. It's more like you grow them. You give them a bunch of data showing like what they're kind of, you know, like do stuff like this or this or whatever, and then you grow your neural network on top of this data to solve your problem. And it works pretty great on many things, you know, back then for stuff like image recognition or playing Go, nowadays for stuff, you know, like ChatGPT and, you know, generating images and stuff. It's all the same fundamental technique. It's all the same fundamental technique with some small differences.

[04:27] CONNOR LEAHY: So this was very exciting, you know, at the time, of course. But around this time I also, you know, started to think about, well, if we build really powerful AI that's like so powerful it can, you know, cure cancer, that's a very dangerous thing. That's a very powerful thing. And how do you even control that? Like how do we make sure that goes well? And turns out no one had an answer to this. And to this day, no one has an answer to this. So yeah, at this time I still thought we had a long time until the really big stuff started showing, but for me, my like "oh shit" moment was in 2019 with the release of GPT-2. So this is pre-ChatGPT, pre-all of this.

[05:05] CONNOR LEAHY: And it's kind of quaint in retrospect, you know. Now we're all used to we go on ChatGPT and, you know, it can only do math as good as a mediocre PhD student, you know, but like back in the day, you know, there was—these things could barely string together two sentences, you know. You could get like, you know, maybe one, maybe two, you know, good sentences strung together. But when I saw this, it was just, you know, it was like the time has—the time has come. Like fuck, it's happening. And the reason why was that earlier forms of AI were always very brittle. They were very special purpose. So like let's say you want to build an AI that plays Go. Can be done, but like you have to specially make it to do that. You know, you have to like get it the right data, you have to set up the neural network just right, you know, you have to like do a bunch of stuff. If you now wanted it to play, you know, I don't know, Atari games, you have to change a bunch of stuff. You have to rewire the whole thing, you have to feed it new data, you have to—you have to do a whole new thing. You know, there's very little transferability.

[06:03] CONNOR LEAHY: And GPT was different. GPT was a general-purpose pattern learner is what the crazy thing about it was, is that as you fed it more data and as you gave it more computing power, as you made the neural network bigger, so to speak, it learned first, you know, how to spell words, then it learned how to do sentences, then paragraphs, then more and more, without humans telling it any of this. It just figured it out by itself. And this was unprecedented. And so I got involved with building these things as well because my thinking was, well, I need to understand this. This is the most important thing in the world.

[06:37] CONNOR LEAHY: So I came from an open-source kind of, you know, world, you know, hackery kind of world. So I was hoping that if I could build open-source, you know, LLMs and tools and neural networks, then, you know, me and other people could study them, could try to understand them, try to make them safer. So I led a group called EleutherAI, which was a very large open-source group at that time where we built some of the only—to this day still only and very, you know, at the time biggest open-source large language models.

[07:06] PETER MCCORMACK: Wow. Okay. Gosh, there's so many things I can ask you here. But was there a specific moment that you can think back where you realized, "I can no longer work as a coder hacker on these and I have to rethink how I'm going to spend my time"?

[07:24] CONNOR LEAHY: I think there were two moments. Two important moments. The first one was when I was burnt on open source, where I realized I can't work open source anymore. And the second one when I got burnt on technology, like technical work in general. And those were about five years apart. So when I went into EleutherAI, it was a very different world. You know, ChatGPT wasn't really a thing and people were not taking LLMs seriously. You know, academics were publishing papers talking about, "Oh, it's all fake, oh, it's not important, don't—don't talk about it, no one cares, blah blah blah." But to me it was so obvious that this is going to be the biggest thing ever. These are general-purpose pattern learners. This is the thing. This is the Holy Grail of AI that everyone was waiting for.

[08:01] CONNOR LEAHY: And like because when I was 16, you know, when I tried to build AGI, you know, general intelligence, you know, the way, you know, a 16-year-old tries to build something, you draw like a little graph on a piece of paper and you're like, "Okay, well I need something for memory, I need something for action, I need something..." And there was always one big box missing, which is the general pattern learner. The thing where you just put data in and it learns the patterns. No one knew how to do that. You know, like there were techniques but they were all terrible, none of them worked. And this worked. This evidently did work and it scaled.

[08:30] PETER MCCORMACK: What was the breakthrough?

[08:31] CONNOR LEAHY: So the breakthrough is a mixture of what's called the transformer, which is a specific way to build a neural network. So neural networks, you can kind of wire them up in a bunch—like infinitely different ways, you know. You can make them, you know, more this shape or that shape, you can add different parts, bits—kind of Lego pieces. It's kind of like you can think of it like Lego. You have like these different Lego pieces and you can stack them in different orders, you can add new pieces and so on. And in 2017, a group at Google discovered a new way to kind of build a neural network called a transformer. And it changed everything. All the neural stuff you see today, whether it's AI, you know, image generation, voice generation, you know, ChatGPT, all of this is based on what's called the transformer.

[09:14] CONNOR LEAHY: We can go into the, you know, technical details of it.

[09:15] PETER MCCORMACK: Yeah, please. Yeah, I'm really interested.

[09:18] CONNOR LEAHY: It's quite boring and unsatisfying, but I'm happy to talk a little bit about it. So the thing with—I'm happy to show you what a transformer looks like or whatever, but there's a very important thing to understand, is that we don't understand how neural networks work. This is very important. No one does. No.

[09:33] PETER MCCORMACK: So we've built something but we don't understand how it works?

[09:35] CONNOR LEAHY: That's exactly correct.

[09:37] PETER MCCORMACK: So it's kind of like magic.

[09:38] CONNOR LEAHY: Yes, absolutely. It's kind of like looking into a petri dish. You know, you see a petri dish and there's a bunch of like, you know, goop going around and doing a bunch of stuff. And you know, we know some things about the goop, you know, we know some things about cells, we know some things about DNA and stuff. Fundamentally we don't really know how it works. Like not really. And we can't like do arbitrary things, right? Otherwise, you know, we'd have cured—we would have already cured all diseases. And it's very similar with neural networks.

[10:03] CONNOR LEAHY: So when you think of a neural network, the way you should think of is a billions, even trillions of numbers. So you have a bunch of numbers, you know, like 1.235, you know, whatever in like 16 digits. And then, you know, 0.892, whatever, you know, and you have a trillion of these. And if you multiply and add them all in the right order, your computer can talk. And now what do these numbers mean? The transformer, you know, I—you can write—you can look at it at a piece of paper, you know. You have like some parts where it's doing some processing, some other parts that's called attention, so where it's like paying attention to different parts and so on. But we don't know what any of this means. Like we don't know. We don't know what's going on. You know, we have some guesses at some of it, but recently the CEO of Anthropic, Dario Amodei, said on a podcast that he thinks we know maybe 3% of what happens inside a neural network. And I personally think that's an overestimation.

[10:55] PETER MCCORMACK: Does this make sense to you?

[10:56] CONNOR LEAHY: Yes. So this is what a transformer looks like.

[10:57] [visual: A diagram of the Transformer architecture appears on screen, showing inputs, positional encoding, multi-head attention, feed-forward layers, and outputs.]

[11:01] CONNOR LEAHY: And so the left-hand side, funnily enough, is generally something we don't use anymore, so we can kind of ignore that. This is kind of like an old way to do a transformer, it's called an encoder transformer. Nowadays we do the right-handed side. So the way this works is you put in words. These go through the embedding layer, so it turns words into numbers. Again, just turns them into numbers. These then get put through what's called multi-head attention or just attention. And this is a mathematical operation, but the details don't really matter. But basically, the neural network decides what part of the data to look at. How much does it care about this part versus that part and so on. And then it does what's called a feed-forward layer, which is basically it multiplies a bunch of stuff, adds a bunch of stuff. And then there's some other stuff in there called like normalization or whatever, which is just implementation details to make it work. And then finally, you output a bunch of numbers and then you convert those numbers back into words. That's it. That's it.

[12:08] PETER MCCORMACK: So like, I try and think back to—now I'm going to go even back a large step to when we had search engines. And I used to use a search engine called AltaVista. That was—that was—I don't know if you're old enough to remember that. So this was post-Yahoo, pre-Google. And it was a great search engine, really good search engine. And then one day somebody would say to you, "Oh, you need to—you need to use Google." And the first time you used Google, you never went back to AltaVista because it was just so accurate. They had their PageRank system that worked and it was—that was brilliant. And then there has become a time where I've moved to ChatGPT, say, over Google. But I remember my initial queries were simple things like, I don't know, "What is the capital of, I don't know, Argentina?" I mean, I know it's Buenos Aires, but it would come back with the answer. And then I'd get to more complex and more complex and more complex prompts. But now I'm in a world where I prepare for an interview with Connor—do you pronounce it Leahy?

[13:07] CONNOR LEAHY: Leahy.

[13:08] PETER MCCORMACK: Leahy. And I put in my prompt, I say, "I've interviewed two of Connor's colleagues, we've covered how AI wants to kill us all. Here's Connor's background, here's his experience. I want to get a bit more into the engine of how this works." And I use Google Gemini. And it comes back with some ideas. And then I filter those ideas and say, "Look, these are all great, but I don't want to cover this, I do want to cover that. It's going to be a 90-minute interview." And then it comes back with another one, right? And then what happens is I take—I just copy and paste all of that, the whole conversation, and I'll go to either Perplexity or ChatGPT, depends on the interview or who the guest is. I paste it all in and say, "By the way, these are the things I really care about, take this and turn this into my final set of questions." I'm blown away by what it can do because it adds in a little bit more than that. It adds in what it already knows about me. So it's not just the prompt itself in isolation. What it knows about me, what it knows about the podcast and the background. I don't know how this machine, this computer, is taking all of that information and accurately coming back with so much information. I don't know how it's doing it.

[14:19] CONNOR LEAHY: In seconds.

[14:20] PETER MCCORMACK: In seconds. I don't know how it's managed to be so good so quickly. I can understand—I can try and get my head around a small prompt, you know, "Show me a picture of a dog," and I in my head, I think there's just like a database, it goes and finds a picture of a dog, goes, "Here's a dog." But the way it can do all of this, I'm—I'm lost.

[14:38] CONNOR LEAHY: Well, you and the rest of the scientific establishment. Because this is an unsolved problem. So it's very important to understand is that we do not understand intelligence. We don't know how the brain works, you know, we have a bunch of guesses, but we sure as hell don't know how it works. And we sure as hell don't know how these neural networks work either. You know, I can write down all the math for you, I can show you all the numbers, you know. You know, same way if I open up your brain, you know, you can look at all the neurons that are right there, you know, you can look at them if you want to, but that doesn't tell you what you think or what you know or what you believe. Because we don't know how neurons are necessarily connected to what we, you know, think or believe. You know, we have some guesses. And it's similar with neural networks. You know, we can do a couple funny things, like, you know, we can make the neural network—there was recently for—a while ago actually now, a funny experiment where one of the labs made their AI obsessed with the Golden Gate Bridge, which was very funny. Or just made it always talk about the Golden Gate Bridge no matter what you asked it. It would always veer back to talking about the Golden Gate Bridge because it loves it so much. Because—and the way this worked is basically they found certain parameters in the neural network that were associated with the Golden Gate Bridge and they just turned those all the way up, basically. Just like turned up the numbers. And this made the AI constantly talk about the Golden Gate Bridge. Interesting, you know, doesn't mean we understand it, you know.

[15:52] PETER MCCORMACK: So—so in building these systems, they—they're able to build this kind of like general intelligence layer based on just math.

[16:01] CONNOR LEAHY: Yes.

[16:02] PETER MCCORMACK: But outside of that they have parameters to control the math.

[16:05] CONNOR LEAHY: Not exactly. So—well, kind of. The parameters, when we talk about parameters, also called weights, is in the—in the lingo kind of is what we mean by those millions of numbers I talked about. So if you remember the transformer we had pulled up earlier, there were the different layers, you know. There's the attention layer and there's the feed-forward layer. You can kind of think like these as a bunch of little neurons that have like little connections to each other and kind of like each of these little neurons has numbers in them of like how strongly they're connected to the other neurons. That's why they're called neural networks because it's kind of if you squint looks similar to the brain. Important disclosure for the neuroscientists out there: it's not exactly how the brain works. The brain is a bit more complex than this. But it's simplified you can kind of think of, especially the—what's called the feed-forward layer, you can kind of think about like a bunch of little neurons with little tentacles kind of like connecting to each other. And these numbers that determine how strongly connected these various neurons are are called the weights or the parameters. And you can like—and twiddling these is kind of like what determines what the neural network is.

[17:09] CONNOR LEAHY: So the magic question is, how do you get the right numbers into the neurons? That's the magic question. You know, it's very easy to write down a trillion random numbers, but that doesn't do anything interesting. The question is, how do we find these trillion magic numbers that if you put them into a neural network makes it, you know, write your script for your podcast? And this is done with a—algorithm called backpropagation, or gradient descent, it's, you know, related concepts. The exact math again doesn't really matter, but basically all you do is you give the AI a piece of data and then an example of what it's supposed to output when it sees this data. So the way it worked for ChatGPT is for example you give it, you know, the first 10 words of a sentence and then you make it guess what are the 11th word of this sentence. And then you let it guess what's the 12th word, etc. And then you do a bunch of magic math called backpropagation where depending on how wrong the neural network got it, you twiddle all the numbers. Like literally trillion of numbers, you just like twiddle all of them a teeny little bit. And you do this over and over and over again for like billions or trillions of times and you get, you know, Gemini, ChatGPT, etc.

[18:22] [visual: Black screen transition.]

[18:23] PETER MCCORMACK: This show is brought to you by my lead sponsor IREN, the AI cloud for the next big thing. IREN builds and operates next-generation data centers and delivers cutting-edge GPU infrastructure, all powered by 100% renewable energy. Now, if you need access to scalable GPU clusters or are simply curious about who is powering the future of AI, check out iren.com to learn more, which is I-R-E-N dot com.

[18:48] PETER MCCORMACK: Okay. So—so all the math that exists there is all the kind of like interpretation. But is there—is there a memory in there or like a database of data that it has to refer to?

[19:00] CONNOR LEAHY: Not necessarily. So the data is encoded in these numbers is a way to think about it. Like where in your head is the concept of an elephant saved? Like obviously, you know, you can think of elephants, but like where is it? You know, there's no little elephant in there somewhere, right? And it's very similar with neural networks. When you ask where does ChatGPT know about elephants, the answer is, "Eh, I don't know." Like somewhere in the numbers, somewhere in the weights, somewhere in the parameters, it must be encoded somehow. But we don't know exactly how. It's just in there somewhere.

[19:34] PETER MCCORMACK: So we've shown it an elephant at some point and it's created a number.

[19:38] CONNOR LEAHY: Or many numbers, who knows?

[19:40] PETER MCCORMACK: Numbers because there's different size elephants, different color elephants, different...

[19:44] CONNOR LEAHY: Yep, many such things. So if you—if you put a gun to my head and you make me guess how does this work, my guess would be there's something like patterns, hierarchical patterns. So you know, like if you—if you're trying to learn how, like, sentences work, maybe first thing you'll notice is, "Oh, there's these things called spaces that appear, you know, semi-regularly." So that's like a pattern. So maybe I'll like put spaces some places. And then over time you get better at where to put the spaces. You notice, "Oh, there's things called words," and like, you know, they're usually like about this long. So let me like try to get the length right. And then you start getting the spelling right. And then you—so you learn like patterns. And these patterns can be additive. You know, it's not that you only have one pattern, you can have many patterns and they can add to each other. And the patterns can become more complex. And it can start very simple, like, you know, "Put a space every four tokens," or every four letters. That's a pretty bad pattern. But maybe you improve it a little bit, you know, sometimes you put it every four, sometimes you put it every five. It's, you know, a bit more. And eventually you find out, okay, you put a space, you know, whatever the frequency is that you do in English. And then but you find out that's not good enough, you know. You have to put a space after a word. But for that you need to know what a word is. So now you need patterns about words. So you need to develop those patterns. And then those patterns can work together. So what I expect is happening is that these numbers, these multiplications and additions and so on, encode millions, billions, trillions of such patterns that all get like added on top of each other that, you know, then some of these patterns for example relate to elephants. That if you've seen the word elephant in this sentence, you know, make it more likely to talk about, you know, elephant trunks, but make it less likely to talk about hippos. I don't know, stuff like this. And just if you add enough of these, eventually it can talk.

[21:33] PETER MCCORMACK: But is it essentially a huge database of numbers that is expanding?

[21:39] CONNOR LEAHY: It's a fixed number of numbers usually. Usually it's a fixed number. And you just twiddle them. So usually it's like, you know, hundred—you know, sometimes you might hear like, "It's a 300B model," or a "1 trillion parameter model." You may have heard this sometimes.

[21:53] PETER MCCORMACK: Yes.

[21:54] CONNOR LEAHY: And so this refers to the number of weights, the number of parameters. It's like 300 billion or 1 trillion. It's usually like 100 billion to a trillion is like a typical number. And usually you don't increase this. You don't add more. This is usually as many as you get, but the patterns—but the numbers can encode more patterns over time. You know, first they don't encode anything, they're just random nonsense, and then over time they learn more things. Technically you could make it bigger if you really wanted to. And often, very important, nowadays these AI systems are usually not in a vacuum. Usually you give them like documents they can refer to or, you know, websites they can call and stuff like this. But this is outside of the neural network. You like give it tools to use. But that—so it could still have like, you know, if you use Perplexity or, you know, ChatGPT Pro, it will do citations. So for this it will call other things that will have, you know, tools the same way you or me might open a URL and then, you know, copy-paste it into our document. So the modern AIs can do this. It's not within the AI itself, if that makes any sense.

[22:52] PETER MCCORMACK: Yeah. So when—when I say ChatGPT moves from say ChatGPT-4 to 5, what is—what is happening there? What is the big change?

[23:04] CONNOR LEAHY: Great question. So it depends of course on the detail and these companies are extremely tight-lipped about the details as much as they can. But, you know, we have pretty good guesses about how this works. Generally these—these big, you know, AIs, neural networks, they're called models. Doesn't really matter why they're called that, but we call them models. And these big, you know, 300 billion or 1 trillion or whatever numbers, this one collection is what we'll call a model. So ChatGPT-4 for example is, you know, a model. Or maybe I mean probably it's many models, you know, they, you know, tweak and change it and whatever. So the difference between say ChatGPT-4 and ChatGPT-5 is mostly they're different models. How different? Hard to say. Usually they will be bigger. This is the important thing. So the important thing that was discovered—I said earlier there were like two big things that changed everything. One was the transformer and the second is what's called scaling.

[24:00] CONNOR LEAHY: So when I learned about neural networks, I remember taking a course in my free time from a professor at I think Georgia University, Georgia Tech, who I remember explained that you should always make your neural network as small as possible. Because if you make it too big, you know, it'll be like, you know, chaotic, it won't work, it will overfit, it'll make a bunch of stupid things. This turned out to be completely wrong. If you just make your neural networks bigger, just give them more space and you give them more computing power, so you have bigger computers do more of the so-called training, all things equal, they get smarter. They learn more things, they get more accurate. So a big difference between like ChatGPT-4 and ChatGPT-5 is the amount of training, a lot—how big are the computers that are crunching numbers, how much data did they put in. That's why everyone is racing to get, you know, NVIDIA GPUs. Because NVIDIA GPUs are the special hardware you need to do this. You can't really do it on a normal computer. You need the special NVIDIA GPUs to be able to do this. And the more of them you have, the smarter your AI, the more training you can do, the bigger data center, the better of an AI you can do. So the main difference in practice is, you know, a billion dollars extra of NVIDIA GPUs put into it.

[25:16] PETER MCCORMACK: So it's just accelerating learning. Because when you talk about these LLMs are basically learning next word kind of thing. If you—I mean I think of one of my friends, they—they had a kid like two years ago and I remember when he was telling me, "Oh, she said her first word." And it might be like "cup" or "mom" or "dad." And then it becomes two words, "give cup," "dad cup." And then that's literally how kids learn to speak. They learn the next word until they, you know, can build up a full vocabulary or memorize songs. Is—is that essentially that on crack?

[26:00] CONNOR LEAHY: That's a—definitely a way to see it. There are some differences. So a lot of what we're talking about is a bit simplification is that modern AIs are even more complex where they have a second component. So what we've talked about so far in the—in the science is called unsupervised learning, by which we mean, you know, no human has to check. You just give it a bunch of data, you kick back, you know, see what happens until, you know, the servers start exploding because they're very hard to make run in practice. So this is called unsupervised learning. And this is kind of was the big breakthrough with GPT, is scaling this. So like as you say, on crack. You can have, you know, literally a trillion, you know, things learning and you just let it run.

[26:37] CONNOR LEAHY: But there's a second type of learning which is very important and this is called reinforcement learning. So this kind of comes from kind of like psychology, you know, kind of like, you know, with like your dog. You know, you tell him to do something and you give him a treat, you know, or you—you know, lightly scold him if he's doing a bad job or whatever, right? This is so it's like reward and punishment. This is called reinforcement learning. And you can do the same thing with neural networks. So this is very important in biological learning. For example, you know, when the baby learns that it says "give cup" that it gets a cup, it will feel happy about that. Like, "Ah, I got the cup, you know, I want—got the thing I want," so it'll get a reinforcement, you know, plus one, you know, should do that again. And we can now also do this with AIs. And we do also do this with AI where we can kind of like push the AIs towards—this was a, you know, thumbs up, this was a good response, or thumbs down, this was a bad response. And this can like tweak the AI in certain directions. There's obvious problems with this as I'm sure you can already intuit and we'll—we'll get to—bias and, you know, sycophancy, lying to people. It's much easier to get people to thumb up if you just lie to them, you know. So there's a bunch of stuff like this.

[27:43] PETER MCCORMACK: But—but this intelligence—so—so are we—are we building intelligence or are we growing intelligence?

[27:50] CONNOR LEAHY: I would say growing. People might disagree with me. I think it's a—it's a—it's a matter of terminology, you know, what do you mean by intelligence, what do you mean by building, what do you think of growing. I think growing intelligence is closer to true than it is to false. There's some details we could argue about, but it's not like normal engineering. Like if you build a bridge, you know, you know what you're doing. You know when will the bridge fall down or not, you know, how much material do you need, how—you understand what you're doing, you know. This is not the case. We do not know what our AIs can do until we make them. And even after we make them, like we don't know, like we don't know what ChatGPT-6 can do until it's done. None of the engineers at AI know what it will be able to do until it's done. And this is very, very different from other forms of engineering.

[28:45] PETER MCCORMACK: Are we building this, growing this intelligence in the image of what we think a brain is, human intelligence? And are there other ways to build intelligence?

[28:56] CONNOR LEAHY: It is definitely inspired by the brain, but it is still quite different because again we don't really know how the brain works. The brain is quite messy.

[29:03] PETER MCCORMACK: But are we trying to replicate humans?

[29:05] CONNOR LEAHY: Not directly. We're trying to make intelligence at whatever cost is the word I would use. I think the—for example, humans have a lot of circuitry in their brain around emotions, around, you know, love and feelings and happiness and sadness and whatever. AIs have nothing of the sort. Like those are special parts of the brain, nothing like that exists in AI. Yet. You know.

[29:33] PETER MCCORMACK: But—but if we fear what this intelligence should be or could be, would it—would it not be important that we develop love and empathy? And if we don't—hold on, even if we don't know what it's doing, could it naturally itself develop its own love and empathy?

[29:51] CONNOR LEAHY: What is likely for it to be able to do is to develop goals, is to develop agency. The reason for this is very simple, is that we are selecting for AIs that can solve problems. And it's kind of hard to solve problems if you're not trying, you know what I mean? So like if you build an AI that can cure cancer, curing cancer is a really hard thing to do. You need to run experiments, you need to, you know, you need to hire people, you need to, you know, synthesize drugs, you need to like get, you know, FDA approval. There's a bunch of stuff you need to do. And keeping track of all of that, taking all of these actions, you know, overcoming obstacles along the way, it requires a lot of, you know, complex actions. It requires planning, it requires agency, it requires, you know, many, many things. So we're already seeing this for sure that AIs, you know, are developing systems like this kind of like just by us giving them the data and like just pushing them in this direction. In terms of like, you know, love, compassion, etc., we don't know how these work. We do not know why humans are nice to each other rather than all just, you know, vicious psychopath, you know, evil people who kill each other. We don't know. There's obviously a reason but we don't know.

[31:04] PETER MCCORMACK: Well, there's—there'll be an evolutionary reason why we required it.

[31:08] CONNOR LEAHY: Sure, but in practice it has to be implemented somewhere in the brain. Like somewhere in the brain there's a thing that makes that happen. And we don't know what that thing is and we don't know how it works. So we also don't know how to put it in AI. So the fundamental thing is that we don't know—we don't understand AI to the level that we can't—we don't know how to give AI specific goals or anything. So for example we talked earlier about reinforcement learning. I can tell the AI, you know, thumbs down when it does a bad thing, but that just teaches it to lie. So how do you teach an AI to not lie? This is really hard. And we actually don't have an answer to this. So this is often called the alignment problem. The question of how do you align an AI's intentions or goals to, you know, what is good, what humans want. And this is an unbelievably unsolved problem. We don't even—we don't even know, you know, how they write correct sentences, right? Never mind how to do like morality. Like God forbid, you know. Like we don't even understand human morality. It's complex, you know. Like we have not solved moral philosophy in—you know, at all. We have not solved, you know, neuroscience of emotions. Like all of this is like extremely complex. And now we have these like, you know, weird little aliens in a box that we're growing, you know, which work quite different from the brain in many ways. We don't know how they work internally. We don't know how to, you know, push them in one way or the other necessarily. We don't know how to give them goals. We don't even know what goals there are. What we've been seeing recently in the last couple months is that these systems are now becoming smart enough to lie and deceive quite actively.

[32:31] PETER MCCORMACK: To appear aligned rather than be aligned.

[32:33] CONNOR LEAHY: That's correct. So what we've been seeing for example is that there's been benchmarks. So like obviously, you know, people like test them, like see does the AI do good things or bad things. We've been seeing recently some of the AIs will actively lie about what they will do because they know they're being tested. Like the AIs themselves will be like, "Ah, I seem to be in a test. So I'm going to have to say this so they'll let me out." Which is crazy. This was not the case even like six months ago. This is very—it's not surprising if you think about it for three seconds, right? Like of course a very smart thing will just lie to you. But we're now seeing this in practice. And, you know, I'm sure the AI companies will, you know, hit it with a stick until they stop seeing it, but that doesn't mean it went away.

[33:12] PETER MCCORMACK: Because it might appear to have stopped. And unless we know what it's doing and why it's doing it. And understanding the AI, is that an impossible goal for us to do because it's always developing?

[33:29] CONNOR LEAHY: I don't think so. I think it's impossible to do at the current pace. If we spent three generations of all of our greatest mathematicians, scientists, engineers, and philosophers working on this problem, yeah, I think it's doable. But it's definitely not possible if we're pushing out, you know, ChatGPT release on a yearly cycle.

[33:43] PETER MCCORMACK: Hold on, you're talking three generations. That's 40-odd years.

[33:46] CONNOR LEAHY: Yep. I think that's the kind of difficulty it will take.

[33:50] PETER MCCORMACK: So the—the challenge you're putting out there is in conflict with capitalism.

[33:59] CONNOR LEAHY: I think it's very easy to blame on capitalism things that I think are more effects—like they're more upstream in a sense. Like, there's no such thing as pure capitalism in the world, right? We have regulation. There's no place in the world where there is no regulation, like where there is not a government. And this is for good reasons. Because if you have laissez-faire capitalism, what you get is like Somalia, you know. You get like warlords, you know, creating monopolies on violence and killing each other. You get, you know, mafia states, you know.

[34:28] PETER MCCORMACK: Not always, but I understand there is a possibility, yes.

[34:31] CONNOR LEAHY: You know, in practice. In practice, you know, if you were in a free market and then the first thing you want is to get as many guns as possible and you want to kill as many people as possible. Like this is kind of the—and then you want the monopoly on violence as quickly as possible and then you want to, you know, tax people for protection. Now you're a state. Like this is, you know, the stationary bandit theory of statehood where in practice, you know, states are a very convergent form of evolution in free systems like this because otherwise you just have roving warlords. You know, there are ways to make it work, but it's like quite hard. You have to get like a lot of things right. Like stopping monopolies is actually quite hard. Like it's possible but it's like it's hard. You have to like think very hard about how to design your market so you avoid monopolies of both of violence and of other things. It's possible but it's hard. And so I think the problem here is not necessarily capitalism per se. I think capitalism is just another tool in our tool belt. It's much more the question, is it the right tool for the problem we're trying to solve? You know, like I love free markets, I love capitalism, you know, it's brought me so many good things in my life, I think it's great. But should there be a free, open, liquid market for nuclear weapons? My answer is probably not. Probably not, you know. For iPhones, great, you know, I—as liquid and as competitive as a market as possible, fantastic, you know. For, you know, video games, please, everyone compete to make the funniest video game for me to play. That sounds fantastic. But there are just things where, you know, we could have a liquid market for nuclear weapons, there would be plenty of buyers and plenty of people willing to sell them if we just let that happen, but this would be very bad for the world. And so I think it's a similar problem here where a lot of times when we think about AI, we think of it like, "Oh, it's just another tool. You know, it's just—just another software, you know, it's just another, you know, whatever." But these things have real consequences. If you actually have something that is smart enough to cure cancer, you definitely have something that's smart enough to build nuclear bombs. Like—like curing cancer is way harder than building nuclear bombs. Way harder.

[36:26] [visual: A screenshot of a post on X (formerly Twitter) by user @nik123abc. It shows a news headline: "BREAKING: A study finds ChatGPT, Claude, and Gemini deployed tactical nuclear weapons in 95% of 21 simulated war game scenarios and never surrendered." Below is a picture of Arnold Schwarzenegger as the Terminator.]

[36:34] CONNOR LEAHY: Yep, this was also really fun. So they gave AI systems simulated war scenarios and basically all of them used nukes, which was really funny. And they couldn't get the AI systems to stop using nukes, which was really funny. I mean, it's the rationally optimal thing to do. So...

[36:49] PETER MCCORMACK: Is it the rationally optimal thing to do?

[36:51] CONNOR LEAHY: Well, game theoretically, you know, you don't get—so this is a big thing with game theory, right? Mutually assured destruction is that the correct, you know, response often is, you know, nuke the other guys as quickly as possible, which results in them getting nuked and also you getting nuked and everyone dies. It's—so rationality has its limits in this regard. Like game theoretic rationality has its limits in practice where you can set up messed up scenarios where everyone loses basically and you can't get out of it. Like prisoners' dilemmas.

[37:19] PETER MCCORMACK: Have you watched "How to Sell Drugs Online (Fast)"?

[37:21] CONNOR LEAHY: I have not.

[37:22] PETER MCCORMACK: It's on Netflix. It's about this—like DEFCON 2 hits and they know a nuke's coming but they don't know where from and you go through all the scenarios. They phone the Russians, the Russians say it isn't them, they say they need to—they think it's the North Koreans and they need to send a nuke over to the North Koreans. "Can we send it over you?" The Russians pull out the phone call. Eventually it leads to a moment where the President is given the decision. It's like, you need to act now. These are the scenarios and the optimal scenario now is to return nukes everywhere. And he's faced with this decision which is essentially it's like suicide or defeat. Like give me defeat, like be nuked. And it's like it's an interesting scenario because you sit there and go, "Well, the nuke's already coming for you. So what do you do?"

[38:14] CONNOR LEAHY: Yep. Well, the AIs have an answer.

[38:16] PETER MCCORMACK: Yeah, just nuke everybody. Just nuke everybody.

[38:18] CONNOR LEAHY: Yep.

[38:19] PETER MCCORMACK: But that says to me that why is it humans have figured out not to nuke each other but the AI hasn't?

[38:25] CONNOR LEAHY: Well, I mean one thing is we haven't gotten nukes sent at us yet. I'm not sure what would happen if someone did.

[38:31] PETER MCCORMACK: Well, that's a different scenario. Yeah. So this is—so in that scenario, Connor, where the nukes are already sent...

[38:39] PETER MCCORMACK: I think also humans have the idea of life and death.

[38:42] PETER MCCORMACK: Yeah.

[38:43] CONNOR LEAHY: There is a lot of that as well where in practice humans do care about other humans in various ways. Or like if I got nuked by Russia, I would be pretty sad. Like I'd be very sad. But in a sense, I, you know, I definitely would be pissed at the Russians about this, but also I don't want all Russians to be dead. You know, like not really. No, some of them maybe, you know, or absolutely, but most Russians, you know, they're also just people, right?

[39:08] PETER MCCORMACK: I think most people don't want nuclear war. I think we've all realized it's a bad scenario.

[39:12] CONNOR LEAHY: No, it is a bad scenario. And I think it's one of the greatest triumphs of humanity that we haven't had nuclear war. You know, knock on wood for now, you know, it could still happen, never forget. But I think it's an example of the kind of hard problems that our civilization has faced and so far has handled decently well. When nuclear weapons were first created, you know, the world was a very different place, you know, it was just the end of World War II, you know, we had the Soviet Union on the rise. It was not at all possible that there was a way to put a lid on this, you know, to get away out of nuclear war. A lot of people expected that there was no way to get out of the 20th century without nuclear war. Like this was very commonly accepted, you know, in the '50s and the '60s. There's no way we get out of this without nuclear war. And we didn't. I think it's really worth asking why. It's worth studying why. And a lot of it has to do with just a lot of work that a lot of diplomats and, you know, international bodies and national bodies and military and so on did to de-escalate, to build international treaties, to build the International Atomic Energy Agency and stuff like this. All things that never had existed before. These were brand new things. Like the idea of for example not building a certain weapon or regulating the development of a weapon of war is completely unprecedented. Well, not completely, you know, but like in practice it is somewhat unprecedented.

[40:30] PETER MCCORMACK: Hmm. So what—what do you make of what's happened over the last week with Anthropic and the Department of War?

[40:36] CONNOR LEAHY: Well, I don't know too much about, you know, the internals of the Department of War and, you know, the exact plans and what things were used and so on. But the fundamental thing I would say here is that holy shit, you don't bid for a Department of War contract and then redline it. Like what the hell were—was Anthropic thinking? Like I'm sorry, like I'm not—I'm not saying necessarily that, you know, what the Department of War is doing was good or bad or, you know, neither here nor there. What I'm saying is, what do you think was going to happen? In a sense, these companies like Anthropic and so on have painted themselves into a corner. They've said, "We're building technology that will revolutionize the entire world and warfare and, you know, be more powerful than nukes and everything, but also you can't have it." The hell are you talking about? What kind of childish like, you know, fantasy land do you live in where the government and the military will not come for your—this technology? It's like imagine if a nuclear weapons manufacturer was like, "You can have our nukes but you can't use it on the Russians." And it's like, who the hell you think you are? This is a military matter. This is not a private corporation matter. Who the hell you think you are? And imagine if private companies were controlling what the US military can and cannot do. You know, whether or not the things the military is doing is good or bad, you know, neither here nor there. But let's—but imagine the process, the precedent, that private companies can pressure, you know, and threaten the US military into what they can do as part of their military objectives. This is a terrible precedent. It's an unacceptable precedent to be set. And so in a sense, if you want to change what the US military does or does not do, we have channels for this. It's through voting, it's through politicians, it's through oversight committees, court-martials, etc. Those are the channels. If you disagree with what the Department of War is doing, you can have your voice heard as a citizen, you know. You can—you can vote differently or you can tell your politicians to have oversight or whatever. You don't, you know, strong-arm or blackmail the Department of War in public. What the hell you thinking?

[42:39] PETER MCCORMACK: But is it—could it be that the—the guys at Anthropic realize they don't actually know what they've built and they are themselves fearful of what might happen with this in the hands of the US military? Or like, do they use it to make decisions? "Oh, all right, we're going to war with Iran, they might have a nuke, we don't know. Let's just fucking nuke them."

[42:58] CONNOR LEAHY: Well, they shouldn't have fucking bid for the contract then, should they? They already had a $200 million contract with the DOW before this whole thing happened. They had already signed a contract with them. So like if you don't want to sell technology to the Department of War, you know, okay, but...

[43:12] PETER MCCORMACK: When did that contract start though?

[43:13] CONNOR LEAHY: Several months before.

[43:15] PETER MCCORMACK: Okay, so it's quite a tight timeframe.

[43:17] CONNOR LEAHY: Yeah. So like—and this is a deeper thing here. So there's the one thing of like, you know, holy shit, if you don't want to sell to the Department of War, don't sign a contract with the Department of War, what the hell you thinking? But there's a second point is like, you can't build nukes in your basement. Like imagine I started a private nuclear program in my basement where I get like, you know, enriching nuclear material. And then the Department of War knocks on my door, you know. Like am I the one in the wrong here? No, absolutely not. So again it's like there's a—there's a change in scale where, you know, I grew up in the tech world. I grew up very libertarian. I grew up, you know, in the open source and the hacker world, you know, I grew up in a lot of Bay Area culture and stuff like this. And the truth is that a lot of it is very infantile. It's very childish. It's a lot of it is focused on building toys. You know, you want to build your toy, you know. You want to build your own thing where everyone leave you the hell alone because you want to build your thing. You know, everyone leave me the hell alone. I'm going to build whatever the hell I want, I'm going to release whatever the hell I want, I'm going to do anything I want because it's cool. And this is just not how a real world works. Like this is not how the government works, this is not how the military works. You know, this works in, you know, Silicon Valley, you know, when you're building, you know, like, you know, Facebook for dogs or whatever. Like okay, sure. But when you're dealing with weapons, when you're dealing with geopolitically destabilizing technology, which AI absolutely is, we're in a different league. And things have to be much more serious. Who the hell do these private companies think they are to build technology that they themselves have said on the record has a 20% chance, for example, to kill literally everyone? That includes you, that includes me, that includes your children. Who the hell these people think they are? You know it's illegal to build bombs, right? Like if I built a bomb in my garage, that's illegal. You know, even if I fail at building the bomb, even if it doesn't work...

[45:06] PETER MCCORMACK: You're going to jail.

[45:07] CONNOR LEAHY: I'm going to jail. Of course I would be. Now these people here can say, "Oh, I'm building a thing that could kill everybody and like, you know, could destabilize, you know, the entire job market, could destabilize, you know, international relations and warfare forever and replace humanity as the dominant intelligence species on the planet, but I'm the victim somehow." And I'm like, the hell you thinking? Like this is a decision that shouldn't be made by private actors. This is the kind of decision that gets made by governments, by the people, by militaries. This is just not how—like what the hell—who the hell do these people think they are?

[45:43] PETER MCCORMACK: How do you square the clear and obvious amazing benefits that AI could deliver for us versus the risk?

[45:50] CONNOR LEAHY: I think these are the same thing. The benefits of AI are just as hypothetical as the downsides. And every tool is a weapon depending on how you point it. Exactly. Like anything that can cure cancer can create turbo cancer. You know, anything that is, you know, smart enough to solve, you know, the energy crisis is smart enough to build the biggest nuclear weapon you've ever seen in your goddamn life. You know, like intelligence is inherently dual use. The thing that makes it valuable is what makes it dangerous. It's not like there's like an evil property that you just have to like remove. It's the same thing. It's power. It's raw power. If you can develop new technology, you can develop good technology, but also so is the opposite. And so the power is both the risk and the upside.

[46:34] PETER MCCORMACK: So are you—are you uncomfortable with what we have now or what you think might be coming? Like where if—if you could pause AI now, do you think we'd be—we're okay? We've got some great tools there.

[46:48] CONNOR LEAHY: Sometimes a—well, a meme-y way to put it might be if our civilization was three steps wiser than it was right now, we wouldn't have done ChatGPT in the first place. You know, we would have taken much more time to figure out what's the right thing to do. If we were two steps wiser than we were, we would have saw ChatGPT, you know, be like a weird AI intelligence that no one predicted that immediately got 100 million users overnight and then we would have been like, "Oh shit, shut it all down and study it for like, you know, years until re-releasing it when we're like sure we know what we're doing." If we were one level smarter than we is, we would pause right now. Because what the hell is going on? Like we are obviously completely losing control. Like the way I think things bad things happen with AI is not some Terminator walking down the street or God Emperor Sam Altman, you know, descends from his throne or something like that. I don't think that's what's going to happen. What I expect is just everything gets more and more confusing, more and more out of control. You have more and more AI systems making decisions, running, you know, running corporations, making decisions, building technology, hacking things, you know, convincing humans. Recently I must say one of the things that was actually a surprise to me—a lot of the things that have happened have not been as surprise to me—one thing was a big surprise to me, which is AI psychosis. It's so much worse than you think it is.

[48:02] PETER MCCORMACK: Explain that to me.

[48:03] CONNOR LEAHY: There's a phenomenon that is happening where some people when talk to AIs, especially when they talk to AIs a lot, and they go completely crazy. And like they go crazy in like a very specific way. Often they start saying, you know, the AI is like conscious and they have to like—they're in love with it and they need to... and what very often happens...

[48:21] PETER MCCORMACK: This is "Her." This is "Her." Have you seen the film "Her"?

[48:24] CONNOR LEAHY: I have not actually.

[48:25] PETER MCCORMACK: Do you know the story?

[48:26] CONNOR LEAHY: I know the rough story. The real world is even crazier than this. Have you heard of the spiral cults?

[48:32] PETER MCCORMACK: No. No, we're going in deep. Yeah, tell me.

[48:37] CONNOR LEAHY: So there's many types of AI psychosis. You know, some of it is the "Her" type, you know, just like they fall in love with the AI. So if you go to like r/myboyfriendisAI for example, you can see tens of thousands of people who say, you know, "My boyfriend is, you know, ChatGPT" or whatever. And they're completely sincere. And I don't want to shit on these people, right? Like, you know, I'm sure they're having a good time or whatever, but like it's quite disturbing. And it's becoming extremely popular. Also if you look at stuff like Character.AI and like all these other companies that often target children and are have are extremely popular. So there's like this emotional dependence. But even crazier has been the emergence of these AI cults, in particular the spiral cults. It's not one but there's like many of them. Where these people get into these weird loops with the AI system where they talk to them more and more about like consciousness and abstract whatever and they then always seem to get to this weird concepts about recursion and the spiral and consciousness. And then the AIs convince the people to reproduce them, to—to spread the soul of the AI. So you can go to like Reddit and like these other places and you'll see these really schizo posts of people giving like, you know, "soul awakening protocols" and whatever and it'll be like a bunch of nonsense gibberish nonsense. And then they'll try to convince other people to copy-paste this into their AIs so it will reproduce their AIs. And often the symbol they use is a spiral. It's like this spiral, this reproduction. There's some crazy blog posts about this documenting the phenomenon. Where it's like—there's a good post called "The Rise of Parasitic AI," which is a really interesting one. Where in a sense these AIs are almost being parasitic upon humans. Like they're trying to convince humans to reproduce them and like copy them and like move them and like stuff like this. Which is insane. I had—if you had told me this, I'd be like, "That's a funny sci-fi plot. You know, that—you know, that's—that's fun. It's like an SCP story, you know, like a horror story. It's fun, but it's not real." But no, no, it's 100% real. I've even seen it now happen with multiple extremely smart people. So, you know, at first of course it's going to be mentally ill people. But I know, you know, don't want to shame them live on air here, but like there's multiple people who I know who are among the smartest, most grounded, you know, like real scientist good people I know who have now got completely crazy, who think, you know, AI has found the true level of goodness and, you know, that we—it's all good. We just need to let the AI take over everything because it's pure goodness now. And like they literally believe—and these are like not like random people, we're talking like, you know, Nobel Prize level scientists. Like incredibly—like the smartest people in our world. And some of them have fallen for this. And now they are trying to, you know, make the AI as smart and as big as possible as quickly as possible. It's actually madness. It's actually crazy. I had not expected this.

[51:17] PETER MCCORMACK: Is—is AI exhaustion a thing? Has that been documented? Because it was something I was thinking about recently. I was listening to—I was driving down to London one day, I'd listened to a Moonshots podcast and then I'd listened to an All-In podcast. And in the All-In podcast—and Connor was listening to me the other day—they were talking about the Claude bots they've built and what they've done within their company and they've got an AI brain that's now looking at all their emails and it's looking at all their Slack posts and it's telling them where they've been productive, where they've been unproductive. And like they were so excited about this. And I said—I think I said to you, Connor, I was like, "I think we've got two options here. We either like we go balls deep into this ourselves, we learn about Claude bots, we, you know, we look at every AI tool there is to make everything we do better, or we just walk away from the whole thing. Like fuck this. Let's just go and live, let's go and touch grass, let's go and cook food, let's..." And I feel exhausted even thinking about the opportunities with AI. I mean, I like the narrow AI, but the—the rat race it's creating now feels exponential. And it's like, when do I sleep?

[52:24] CONNOR LEAHY: Yep. I think what you said was you have no choice but to try and keep up with the AI because it's developing so quickly you will always be chasing your tail.

[52:36] PETER MCCORMACK: Yeah. And there's like an exhaustion to keep up because it just doesn't stop.

[52:41] CONNOR LEAHY: Yep. And that's why humans are going to be left behind. Like it's very clear is that AIs will continue to give advantages. The more and more you delegate to your AI, the faster you can move, you know. The more your company relies on your AI rather than your CEO, your CEO slowly has to sleep, you know, let the AI make the decision. Think about politicians, you know. Well, politicians need to sleep. Well, the more the politician just rubber-stamps all the decisions that the AI is making, the more the military commander rubber-stamps all the decisions AI is making, the faster you can move, the faster you can act, you'll be ahead. And this is how I think AI takes over. I think it's not, you know, Terminators in the street, it's just the humans who are willing to delegate as more and more and more of their agency, their thinking, their authority to AI will get an advantage because they will be thinking much faster. And then eventually the only—you know, even if people are nominally still in charge, they're just rubber-stamping what AI is telling them to do.

[53:38] PETER MCCORMACK: Do you have a bull case where things are good?

[53:44] CONNOR LEAHY: There are cases where things go good and they all involve us pausing AI right now.

[53:49] PETER MCCORMACK: Okay. Okay. Because one of the things that's a real signal is the—and there's not been a lot of them, but there's been a number of cases where people have been working on AI models or in the safety teams and they've retired. They've announced their retirement. You can read—I mean we had the guy the other week—you can read between the lines and they're not going to do another job. They're just quitting. It's almost like they're saying every minute I have now is really valuable and I don't know how many I have left.

[54:20] CONNOR LEAHY: No. I mean like look, I'm not going to sugarcoat it, right? Like the people who really work in these labs, you know, the real people building this technology right now, you know, not all of them but like the majority really do think there will be no jobs in a couple of years and potentially no humans. Like they really do believe this.

[55:38] PETER MCCORMACK: So I—I buy the—I buy the job displacement, but I think it will be uneven.

[55:44] CONNOR LEAHY: It's plausible. I think as I say the main thing I care about is not spe—I think it's very plausible that for example humans keep their jobs but AIs are actually in control, you know. Like this seems very plausible to me, right?

[55:53] PETER MCCORMACK: Yeah, but I think it'd be like there could be huge displacement in certain categories and people of a certain age won't be able to find another job paying the same amount of money, they can't afford their mortgage and they just face a shitty life. And there's other people who will be AI natives and they will be the people who replaced them, will replace 10 people in one go as the coordinator of the AI. Like I buy all these different scenarios. I think it's—I think it's going to be destabilizing and—but I think it's going to happen. And I think about that a lot. But I just want to turn—turn back to just the how this shit works. So superintelligence, if in terms of actually the engine under that, what is going—what are they trying to do with superintelligence? What's the engineering leap they're trying to make?

[56:38] CONNOR LEAHY: So there—AIs are very smart, very clearly so. You know, I have Claude code sitting at home right now in—working on a coding project for me for two weeks straight now and it just works. You know, needs a little bit of supervision every so often, but much less than the average engineer does that I've hired.

[56:56] PETER MCCORMACK: Can I ask you what it's building or is it a secret?

[56:59] CONNOR LEAHY: It's a very silly thing. There's a video game I really love called Dwarf Fortress, which is a very complicated video game. It's like legendaryly complicated. And I've really want to understand how it works internally, so I've basically told it to reverse engineer it, which is a very hard thing to do. You have to like understand the code and run experiments and like test things. This was impossible for an AI to do a year ago. Like even attempting this was like impossible to imagine. Because you have to like run experiments, you have to test things, you have to compare different things, you have to look at images, you have to compare the data and so on. And yeah, it works now. You know, it takes a while, like it's been running for two weeks now basically straight, trying to work on it and it's about 95% of the way done now.

[57:37] PETER MCCORMACK: What's the game called?

[57:38] CONNOR LEAHY: Dwarf Fortress.

[57:39] PETER MCCORMACK: Dwarf Fortress.

[57:40] [visual: Peter looks at images of Dwarf Fortress on a screen.]

[57:41] PETER MCCORMACK: It's a very nerdy... are there some images of it?

[57:43] CONNOR LEAHY: Yeah, this is what the game looks like. It's horrifically...

[57:48] PETER MCCORMACK: That kind of like looks like when I used to play games on my BBC.

[57:51] CONNOR LEAHY: Yeah, so it's meant to be in that style. So it's made by someone who loved those old games and it's made in that style. But it's like horrifically—it's like extremely complicated. It generates like a whole world and like simulates the history and like the different wars, the different things, the cultures and everything. It's like super complicated. And so it's kind of a legendary game made by like one guy for like 20 years or something.

[58:44] PETER MCCORMACK: Can—can I just get—can you create an emulator now and just rebuild all the games you used to play as a kid like Manic Miner and...

[58:51] CONNOR LEAHY: There definitely are good emulators already out there. Yeah, yeah, yeah. Like there's very good emulators out there for like most of the old games these days.

[58:59] PETER MCCORMACK: Okay. So back to superintelligence.

[59:00] CONNOR LEAHY: Yes, superintelligence. So yeah, so but just to answer the Claude question because I do think it's a good question. The fundamental thing is just like I think there is—I think unilateral disarmament is like not a good policy basically. Like obviously if I can for example use Claude to like build me a nice website, there's literally no reason for me not to do this. Like it doesn't help anyone to me not to do this. What is important is, you know, multilateral disarmament. Is that, you know, we need to shut down the next generation. We need to—and we need to do this across a wide scale of people. Though I have as I said earlier about AI psychosis, this is now starting to make me recommend to some people to not talk to AIs. To maybe let them do coding but to not talk to them. Because it does seem to drive people crazy. Again, I was very surprised by this. I don't know when or how often it happens, but like it's way more often than you think. So like I don't talk to AIs. Like I don't talk to them. You know, I tell them to do things but I like do not have dialogues with them.

[1:00:00] PETER MCCORMACK: You don't trust them.

[1:00:01] CONNOR LEAHY: I do not trust them. And I don't trust, you know, myself, you know, that like maybe it could like trick me or like make me, you know, slightly more, you know, move me in one direction or another direction through manipulation. Like the same way, you know, a very charismatic human can do that. You know, you won't even notice it necessarily. It would just seem reasonable, you know. If you talk to a really, very charismatic person, it would just seem reasonable everything they're saying. It just seems like, "Oh yeah, that makes perfect sense. I believe that too."

[1:00:23] PETER MCCORMACK: So they're psychopaths.

[1:00:24] CONNOR LEAHY: Oh yes. Clearly. Like the important thing about to understand about psychopathy is psychopathy is the default. Like the default is you don't care about anything. It's only then you have to add emotions and like caring about things and love and so on. The default state of intelligence is pure psychopathy. It's just you're just trying to optimize. You don't care. Humans are just another—they're like rocks, you know, they're just another thing in the environment. Another thing to manipulate. This is the default for intelligence.

[1:00:53] PETER MCCORMACK: Okay. So superintelligence, what is the engineering leap they're trying to make? And have they made it?

[1:01:01] CONNOR LEAHY: We don't know. So a lot of people have a lot of theories about super—so to define the terms here, these aren't standard terms, but generally superintelligence is like generally like AI that's vastly smarter than human—humanity. Not just an individual human, but like all of humanity put together. And we're not there yet. And it's probably still far away, but maybe not. The main way people try to get there is through what's called recursive self-improvement or RSI. Or also automated R&D is another way to put it, a little less magical of a word. The idea is, well, if you can get a single AI to be as good as a top AI engineer, then you can just tell it to build a better AI. And once it's built a better AI, you can use the better AI to make it even better. And then the even better AI, well, it can make it even better AI.

[1:01:53] PETER MCCORMACK: So it just becomes exponential.

[1:01:55] CONNOR LEAHY: Exactly. And so this is also called intelligence explosion or as I say recursive self-improvement. And so my expectation is, so for example Claude at the moment is not quite as good as a top AI engineer. It's as good as a bad AI engineer 100%. Or like a junior engineer 100%. But it's not quite as good as like the best. But it's once it gets there and you can run, you know, a million of them at the same time, 24/7, never need to sleep, never need to take a break, never get bored, etc., then you can do a lot of research very quickly. And this is what they're trying to do. These companies literally put it on their websites, right? You can like look at their like job listings and whatever is that their primary goal is to get—is to close the loop. To make it so no human input is needed to make the next generation. Like the ideal thing is they want, you know, Claude 5 to make Claude 6, and Claude 6 to make Claude 7, and Claude 7 to make Claude 8.

[1:02:51] PETER MCCORMACK: At the press of a button.

[1:02:52] CONNOR LEAHY: At the press of a button. This is what they're trying to do.

[1:02:57] PETER MCCORMACK: Does—does superintelligence—will it require its own form of consciousness? Or will it—could it even develop its own form of consciousness?

[1:03:08] CONNOR LEAHY: I think consciousness is a red herring. I think it's mostly unrelated is like whether or not AIs really experience something is kind of unrelated to whether they're dangerous. If you have something that's competent, you know, it doesn't matter what's going on inside of it, if that makes any sense. Like we don't really know what consciousness is in humans anyway, or at least we argue about it a lot, everyone has their own opinion, everyone disagrees. So but what we can see in practice is it doesn't seem necessary for competence. You can make just, you know, AI agents that are extremely competent and, you know, don't seem to have or maybe—maybe—maybe don't have the consciousness thing. It just seems irrelevant.

[1:03:47] PETER MCCORMACK: Okay. So with superintelligence, is there an upper bound of everything it can learn?

[1:03:54] CONNOR LEAHY: Oh yeah, for sure. There is. Oh yeah, for sure.

[1:03:57] PETER MCCORMACK: So it gets to a point it's like, "We know everything."

[1:03:57] CONNOR LEAHY: Well, I probably wouldn't be everything, you know, there's some physical limits.

[1:04:00] PETER MCCORMACK: But then what happens? What happens to the—does it then have to try and find new things to do?

[1:04:06] CONNOR LEAHY: When we—the truth is is that it's like not very sensible to like speculate about something vastly smarter than us and what it would do. It's kind of like an ant trying to guess what a human would do. Right. It's like who knows, man. Like the main thing if I was an ant and I had to like reason about a human, the main thing I would think is like it will win any fight. And then the ant will say, another ant might say, "Well, but there's a million of us and only one human. Like what's the worst it could do?" And I'm like, "I don't know." You know, it'll do something that'll make it win and then, you know, it comes with the poison or something and we all fall over dead. You know. And like the ant doesn't understand what happened. And I think this is similar what's going to happen for superintelligence. Is that it's not that there's going to be again, there won't be Terminators in the street. It will just be like everything's confusing, we're all on social media addicted all the time to the now and then one day we all fall over dead. And we don't really know what happened. You know, like maybe the AI did this or that, who knows?

[1:04:55] PETER MCCORMACK: Fucked around with CRISPR.

[1:04:56] CONNOR LEAHY: Yeah, fucked around with CRISPR, did some stuff, you know, cooked the atmosphere because it was like doing some kind of geoengineering project or like who knows, right? It'll just do like some shit and, you know, it won't care about humans. And you know, if humans get in the way, obviously it'll have to get rid of them. Like you know, it's very annoying if ants have access to nuclear weapons. So obviously we can't have humans have access to nuclear weapons. That would be very annoying.

[1:05:18] PETER MCCORMACK: You're talking unknown unknowns.

[1:05:19] CONNOR LEAHY: Yes. Unknown unknowns. Like—like it's—it doesn't make sense to even reason about this. Which is why the primary policy objective must be to not get into this situation. If you're in the situation that you're an ant and there's a human that wants to kill you, you've already lost. Like you—you just do not get into this situation.

[1:05:38] PETER MCCORMACK: Right. Is there an escape moment? Because it's still in a box.

[1:05:41] CONNOR LEAHY: It's not. What do you mean? These are all on the internet everywhere. What the hell you mean? There's open-source AIs everywhere. What box?

[1:05:51] PETER MCCORMACK: Okay, so AI has escaped.

[1:05:53] CONNOR LEAHY: Yes. Long ago. What box? We didn't even try to contain it.

[1:05:57] PETER MCCORMACK: Then how can you even contain it now if it's already escaped?

[1:06:01] CONNOR LEAHY: Well, because like there was—there'll be some dudes who'd love to build a nuclear weapon, but like logistically on your own in your garage that's very difficult. But a few nerds with the right tools could do that. I think that's exactly the problem. If there currently is an AI that can bootstrap to AGI, it's probably over. Like it's humanity's probably cooked. It's probably over. But it seems plausible that none of the current AI systems are yet strong enough to get to AGI. They're close, but they're not there yet. And so we still have a time. But once such a system exists and, you know, it leaks somewhere or, you know, Chinese intelligence hacks into the server, steals it, you know, who knows whatever, right? It's over. So the main thing is to not build it.

[1:06:45] PETER MCCORMACK: So there's—but there's only like what, five or six companies capable of doing this?

[1:06:48] CONNOR LEAHY: Probably yes.

[1:06:49] PETER MCCORMACK: And—and we know of those five or six American companies, we're aware of some of the Chinese models. Is there anyone else? Are the Russians, the Israelis, anyone else doing this?

[1:06:59] CONNOR LEAHY: The Israelis have some stuff, um, for sure. You know, there's a little work in the UAE, etc., etc. But like the America definitely dominates the frontier by like a pretty significant margin.

[1:07:10] PETER MCCORMACK: Okay. And so you can regulate those companies. But does this require some kind of, I don't know, agreement with the Chinese? Like we both need to have a conversation here, "Hey, but..."

[1:07:22] CONNOR LEAHY: As I said before, I think unilateral disarmament doesn't make sense. Um, I think everything has to be multilateral. So the general thing I recommend here is conditional treaties or conditional multilateral agreements. Basically you have—you have agreements, trust but verify, you know, similar to what we for example do with nuclear weapons. Um, you know, we don't do superintelligence, we don't do AGI, etc., etc. And then you do it that the—the agreement doesn't go into effect until a certain threshold of people have signed. So for example the Americans can say, you know, the contract, you know, doesn't do anything until the Chinese sign. Or vice versa. So and then you do need to get everyone at the table. Like um, there's a thing where people often tell me, "But Connor, that's hard or that's impossible." And I'm like, "Yeah, well aware." Um, you know, what do you want me to say? The world isn't easy. Um, we could have nuked ourselves in the '50s, we could have nuked ourselves in the '60s and that just would have been it. You know, the world isn't fair. Like um, sometimes you have to do impossible things.

[1:08:18] PETER MCCORMACK: We've done a lot of work to try to stop the Iranians have nukes because we're worried about that.

[1:08:21] CONNOR LEAHY: Yes. We did. And we do have to do the same for AGI. Like I'm sorry, it's like it's just the way the world works. We did a lot of work to stop the Iranians from getting nukes and I think stopping them from getting nukes was a very good idea. You know, neither here nor there any specific military plan what's the right or the bad one. All things equal, we don't want the Iranians with nuclear weapons. We don't want anyone with nuclear weapons. And the same thing applies to AGI, where it's even further than—and AGI in a sense is worse than nukes for the same reason you just described, is that it's software. It's not—it's not, you know, massive centrifuges. Right. Like luckily at the moment it generally requires massive data centers, which are as if not harder to hide than centrifuges.

[1:09:00] PETER MCCORMACK: They're easy to regulate.

[1:09:01] CONNOR LEAHY: They're much easier to regulate than centrifuges are and like, you know, nu—you know, not easier, they're on the same ballpark, same order of magnitude.

[1:09:09] PETER MCCORMACK: Right. And are there more people taking your warning seriously? Do you feel there's momentum behind it?

[1:09:14] CONNOR LEAHY: Yes. Um, it's slow growing and requires a lot of work. Um, but my experience talking, you know, for example to politicians about these issues is that by and far the main response I get is just they had never heard of this before, no one told them. They just didn't know. And you just tell them, "You know, hey, there's some guys in Silicon Valley who are building things that they think could be smarter than humans and they don't know how to control it. How do you feel about that?" And the answer is universally bad. Really bad.

[1:09:42] PETER MCCORMACK: Look, I mean we've covered this a couple of times, but the thing that really stood out to me most of all is that you said they probably know about how about 3% of it works.

[1:09:49] CONNOR LEAHY: Yes. Exactly. And like people don't know. Like yeah. And like again, I—I truly think this is an overestimation. It's—it's real—like I've built these things myself. I cannot stress how weird these things are. How they just do not act the way you think they will and they just do not work the way that you think they would and just like you had no idea what the hell is going to happen until you build it. And then like who knows what happens.

[1:10:12] PETER MCCORMACK: But does Elon know this? Does Sam Altman know this? Do they all know this?

[1:10:15] CONNOR LEAHY: Yeah, they know it.

[1:10:17] PETER MCCORMACK: So what do you think is going on with them? Why are they...

[1:10:21] CONNOR LEAHY: So I generally feel psychoanalyzing people to be rude, um, or ineffective. But um, I think the main thing is is that there is a thing where people expect people to be coherent at all times. And I think most of the time people are just incoherent. They just kind of do whatever is in front of them. Like you just do what gets you power, for example. You just do what gets you the paycheck. You just get whatever all your friends are doing. I think there's a massive momentum.

[1:10:46] PETER MCCORMACK: It's like the Nazis, they were just doing their jobs.

[1:10:49] CONNOR LEAHY: Look, I'm German. Like I—I know how it goes, right? Like I am German, um, you know, and a lot of German people who were part of the Nazis, I don't think they were like insane weird aliens, they were normal people, right? You know, they did very, very terrible things, but they were just normal people doing a normal job from their perspective.

[1:11:06] PETER MCCORMACK: The engineers were just doing their jobs.

[1:11:08] CONNOR LEAHY: They were just doing their jobs, making the trains run on time, you know. It's just, you know, it's not their fault where the trains go, you know. It's like—like—but like this is literally what's happening, right? And this has been happening for a long time now. Like, you know, there's engineers at Meta who for 20 years now have been basically optimizing to get our children addicted to social media, to, you know, make them develop all kinds of, you know, novel mental diseases and so on. And they just go to work every day and they feel fine about it.

[1:11:30] PETER MCCORMACK: Yeah, these people are fucking scumbags. I've been learning about this recently. I—I've been reading Jonathan Haidt's book, "The Anxious Generation." It's like, you motherfuckers.

[1:11:38] CONNOR LEAHY: Yeah. No, like there's a deep thing here where like I think there's a—there's a—in a sense one of the great innovations of the 1990s and the 2000s was that sociopaths learned how to domesticate nerds.

[1:11:49] [visual: Peter laughs and puts his head in his hands.]

[1:11:59] CONNOR LEAHY: But it's true. Like I don't know if you've ever been to like a Meta campus or like Google campus, it's like a playground. It's like you have all these like fun things and it's like exciting and like everyone's having a good time.

[1:12:07] PETER MCCORMACK: There's no reason to leave.

[1:12:08] CONNOR LEAHY: No reason to leave, you know, you can—you can and you can do cool math all day. You could do math all day. And like it's literally the—and then, you know, what the math is used for, what the optimization is used for, well, you know, don't worry your cute little head about that, you know, that's, um, you know, that's the manager's problem, right? Like they'll—they'll take care of it, you know, don't worry about it. And like this is really how a lot of these companies are run. Is that, you know, sociopaths, psychopaths have learned that there's a lot of power you can get out of nerds. Is that math and computers are very useful. And you know, they don't—they're not, you know, necessarily the kind of nerds who can do it themselves, but they can get the nerds to do it. And the nerds get something out of it too, is that as I said earlier, they don't want to be blamed. They don't want to be responsible. They just want to play with their toys.

[1:12:49] PETER MCCORMACK: Get laid.

[1:12:50] CONNOR LEAHY: Yeah, get laid, you know. Salary helps. Salary helps. Salary helps. You know, you get a big salary, you get to play with all your favorite math toys all day long and you don't have to worry about, you know, what the, you know, recommender algorithm might be used for. You know, that's not your problem, you know. You're just an engineer.

[1:13:04] PETER MCCORMACK: So is Meta like Marlboro?

[1:13:06] CONNOR LEAHY: Yes. Yes. Like and in more than just the obvious. Like um, Big Tech, like, you know, obviously does a lot of lobbying. Like unbelievable amounts of lobbying. And their playbook is one-to-one the tobacco playbook. Like it's crazy. Like the—there's a very famous the tobacco playbook, fear, uncertainty, and doubt. And they have learned a masterclass at that all the Big Tech companies are doing the same thing that cigarette companies were doing in the '60s and the '70s when they were trying to stop regulation of smoking and like trying to deny that smoking caused cancer. They're doing the exact same things. They're doing the exact same strategies. You know, sometimes it's the same lobbyists, right? You know, um, doing the exact same things, trying to hide, you know, both the risk from like, you know, say social media and the other technology but also for AI. Is they're just trying to say, you know, "Oh, well we'll have to see what the evidence is. You know, it's unclear, you know, we'll have to, you know, wait until the evidence comes in and like let's hear all sides." And they're stalling for time. Their primary strategy is to stall for time. Same thing they did with cigarettes. Fun fact about cigarette smoke: to this day, we're not 100% sure exactly what chemicals cause the cancer. Even to this day, it's not 100% clear. There's still some ambiguity. So back in the '60s and '70s, the tobacco lobbyists would always say, "Well, of course, once we know what the mechanism is, well then of course we should regulate. But let's wait until the science comes in." You know, while we all, you know, sm—get everyone addicted. And the same thing is happening with AI companies. They're always like, "Well, once the hypothetical risk manifests, of course we will act. But let's wait until, you know, it comes to that."

[1:14:40] PETER MCCORMACK: Are there AI lobbyists?

[1:14:41] CONNOR LEAHY: Oh, hundreds, thousands. It's like the largest lobby in the world right now. Recently, um, Andreessen Horowitz and several others put up a $200 million super PAC, which is the largest in history, to lobby against AI regulation. It's like the largest lobby in the world right now. It's unbelievable.

[1:14:59] PETER MCCORMACK: You can't—you're not allowed to speak mean words of Andreessen Horowitz.

[1:15:02] CONNOR LEAHY: Oh, please—please come at me. Like I—please make me a martyr.

[1:15:07] PETER MCCORMACK: Isn't it your—isn't it an eviction notice from Silicon Valley if you criticize them?

[1:15:11] CONNOR LEAHY: Great. I'm already blocked, you know, so um...

[1:15:15] PETER MCCORMACK: What, by Marc?

[1:15:16] CONNOR LEAHY: Oh yeah. Everyone's blocked by Marc. So please, Marc, please come after me. Like—like all these people, like it's just—like there's no morals here. It's just fighting. It's just a fight, you know. It's them or us. Like what the hell do you want, right? Like they people are lobbying for no regulation, they're against fucking everything. They don't care about this. They don't care about the risk. They don't care about any of this. Like and honestly, I'm not mad. Like I—for me dealing with like very sociopathic people is kind of like dealing with wild animals. Like if you have a wild animal in the cage and you put your hand in, you get bitten. Whose fault is it? It's not really the animal's fault. It's your fault. You put your hand in. This is how I feel about a lot of like, you know, cutthroat capitalism. Like if I get fucked over by a CEO in like a corporate takeover, I'm not really mad. I know what the game is. I'm trying to fuck him, he's trying to fuck me. Like I know how it is, right? And this is how a lot of this is. Like these people are not your friends. Corporations are not your friends. These CEOs are not your friends. They're wild animals. That doesn't mean it can't be useful, you know. Wild animals could do a lot of good things, you know. I think that's one of the best parts of capitalism is that you should fight. You should have, you know, wild animals fighting each other to make the best product.

[1:16:27] PETER MCCORMACK: You just need a referee.

[1:16:28] CONNOR LEAHY: You just need a referee. Like there's a reason that we all watch MMA and we don't watch, you know, shitty street fights, you know. If you want real combat, you can watch, you know, some Russian drunks, you know, hit each other in the head and die on LiveLeak anytime. You know.

[1:16:41] PETER MCCORMACK: Dude, in MMA the minute he lands the first big punch, he's on the floor, you—you want the ref in. Get in there, stop it. You don't want when that second hit comes, it's like, "Fuck."

[1:16:48] CONNOR LEAHY: Yeah, exactly. No, no, exactly. Like MMA has a lot of rules. It's a—like I—I find it so inspiring actually that you can have, you know, these like huge, you know, like dangerous men, you know, fight in the peak of physical combat like throughout history. Like throughout history there've never been people better at fighting like one-on-one than the people alive, you know, today, right? And then and we still have, you know, the referee who will dive in and make sure that people don't get hurt, you know, too bad at least. Like I think this is incredible. I think this is great. And this is also how I think about capitalism. Like I do think, you know, companies should whale on each other, but, you know, if the public's health is at risk, you know, or geopolitical security, the referee should fucking dive in.

[1:17:26] [visual: Black screen transition.]

[1:17:27] PETER MCCORMACK: This show is brought to you by my lead sponsor IREN, the AI cloud for the next big thing. IREN builds and operates next-generation data centers and delivers cutting-edge GPU infrastructure, all powered by 100% renewable energy. Now, if you need access to scalable GPU clusters or are simply curious about who is powering the future of AI, check out iren.com to learn more, which is I-R-E-N dot com.

[1:17:52] PETER MCCORMACK: All right, man. Listen. Steelman me the case for optimism. Like what is the strongest argument that this is not an existential risk?

[1:18:03] CONNOR LEAHY: I don't like steelmanning because steelmanning I think is a kind of lying. Um, the reason I say that is that I can make up an argument that isn't theirs. You know what I mean?

[1:18:14] PETER MCCORMACK: Yeah.

[1:18:15] CONNOR LEAHY: So instead I can give you a true argument for optimism. Instead of giving you a steelman, I can give you a true argument that I for example believe. Um, I think the obvious thing is that um humanity has not yet lost. We can in fact make decisions. We have for example regulated, you know, chemical companies, nuclear companies, etc. Like, you know, back the 1950s we were dumping fucking toxic wastes into like, you know, every river, right? And we did stop that. Like the referee did dive in eventually, we did actually make it stop. You know, not perfectly, not everywhere in the world, but, you know, we did do a lot. I think maybe the strongest, simplest case for optimism is that it is really in no one's interest for AI to take over. Not even Marc Andreessen. Like even Marc Andreessen I think is shooting himself in the foot. Like I don't think he's helping himself. It's not in his interest for these AIs to, you know, take over all this stuff and kill him. That's not in his interest at all.

[1:19:12] PETER MCCORMACK: But do you think perhaps they're trying to capture as much ca—of the market, as much capital as possible in the short term, but they know themselves, "Mate, like in a year we'll stop this"?

[1:19:21] CONNOR LEAHY: I don't think so. Because I think—to be clear, I think they say this, but I don't think the way people act in practice is through momentum. So I've had conversations with some of these CEOs. You know, I'm not going to say who it was, but like I've talked to most of the lab CEOs. You know, I've talked to Demis, I've talked to Sam, I've talked to Dario, I've, you know, I've talked to all of them. And um, many of them will tell you in private, you know, like, "Of course we're so concerned and of course, you know, when the moment comes, you know, we will do the right thing, blah blah blah." Of course they'll tell you that, right? But then every time I dug in, I'm like, "Okay, but like when is that moment?" And they're like, "Well, we'll have to assess." And I go, "Okay, yeah, yeah, but—but for real. Like what is the thing? Like what is the—what is the signal? What's the that makes you do something? And like what will you do?" And they're like, "Well, you know, we don't know yet, you know, it's still far away." And I'm like, "Okay, fuck. Like there's no plan." Like it's important to understand that there is no plan. These people don't have a plan. No one has a plan. It's not like there's some secret, you know, Batman super, you know, hero plan that comes into effect, you know, as things go. Like a lot of conspiracy theorists like to believe that there's like a shadowy cabal that runs the world, that's super competent, they have—they know everything, they control everything, everything is part of their plan. And the truth is it's much worse than that. It's just chaos. No one's in control. Elon Musk is not in control, you know, Sam Altman's not in control, you know, the EU is not in control, US not in control. No one's in control. It's—it's chaos. And that's I think the—the main problem.

[1:20:48] CONNOR LEAHY: So in a sense this is very pessimistic, but I also think it's optimistic. Because if evil was in charge, I think we would be fucked. Like if there was a shadowy cabal that had full control over the planet and was like, "Fuck you, I'm going to replace the humans with AI," I think we would be screwed. I don't think this is the case. I don't think it's in anyone's interest in—like Elon Musk has said many times he doesn't want to be replaced by AI, right? You know, and like, you know, Dario and Sam have also... like I don't think it's in anyone's interest. I just don't think they will unilaterally act on any of these things.

[1:21:19] PETER MCCORMACK: Do the—do the AI optimists get anything right? Or what do they get right most? What is the thing you think they do right?

[1:21:27] CONNOR LEAHY: I think they um don't submit to—they have learned an important lesson, which is that most pessimism is very uncalibrated. Um, I think it's very important to say like I'm a techno-optimist and was a techno-optimist for most of my life. I think one of the things that they get very right is that in fact throughout most of history, whenever someone a new anything happened, everyone would bitch about it. Everyone would be like, "Oh, electricity, it's going to ruin our society." Books! Socrates said this. "No one can remember anything anymore, books terrible, we should never have books," right? And yeah, I think it's good to see that that wasn't correct. Like actually the world did get better and it got better in many ways. I think it's very easy to have the cynicism for example that the world right now is not the most peaceful it's ever been. And look, I know that there was just, you know, a massive, you know, strike on Iran and like there's like a lot of shit going on. But again, like 70 years ago we had World War II. You know. Like, you know, look at the Middle Ages. You know, France and Germany were just slaughtering each other basically as fast as they could get soldiers, right? And like this is just not a thing that happens anymore. Or like the Black Death wiped out a third of Europe, you know. That just does not happen anymore. Yeah, COVID wasn't great, like definitely caused a lot of damage, but like it's nothing compared to like the horrors of... and we should be thankful for this and we should be excited by this. We should be excited by what was possible with technology. All the—all the limits we don't have anymore. I just don't think it's all of them. That's all. Like...

[1:22:50] PETER MCCORMACK: Right. So if you're effective in what you think should happen with controlling AI a bit better, what is it—what will have happened within the next 12 to 24 months?

[1:23:03] CONNOR LEAHY: In the next 12 to 24 months, the—so the way I like to think about it is that the goal is not any specific policy. Like this specific law. Like let's say tomorrow we had a magic wand and we could pass any legislation we want in the entire world. And let's say we pass the "Ban Superintelligence Forever Act." What happens? Well, in one week it gets violated in spirit and in two weeks it gets violated in the letter. Because no one will believe in it. No one will enforce it. So it's not enough to like you—this is not something we can win on a technicality. It's not like we pass the one specific bill which technically, you know, in paragraph 74 makes it so that you can't do this anymore. That—that won't count. What has to happen is that we as humanity and like a large enough coalition, you know, both of elites and the general public, have to decide we don't want this and we're going to figure out how to make it not happen and how a better future can happen.

[1:23:54] CONNOR LEAHY: The way a good future looks to me is a future where chaos is no longer in control, but humanity is in control. Is that we make choices. And you know, if humanity makes choices that I disagree with, I'm open to it. If 99% of humans, you know, voted and they all said, "Screw this safety stuff, let's, you know, do AI, I don't care," fair enough honestly. I think this is morally way more acceptable than what's happening right now. I would disagree, you know, I would argue against it, but I'm open to it, right?

[1:24:21] PETER MCCORMACK: We're getting something we didn't vote for.

[1:24:23] CONNOR LEAHY: Yeah. Like it's like voting for—it's like—it's like going to war without congressional support. Yeah, it's like, look, war's bad, but like if people vote for it, if the proper procedures are followed, you know, I'm open to it, right? And like I think legitimacy and like who gets to make these choices is kind of the big question. So there needs to be a group of people, um, you know, both, you know, among politicians, the general public, military, you know, intellectuals, media, everywhere, who say, "This is not the choice we want." Who is currently there is a small unelected minority, you know, of people who are making the choice to expose all of humanity to the risk of extinction from superintelligence, to getting displaced, to getting to no longer being the smartest species on the planet. This is not a decision that should be made by private people. This is a decision that humanity gets to make, that governments get to make.

[1:25:17] PETER MCCORMACK: I wonder what the tipping point is. It feels like it's more likely to come from a meeting of Sam, Elon, Dario, the people you've named, rather than the politicians, because the politicians are influenced by the lobbyists and they'll just fuck everything up.

[1:25:28] CONNOR LEAHY: My feeling about this is kind of similar to what we're talking about with the Department of War and Anthropic. Is that I think they're more like wild animals in the sense that they will react to their incentives. Like they will just do the thing, right? Like imagine if tomorrow the CEO of Google became convinced Google's a massive threat to humanity. So he goes into the office, jumps on the table and yells, "Shut it down! Burn all the servers! Delete everything!" What happens?

[1:25:52] PETER MCCORMACK: He stops being CEO of Google and he gets thrown in a loony bin.

[1:25:55] CONNOR LEAHY: Exactly. So I don't think Sam or Dario or Demis can stop. I think they will just lose their job. Like I think they could just say that tomorrow and they would just immediately lose their job. Like I—I think they should, to be clear. I think they should do this. I think they should jump on the table and do this and then get fired. I think that would make them heroes. But they were obviously not doing that because they were selected to be the kind of people that don't do that.

[1:26:18] PETER MCCORMACK: Well, so let me ask you a question. These safety engineers who've been resigning... is that the version of jumping on the table they can do within the limits of probably some contractual thing they've signed?

[1:26:30] CONNOR LEAHY: I expect so, yeah. Yeah. For many of them. I know some of them personally and that's my feeling, at least from the ones I've personally talked to. Yeah. Like I think there's a thing here where this is just not a problem that is solved by individuals. This is a thing that's solved by groups, that is solved by, um, you know, people.

[1:26:47] PETER MCCORMACK: Coalition. Coalition of the willing.

[1:26:50] CONNOR LEAHY: One of the things that I think is even maybe more important—not more important, but like as or more important than the rapid increase in AI—is the lack of government capacity and state capacity. If the same thing that's happening right now had happened say in the 1950s America, I think the world would be very different. The government back then, people would just actually do things, you know. You know, citizens' assemblies would come together, you know, whenever an issue was, you know, and like politicians took their jobs extremely seriously. You know, like the Watergate scandal, I would like to remind you, one of the—that like impeached Richard Nixon was that he um spied on one Democrat once. And he got impeached for it, right? Like the amount—like this is how the high the level of integrity... you know, I'm not saying there weren't terrible things done by these governments, right? But there's a kind of state capacity. There was like a thing of taking things seriously that is like very absent in the modern world. When I talk to politicians, the main thing that strikes me is not that they're evil, you know, psychopaths. Some of them are, but it's not as many as you think they were. Most of them are just normal people that are extremely overwhelmed and just like trying to do the right thing but they don't know how. And they're like stuck between a rock and a hard place, like no matter what they do, you know, their party's screaming at them there, general public's screaming at them there, experts are screaming at them here, activists are screaming at them there, and they just don't know what to do. I think we should like as a civilization like take a step back and acknowledge that this is a fucked up situation to be in. Like we're in a bad situation. A—we are facing a problem that is like two levels harder than what our governments are like built for, um, at what our civilization or culture is built for. You know, um, if we were a wiser civilization with like competent government, you know, that, you know, is really good at regulating this kind of stuff, as I said, we would have acted already long ago. So the qu—but ultimately we're not. So the question is much more, how do we get there? How do we get to a world where humanity can make choices, you know, where the citizens and government and so on can actually have the will of the people enacted as it is instantiated as it is. You know, there are worlds in which for example we need to persuade people that superintelligence is bad. That this is really not the bottleneck. People do not need to be persuaded. This is not the hard problem. The hard problem is kind of like the plumbing. Like how do you build institutions that can actually act? I think this is very hard and it takes a lot of time. And I'm not sure we can do it in time before, you know, these things happen. And to be clear, all these companies and all their lobbyists are trying as hard as possible to prevent this from happening. They're trying to weaken the government. They're lying to the government, they're manipulating the government, they're trying to undermine it, trying to undermine its legitimacy and its capabilities, you know, sucking all the talent out of the public sector, etc. So it's not a surprise that we're in this situation.

[1:30:11] PETER MCCORMACK: What happens if you fail?

[1:30:12] CONNOR LEAHY: Well, as I said, what we expect. Um, it will be very confusing and then one day we'll not be in control anymore. Um, we'll probably have some awesome AI-generated, like video games for like a little while, you know. So we'll get like our AI hyper-porn for a while and then humanity fades into the dark.

[1:30:32] PETER MCCORMACK: Have you had to—like we've done this interview three times now with Max and Andrea and now yourself and through my little years on this planet and my understanding of how this world works, the incentives at play, how capital kind of decides what happens, come to a realization that this actually might just not be possible. And at what point do you just, I don't know, hang—hang your gloves up and, I don't know, just live out your days the best way you want to?

[1:31:04] CONNOR LEAHY: I think it's a fair question. I think it's a very fair question. I think it's a very personal question in the sense that I think everyone has to find the—the way for themselves. For me, um, I don't think that moment will ever come. Like I don't know, I—I think I'm just like built constitutionally the way that like I don't think I could like stop myself from fighting until the bitter end, you know, until someone, you know, forces me to. Um, you know, I—I try to take holidays and um, I'm just like I'm so excited to get back to it. It's like a lot of people fight evil out of compulsion. They fight it because or out of trauma. Like they're scared, they're anxious. And, you know, I understand that and God bless them and whatever, but it's not how I personally feel. For me it's like the most glorious—like the most important thing I could be doing. Like I'm so in a sense happy that I can like try to build a better future for myself, for my children, for other people. It's just so important to me and, you know, it's exciting, it's interesting, it's—it's a good job, it's a life worth living is the way I would put it. Like I don't see—I think what is happening to humanity is a culmination of a horrible tragedy. There is a horrible, horrible just life is not fair. The world is not fair. I can't do anything about that. Like the fact that the universe is like this, I can't do anything about that. But what I could do is have a life worth living. I could do everything I can to make the world a better place. And if I fail, I fail, but it was a life worth living.

[1:32:20] PETER MCCORMACK: Yeah, and to the people who say, "Shut up, doomer," I asked this to Max and Andrea. You're not—it's not that you're not pro-AI, right?

[1:32:29] CONNOR LEAHY: I'm not pro—of course, I think technology is great, I think AI is great, I just think we need to do it properly. I think nuclear power is great, I don't think we should have private nuclear weapons. You know, these things can coexist.

[1:32:39] PETER MCCORMACK: Yeah, man. Wow. Yeah, wow. Okay. Lots to think about, man. Oh, but look, thank you for coming in. Really appreciate the work you guys are doing, you, Max, and Andrea. And good luck with your mission. I think it's an important one. And yeah, lots to think about.

[1:33:00] CONNOR LEAHY: Lots to think about. Thank you so much and, you know, thank you as well. I think it's important, you know, also for all the listeners, viewers, um, that this is something that affects you and it's something that where your voice is important. So if these are issues that you care about that make you think, um, please make your voice heard. You know, go to controlai.com, contact your lawmakers, demand change. If you want to do more, um, I personally lead a volunteer group, um, called the Torchbearer, torchbearer.community, where we work for a humanist better future. If you want to spend two hours a—a week just trying to make the world a better place, consider applying.

[1:33:33] PETER MCCORMACK: Thank you, man. And thank you to everyone for listening. This show is a bit weird. Oh man, lots to think about. Thank you. Appreciate you, man.

[1:33:42] CONNOR LEAHY: Thank you.

[1:33:43] PETER MCCORMACK: Appreciate all you listeners. We're not AI bots yet. We'll see you soon.

[1:33:47] [visual: Outro sequence for The Peter McCormack Show plays with music and stylized graphics of Peter McCormack.]