“At the moment, LLMs are the worst trade-off” — Karen Hao

It was genuinely fantastic to talk last week with Karen Hao, the author of this month’s Curious Book Club pick, Empire of AI: Dreams and Nightmares in Sam Altman’s OpenAI. It was also great to see a bunch of folks on the call, and really it’s a great honor that authors come to talk with us about their work. 

We covered a lot of ground in our hour-long chat, from the book’s origins to covering the technology industry from different perspectives, to the problems with artificial intelligence and the ways to counteract its negative effects. In all truth, I probably could have talked to her for double the time to get into all the intricacies of her book and what’s going on in the world around us, but the clock comes after us all.

Here’s a lightly-edited version of our conversation. I hope you enjoy.

⌘

Bobbie: One of the most impressive things about the book is that it takes a really big swing—it’s not just about OpenAI or Sam Altman or the things that artificial intelligence advocates believe, but about the whole industry and the price people pay for its advancement. Did you always plan for it to have that big a scope?

Karen: When I started thinking about writing a book, I actually didn't think that I would focus on OpenAI at all. I wanted to build on a series that I did for MIT Technology Review called AI Colonialism. It was a series of stories that was looking at all of these different communities around the world that had been impacted by the commercialization of AI technologies in ways that were bringing them back to echoes of their past.”

One of the stories was about how all these facial recognition companies were coming into South Africa and harvesting data from the population there during a time when the industry was getting all this criticism that they couldn't accurately detect black faces well. South African scholars were telling me, within this context, that the facial recognition companies felt like a recreation of a digital apartheid. There were all these stories that were along the same themes and same patterns. And so I thought: OK, I want to write a book about that. And I started workshopping that idea in January 2022.” 

I workshopped it for quite a long time, then ChatGPT suddenly came out and my agent said “how does this change your argument?” and I was like: It absolutely accelerates every every single argument; they have definitely made this phenomenon way worse. So he said “well, then you have to make it a story about OpenAI, and tell the story of this particular empire.” That came together in March of 2023, so I got the book contract, signed it, and then started working on it, kind of not with too much time pressure. I was taking my time with it, and then the board crisis happened in November of 2023 and my editor was like, “You better wrap this up quick.” So I think all tilt, I ended up spending about a year genuine, like, full time reporting and writing the book, but from beginning to end it was maybe a year and three quarters from the moment it was signed to the moment the manuscript froze.

There’s so much shoe leather in here, so many journeys and trips to see the impact of AI in the places people don’t typically see or think of. Did you have to convince your team that it was worth it?

I didn't ever have to convince my editor that the frame was the right frame and that in order to do that well I had to be on the ground with people. But there was definitely a moment where I had to convince him to give me just a little bit more time to do one final reporting trip, because we were really racing against the clock to try to get the book out while the reporting was still fresh. That was like the hardest part: at the end, there were so many updates happening on a weekly basis that it felt like just even taking an extra month could really cost the book, because it could, it could just start feeling stale. 

Early on I knew I couldn't write this book without doing these trips: it would not be the book that I wanted it to be. In the beginning there was an element of self-consciousness as well: I didn't know how well I would be able to source within the company, and I thought if I can't rest the book on enough insider sources, the other best thing is to really go sprawling out to the far reaches of the empire and document that instead. But the other element was that I made a promise to myself that I'm not going to just write a Silicon Valley book in Silicon Valley. If you tell the story about AI in San Francisco, it's largely going to be a story about how it works and how it's beneficial, because the people that are building the technology, they're building it for themselves with the problems that they encounter. 

So from the very get-go, I said I'm going to set aside a really large chunk of money from my advance to just do this reporting—and fortunately, it ended up working out really well. Sometimes you take a huge leap of faith, where you're going to spend a lot of money to go to a place and then do a bunch of reporting, and then you turn up with nothing. I've had instances where I've done reporting trips where I just felt like I really did not actually get to the heart of the story, or I didn't find the compelling character to really illuminate a particular aspect. 

But it worked out that every reporting trip that I did for this book was incredibly fruitful. In part that was because I did a bunch of partnerships with local journalists who had been covering different aspects of the things that I wanted to do as well. And so we would partner on the ground, and they would spin off their own stories to write in other publications.

Actually, you’re very generous with crediting other reporters throughout the book, right there in the text rather than in footnotes or an appendix.

I was thinking a lot throughout the book that journalism is often also accused of being quite colonial in terms of its history and its practice. The last thing that I wanted to do was write something critiquing one empire and engaging in the same kind of extractive practices by just using a bunch of people's work and not referencing it. So that was the main motivation. And I thought about it for quite a long time. I did get a prompt from my editor at one point: do you really need to add in all these citations? Because it does disrupt the flow a little bit in terms of the narrative. And so I sat with it but then I was like, actually, no, I want to add even more in because there's so much great work out there. 

⌘

So let’s talk specifically about OpenAI. As you lay it out, the basic irony is that this is an organization with a stated original objective of protecting the world from bad AI—but through the things that it has done, it has dramatically accelerated every aspect of AI with an increasing amount of carelessness and voraciousness. When you started reporting about OpenAI, did you see that trajectory for them?

I didn't see the trajectory at all. When I first started profiling them for MIT Technology Review, I went into that with a really open mind of: OK, this non-profit positions itself as being kind of antithetical to Silicon Valley. I took it at face value and was like: let's see how they operate and why they chose to do this. I was asking a lot of just basic questions when I was in their offices: why did you choose to do that; how are you thinking about this; how are you going to approach that? And it just became quickly clear that they were actually just another Silicon Valley company, because they couldn't really articulate much of what they were doing at all, and yet they had gotten an extraordinary, extraordinary amount of money to do it. 

But even then, I just thought we were seeing yet another replay of a bunch of dudes getting boatloads of money with an unarticulated vision to just play around and do whatever they want. I still didn't fast forward and think “And then one day, they'll release ChatGPT and set the world on fire, and everyone will be talking about AI, and then they'll be in this crazy, aggressive race/

The key moment that revealed OpenAI’s main premise, which is the “scaling at all costs” paradigm, they didn't do that until after I profiled them. They were actually already thinking about it, and that was already in motion—this was the jump from GPT2, he model two generations before ChatGPT, which was trained on something like a couple hundred computer chips, and GPT3, which was trained on 10,000—an entire supercomputers’ worth—of chips. And that was just unheard of. 

It wasn't until after I left that they released GPT3, but I think that was the real key moment that they revealed how aggressively they would be willing to accelerate, put pedal to the metal and do whatever it took to get to number one.

They don't talk to you very much right now. But do you get the sense that is that: that they are hell bent forever on this same scale approach?

So why specifically did they choose scale? There were a number of different philosophies and ideologies among the executive team at that time that led them to do that. Ilya Sutskever has just always believed that our intelligence is inherently computational—so it's just a matter of time, if you have more data and more compute, that you're going to recreate the brain… and it is ultimately a scaling game. 

But for Sam Altman, I think he was interested in the scaling approach, both because he comes from the long tradition of Silicon Valley blitzscaling companies, but also because when you take a scaling approach to AI development, the bottleneck becomes money, and that is particularly aligned with his skill set as a fundraiser. 

So in terms of where I see OpenAI continue to go, I think if Altman continues to be the CEO, they will continue to take approaches where the bottleneck is capital, because that is his strength, and it makes the most sense to then direct a company in the places that align with your strengths. 

And that, I think, is part of the reason why open AI's strategy has become increasingly incoherent: because scale is now not a very good way to make AI development any more. The so-called scaling laws have really tapered off, and there are plenty of other open source models that have shown now that you don't just need to scale, you can actually innovate on different techniques and so forth. And OpenAI hasn't really been able to keep up, because they're still on this, “let's scale and use capital and brute force to keep going.” And so now they're going for the other thing where capital is the bottleneck, which is just spray the market with 1,000 different products and services and see what sticks. So I think it all kind of goes back to the fact that Altman is a fundraiser. He is an investor. He is ultimately thinking how do I just use capital to solve the problem?

This book isn’t just about OpenAI. It’s your vessel to talk about this technology and this system and this exploitative approach. You pair technical understanding and explanation with the ability to demonstrate that this is not just zeros and ones: this has physical costs. There is labor involved, there's human effort and cost at every single step except the final one, which is where the user is typing into ChatGPT. And you made it super clear to me just how hungry this technology is. It's hungry for information, it's hungry for natural resources, it's hungry for money, it's hungry for power. But to reach their goals, they’re talking about expanding beyond Earth; like literally, talking about intergalactic scale energy costs. Where the fuck does it end?

The quote that I always think is the most funny and dark is from Ilya Sutskever that I quote right at the beginning of the book, where he says that one day the world will just be covered in power stations and data centers, and that's like the only purpose that this land is for. It's funny, because the things that Sutskever says are always so extreme and so out of this world, but they also often have a truth to them. Like, if we do follow this path to its logical conclusion, he's right, that is basically where we would end up… and at the moment there doesn't really seem to be much of a stopping or slowdown of this trajectory. What I hope will change as more people read my book. 

There was this really amazing piece by Naomi Klein recently in The Guardian called “The Rise of End Times Fascism” and she makes this argument that basically the tech elites have given up on many things. They've given up on democracy, they've given up on the planet, they've given up on organizing ourselves in harmonious societies. These are not necessarily one coherent ideology, but there's sort of multiple different overlapping ideologies that all come to a similar conclusion of what some people in the Valley are now calling the “politics of exit”. We totally screwed up everything, and let's just get ourselves out of here instead of fixing the problems. And I really do think that a lot of the ideas around building AGI, around scaling and not caring about the amount of resources that are being consumed, comes from a very similar mentality of “well, it doesn't really matter. We've already destroyed our Earth anyway, so why not use its last dregs to build something that could bring us to nirvana.” 

Whether or not people within the AI world would actually articulate that or even identify with it… it's just hard to not draw the lines between these different ideologies. Then all of us are supposed to get onto the rocket ship and populate the rest of the universe?

I was speaking with someone during the course of this book, who quite explicitly articulated that argument. I said “this is hugely environmentally damaging” and he said “Yeah, but even if we didn't do this, climate change is going to get worse anyway, and the environment is going to fall apart anyway. So why not then?” Why not use it for something that might have two possible path: do nothing and the world ends, or do something and there's a 1% chance that we figure it out. I don't agree with this ideology. I think it's insane.

One thing that feels different with AI, and large language models in particular, is that people suddenly started ascribing actual intelligence to this technology: giving it agency and anthropomorphizing it and thinking that if I can't tell the difference between a machine and a person, it must be intelligent in a way that I understand intelligence. If we had called the field something other than artificial intelligence, do you think we would still ascribe that intelligence to it? Was this a category error, or maybe a category success by the people behind it, in associating it an LLM with intelligence.

I really do think that if we had called it something else, it would have gone differently. Imagine if we had called a calculator “intelligence”: I think people would have a very different relationship to their calculator. Or if we’d called a laptop “intelligent”, I think people would misconstrue the function of a laptop to be much broader than it actually is.

I think like the reason why we still consider calculators and laptops tools rather than agents that can supersede us or replace us, is because—in part of the name—is understanding that a computer is meant to compute things, and it's not replacing me, it's literally doing something that humans aren't really optimized to do.

So it's a more complimentary relationship, rather than AI just making it sound like, “OK. you're redundant now.”

I could see an argument that that is a little bit oversimplified, but I do think, for me, personally, names have so much power in defining the relationship that people have to things, that it could very well have been the trigger of all of these problems that we see today.

What about the naming of the problem? Roxane Gay recently said that “AI is not useful” as part of this argument about utility—but I suspect utility and usefulness is a terrible framing, partly because it can be useful for some people and it's incredibly damaging for other people. You talk about that a lot in the book. But I also know that saying something is dangerous or damaging seems to have little to no effect anymore. So is there a way to describe the cost and the harm? Is that searching for a name too?

I've just been using the word proportionality. Are the costs proportional to the benefits? It's not that the benefits are zero, but it's almost like one day, we woke up and decided that instead of using any other form of transportation to get from point A to point B, we're all just going to use rockets. Going from the suburbs of LA to downtown LA? Gonna use a rocket. To go from San Francisco to New York? Gonna use a rocket instead of a plane. OK, it gave you utility, you got from point A to point B. But that didn't make any sense, financially, environmentally, public health-wise. It doesn't mean that we should never have rockets—obviously there are certain functions that rockets give us that that no other form of transportation does. 

So that's the thing that I don't see enough of in the conversation is the proportionality of it all. At the moment, LLMs are the worst trade-off: the balance sheets don't balance. DeepSeek is evidence that we could actually, in fact, have LLMs where there is a balanceable way to do this. But at the moment, by and large, the way that people are developing this technology just doesn't add up.

What if it never adds up? If the tech elite are planning to abandon us with the politics of exit, and we’re seeing all this disproportionate wreckage… in the words of one attendee: when are we going to stop being polite about it?

I think about this a lot. When I talk with people who are not familiar with Silicon Valley culture, and I start immediately with this idea of the politics of exit, they just find me completely unbelievable. You want to be a reliable narrator, but these things are so wild that if you start off there, you immediately become an unreliable narrator. So I think there's still quite a lot of work groundwork that needs to be laid to get the public up to speed with the genuine, true heart of the ideologies that we're talking about here and you there. 

There are plenty of people that have already, the Naomi Klein piece I mentioned just came out and said it. There was a podcast recently done by Christopher Wylie, the Cambridge Analytica whistleblower, where he just comes out and puts a name to how insane these ideologies are. [It’s called Captured, with Coda Story on Audible.]

But I was at a journalism conference earlier this year trying to explain to journalists there that they needed to stop striking partnerships with AI companies because they're just cannibalizing their own business, and I was labeled the “AI rearguard”, someone who's put their head in the sand and is so afraid of this technology that I'm clinging on to the last vestiges of like, old school journalism. There's just a huge disconnect, where people are just not seeing the cultural undercurrents that are really deeply problematic in this community.

I was really depressed for a lot of the time that I was reporting the book. But there were all these amazing stories… all these communities that I came across, people who you would think had the least amount of agency in this global power structure, and they actually had the most because they remembered that they could actually do something about it. 

They started aggressively fighting, reclaiming ownership over their labor, reclaiming ownership over their land, reclaiming ownership over their fresh water. And that's how I kind of hit upon my new theory of change around how to really usher in a different vision for what AI development should look like: that every single person should have an active role in shaping the future. 

We should think about the full supply chain of AI development and all of the ingredients that these companies need: the data, the land, the energy, the water, our labor, and all the spaces that they then need access to, like schools, hospitals, businesses, government agencies. We have to remember that these are actually collectively-owned resources and collectively-governed spaces. Carol Cadwalladr has been recently going around saying “do not cede in advance”. I forget what the exact quote is. [She’s often quoted Timothy Snyder’s Lesson 1 on fighting authoritarianism from his book On Tyranny: “Do not obey in advance”… see our book club pick for October 2024 and interview with illustrator Nora Krug.] 

We have gotten so used to it, it’s like “we don't have data privacy anyway, so what is a little more data to these services?” We've lost ownership over these things that we actually do still own. And so we're seeing artists and writers suing these companies, that's them reclaiming ownership over their data. We're seeing hundreds of communities now pushing back against data center development. That's them reclaiming ownership over their land, their energy and their water. And we're seeing teachers and students starting to talk about “shouldn't we have AI governance policies in our schools that actually help facilitate critical thinking, rather than erode critical thinking?” 

I think if we just all remember that as we are being called upon collectively to actually participate in all of this and govern every single stage of the supply chain, we should consider every stage as a site of democratic contestation. We will get to a point where AI just works for us rather than us working for AI. It's not going to be easy, and it will require a massive concerted effort from everyone to do that.

⌘

Karen also gave us four book recommendations that inspired her. The first three were directly linked to the writing of Empire of AI: “The most unlikely one being [Walter Isaacson’s] Steve Jobs, because I really wanted to capture that dynamic narrative pace. I read [Naomi Klein’s] The Shock Doctrine to try and get the reporting-and-argument-intertwined thing happening. And then I read [R.F. Kuang’s novel] Babel to really get into the mood of talking about empire.”

Finally Karen also recommended Hope in the Dark by Rebecca Solnit. “She's really trying to articulate through history, example, scholarship that… there really is a significant amount of action that we can do, even when it seems like there's nothing left. She opens it with remembering that, actually, we've made dramatic changes in the last few decades in terms of progress, social progress, even if it feels on a day-to-day level that everything is going backwards. And those changes happened through people still continuing to act even when they felt like everything was going backwards.”

Thanks so much to Karen and everyone who came along, it really was a great conversation.

Look out tomorrow for our announcement of July’s pick of the month. It’s a fun one that I hope you’ll enjoy.

Onwards

Bobbie