Why techno-utopians want to create modern dystopias

One of the things that I really appreciate about June’s book of the month is how it unravels the knotty, complicated mess of beliefs held by all of the people and groups it features. There are extropians, rationalists, longtermists, transhumanists, and effective altruists; there are fans of the singularity, of AI, of space colonization, of immortality… this is a wild frontier, where few people believe exactly the same thing but they are tied together through their belief in parts of it. 

Here are three things More Everything Forever does to document and pick apart these ideas. In a week when Elon Musk became the world’s first trillionaire, I think maybe they’re more important than ever to pay attention to.

More Everything Forever lays out the stepping stones of extreme thinking very clearly.

How does somebody end up genuinely believing that humanity’s future depends—maybe even demands—that we cover the surface of the planet in data centers? Some of the thinkers examined in the book often end up in very odd (and strongly-held) ideological positions, usually achieved by taking thought experiments to their most extreme. 

Becker does a great job of showing how these logical leaps work, detailed but not overwhelming. They’re often extremely fuzzy, generated by a kind of hand-waving mathematical or scientific inevitability or assumption, and once held up to the light of day they feel utterly unconnected from reality.

Take the explanation in chapter four about one line of thinking propagated by the Future of Humanities Institute in Oxford. According to one line of argument laid out (from around page 182 onwards) the best future for humanity to achieve its potential; and the best way to achieve that potential is to create many many times more people; but to do that we will need to colonize the universe; and in order to do that we’ll need to do that we will need close-to-lightspeed space exploration. To achieve this, we’ll need to power those craft with vast amounts of energy; the best way to achieve that is a “Dyson swarm” of small machines that surrounds the Sun in a mechanical, energy-sucking skin that takes up four thousand trillion square kilometers. 

Take a breath.

Although such advances are “entirely trivial for an advanced human-like civilization” achieving this kind of outcome will, in various tellings, require creating a super-powerful artificial intelligence that can overcome the existing scientific barriers; but to get there, we should be doing everything we can to advance AI; which means we’ll need to generate more energy and compute power than is currently possible to fuel the AI; which means building as many data centers as possible, perhaps even covering the entire surface of the earth; and that requires displacing people living in that village down the road and ignoring their protests. So in one way, displacing a community to build a data center is absolutely necessary because the entire future of the human race rests on it. 

I’m not even kidding. But Becker makes the train of thought make sense.

⌘

More Everything Forever connects the dots. These rhetorical leaps are, I think, what helps some of this unusual thinking pass for genuine philosophy. It wears the clothing of philosophy and makes the sounds of philosophy, yet is unbounded by the human rules that constrain genuine thinking. Becker goes further, though, and traces the origins of particular strains of thought back to where they came from—in ways which show how cultish or peculiar the thinkers themselves are. 

There are many examples throughout the book, but perhaps a central one is the idea of “extropy”, a kind of techno-radical strain of thinking that has helped power various beliefs and sects who have since become much more influential. The Extropian mailing list, born in the early 1990s, became a meeting place for many of the people obsessed with these ideas—a small group that now has an incredible sway over society at large given the power of technology companies.

That’s not to say that small groups are inherently wrong: Interesting ideas often start on the fringes and make their way to the middle. But Becker’s dot connecting points show that it doesn’t mean every idea from the edge is valid: sometimes cult thinking is just cult thinking.

More Everything Forever is funny. For all of Becker’s serious invective against some of the wild, dangerous ideas held by the tech oligarchy, it’s also important to remember how silly some of these things really are. 

I don’t mean that the concepts are necessarily funny—though it’s amazing to read how many times the “how” behind some insanely complex assumption is essentially “oh, that will be a solved problem by then” or “the answer will be trivial”—but just it’s important to maintain a sense of the ridiculous. Becker manages to remember that these ideas are just plain odd. Venture capitalist Marc Andreesen’s belief that “slowing down progress on AI is ‘a form of murder’” or 

The best of all these jokes comes courtesy of Ray Kurzweil, the godfather of the singularity. In 2005 he released a book which predicted that we would invent an artificial intelligence that overtook humanity by 2045 (by which time he would, conveniently, almost certainly be dead) and that by 2010 computers would be able to emulate human intelligence; that by 2020 you could buy this capacity for $1,000; and for human intelligence to be mapped by the mid 2020s.

Those milestones may not have been reached, even with LLMs and deep learning and generative AI, but that hasn’t stopped Kurzweil from banging the same drum as he did in The Singularity is Near again, with a 2024 sequel called, in a darkly hilarious moment The Singularity is Nearer. 

It’s like Monty Python or Douglas Adams. I wait eagerly for a new instalment in 2040: The Singularity is Really Actually Quite Near Now, Thanks. 

Behind all of the bluster, behind all of their power and influence, it’s these moments that remind us that these are just odd, discombobulating, dehumanizing things to think—let alone to use your massive wealth to push on the rest of us.

⌘

If you want to join us and read books like this, you can sign up to get Curious Reading Club’s non-fiction recommendations. For as little as $25/month you’ll get your hands on a handpicked hardback title, and we have several options to fit your reading interests and speed. New books get shipped via USPS on the 1st of each month.