Does AI spell the end for humanity's win-win era?

'What is technological evolution asking of us? What is our calling?' my friend Bob Wright poses, in the opening chapter of his new book,The God Test. 'Is it possible that the future, like gods of the past, is demanding that we move toward spiritual progress, toward a kind of enlightenment—that we become better versions of ourselves? And will we, as some of those gods mandated, pay a steep price, even an apocalyptic price, if we fail at that mission?' Twenty-six years earlier, in the opening pages ofNonzero, the book that introduced me to Bob's work, he'd asked similar questions: 'Our species seems to face a kind of test toward which basic forces of history have been moving us for millenia. It is a test of political imagination—of our ability to accept basic, necessary changes in structures of governance—but also a test of moral imagination. So how will we do on this test?' Now, of course, there's much to navigate with AI. In Bob's view, this vastly increases the urgency of the test we face, along with the chances of a failing grade. InNonzero, he admitted to the possibilities of annihilation—'On the other hand, we could blow up the world'—but ultimately seemed far less invested in that proposition than by how technological innovation reliably compels new levels of human accord.InThe God Test, the reverse is true. We're heading somewhere very fast, Bob observes, including, perhaps, 'dystopian hellscapes,' 'terrifying chaos,' and 'brutal authoritarianism.' And those are some of the less-bad options. We could also be 'enslaved or slaughtered by a pitiless superintelligence…' Which is not to say that Bob has suddenly shifted from Pollyanna to full-blown doomer. In characteristic fashion, he arrives at his positions with a genial, thorough, and nuanced touch, surfacing novel connections across different disciplines and grounding sweeping pronouncements in telling and entertaining details. Thanks for reading Theory of the Game! Subscribe for free to receive new posts and support my work. In addition, a dark shadow has always lain at the heart of his relatively sunny outlook. Part of what makesNonzeroand what Bob callsnonzero-sum-nessso persuasive is the pessimistic take on human nature that informs it. Using game theory as his lens, Bob makes the case that self-interest has always driven human beings to seek out non-zero-sum 'games,' where the outcomes of players overlap, over zero-sum games, where there are clear winners and losers. Whether it's hunting together versus hunting alone, or inventing knowledge-sharing technologies like the printing press, humanity has consistently steered toward social arrangements and technologies that make cooperation the safest, most rational choice for all players. Not out of the goodness of anyone's heart, Bob emphasizes. And not exclusively, or with uniformly pro-social results. Zero-sum games persist. Non-zero-sum games can lead to lose-lose outcomes. They can produce externalities that are not uniformly beneficial. Overall, though, Bob's long-standing contention is that non-zero-sumness, the web of win-win interactions that accrue over time, creates net-positive gains for humanity, with a kind of ongoing flywheel effect. Consider how the invention of the printing press led to more books, which led to faster knowledge diffusion, which led to more inventions. New inventions created new incentives for more complex social structures and institutions that could fully capitalize on them, which in turn led to still more inventions. In this way, humanity went from hunter-gatherer villages to cities to nation-states, from local telegraph systems to the global internet. At the same time, one persisting effect of this dynamic is that continuously growing social and technological complexity also introduces new kinds of risk. Global aviation gave us Paris vacations and fresh sushi from Tokyo, and also faster-spreading pandemics. Global supply chains make goods cheaper, but also more vulnerable to events that happen halfway around the world. A world that can produce automobiles, skyscrapers, and nuclear weapons at scale is also a world that eventually has a pressing need for international weapons treaties and climate change policies. The internet enabled superpowers that billions of people now consider default essentials for daily living, but it also empowered bad actors of all kinds and put new strains on social cohesion, shared truth, privacy, and adolescent development. A technology that reshapes how billions of people work, learn, and relate needs continuous assessments, refinement, and governance.Cooperation, in other words, typically demands more cooperation—from parties who aren't always inclined to extend it, especially if previous non-zero-sum wins have given them new powers to explore new ambitions. We did, after all, get World War I and World War II along the way to a more tightly connected planet. If, as Bob asserts inNonzero, non-zero-sumness expands the 'circle of moral consideration,' it also expands the blast radius of lethal, zero-sum aggression.Now, enter AI. The idea that what Bob calls the 'arrow of human history' is pointing toward 'some kind of global cognitive collaboration, toward a larger cooperative enterprise than our species have ever mustered before' is a central theme of his theory of nonzero-sumness. Along with the internet, AI naturally plays into such scenarios. In fact, it's pretty clear that AI is the only technology that makes a highly complex but still somehow coordinated planet even thinkable. For example, you'd need instant AI translation to make the network's edges mutually intelligible. You'd need AI to metabolize a noisy planet's increasingly massive information flows into something a decision-making layer could act on—forecasting, modeling, and surfacing signal from a volume of data no human institution could parse. But where 'global cognitive collaboration' was an aspirational endpoint inNonzero, the ostensibly better place we'd get to if we passed the test (and 'didn't blow up the world'), 'global cognitive collaboration' plays two roles inThe God Test. On the one hand, it still might be the promised land. On the other, it could be the thing that leads to our downfall. What accounts for this shift? In part, Bob is worried that AI is getting better and we're not. 'In the most plausible [AI] takeover scenarios, the basic impetus….is us: human beings acting the way human beings have been known to act,' he writes. Thanks for reading Theory of the Game! Subscribe for free to receive new posts and support my work. Here, he's not even proposing anything particularly diabolical. It's just that, in his estimation, users will, in characteristic human fashion, want AIs that can bend the truth, because sometimes that's what it takes to protect and pursue one's interests. Life is full of zero-sum games, or at least games that can be played in zero-sum fashion. Think salary negotiations, custody disputes, college applications, and insurance claims, to name just a few. When we retain human agents of various kinds to represent and/or guide us on these and other important matters, Bob notes, we generally want someone who will unequivocally advocate on our behalf. And often what's in our interests, or what we believe to be in our interests, is not what's truest, or fairest, or based on a dispassionate analysis of all available data. So in Bob's estimation, people will naturally want AIs that are highly adept at deception, flattery, and power-seeking, because such traits will make them more effective proxies for us. And even if only some people want this, an arms race dynamic will kick in. If your scrupulously ethical AI, which always tries to act honorably and consider a counterparty's interests, is going up against one that ingeniously misrepresents the truth when necessary and deftly cuts ethical corners, which one is more likely to enjoy an advantage, especially over time? Which one will a majority of people choose to use? Which one will profit-seeking developers develop? However problematic these dynamics may be in the short term, they grow even more alarming over time. '....When it comes to evolution, the question isn't just what traits are possible,' Bob writes. 'It's what traits will get selected—which mutations will survive and proliferate.' At this point, theBlack Mirrorepisode begins to write itself. AIs that are growing more autonomous, increasingly proficient across multiple dimensions, and more embodied in various physical manifestations are also being selectively bred to be deceptive, manipulative, and power-hungry. It is, no doubt, a wicked problem, and ultimately what makesThe God Testa novel and essential contribution to current AI discourse. In Bob's estimation, the real alignment challenge we face is not that we won't be able to get AI to do what we want, but that we will. I also appreciateThe God Test'semphasis on speed and scale. Development continues at a rapid pace, and this creates increasing strains as the stakes grow higher. AI is also both an extremely general-purpose technology and one that benefits from what is perhaps the greatest non-zero-sum technological triumph ever, the internet. As much impact as, say, the Ford Model T had, imagine if it had been distributed globally the instant it rolled off the line—for free—into a world already fully outfitted with all the infrastructure it would require (highways, gas stations, traffic signals, etc.) and all the downstream innovations it would enable (drive-ins, motels, suburbs). Then picture those cars coding, translating, generating videos, and doing countless other tasks along with driving. AI isn't just another technological wave. It's an intelligence layer dropped directly onto a hyper-connected, pre-existing global infrastructure that took massive amounts of mutually interdependent coordination to build. In the face of all this, Bob concludes that 'slower AI progress has some big advantages over faster and faster progress.' In particular, he's concerned about global competition and the potential negative consequences of that, including, say, a country trying to pre-empt another on the verge of attaining superintelligence by launching a nuclear attack.Instead of ratcheting up international arms races, he suggests, we should be pursuing treaties and alliances and other kinds of non-zero mutuality. A core tenet of bothNonzeroandThe God Testis that non-zero-sumness pushes human societies toward more complex and overarching forms of governance—first the city, then the nation, then the globe. InNonzero, Wright referenced the work of Pierre Teilhard de Chardin, a Jesuit paleontologist who observed in the 1920s that early communications networks were already connecting individual human minds into what he dubbed the 'noosphere,' i.e., a planetary layer of collective human thought and consciousness that would, through increasing interconnection and convergence, lead to more coordinated global behaviors and actions. InThe God Test, Bob notes how AI might accelerate the onset of what the influential Oxford philosopher Nick Bostrom calls a 'singleton,' a world order with a single decision-making agency at the highest level, with the power to preempt any rival.Clearly the prospect of a singleton is massively concerning. Get it wrong, and you could have an authoritarian government, forever. Get it less wrong, and you'd still have some form of governance, forever, with the power to pre-empt any rival. What is ultimately best for humanity, in my view, is broad human freedom to pursue distinct paths and values within a framework of non-zero-sum cooperation on the problems that require it, and national sovereignty everywhere else. So I agree with Bob that prudence is essential and that higher levels of international cooperation are worth pursuing. But I also strongly believe that leading in AI development is the most prudent path to international cooperation. Thanks for reading Theory of the Game! Subscribe for free to receive new posts and support my work. After all, as Bob's own books make clear, non-zero-sum thinking takes many paths. Trade and barter often arise spontaneously, because the benefits are clear to all parties. Governance, especially when it manifests as verified constraints, enforceable commitments, and monitoring regimes, is another matter entirely. In game-theory terms, cooperation among rivals is stable only when defection is deterred, and deterrence often requires exactly the capability advantage that Bob diagnoses as the disease. Every rung on the ladder he charts inNonzerowas underwritten by power. Treaties hold because someone can punish the party that breaks them. Even democracies use force to cohere; tax compliance is 'voluntary' only because enforcement makes it so. Similarly, the postwar liberal order, the very set of institutions Bob thinks we need to scale even further to ensure positive AI outcomes, has held for as long as it has largely because Pax Americana backstops it.I also think Bob discounts the degree of non-zero-sumness that informs the space already. Certainly, we see a lot of 'arms race' talk, and also explicitly zero-sum measures, like U.S. export restrictions on advanced semiconductors and fabrication equipment, applied to China. But it's also true that today's frontier models are trained on chips designed in California, etched by Dutch machines, and fabricated in Taiwan. Then they're deployed across global telecommunications networks to billions of people equipped with smartphones more powerful than the mainframes that sent Apollo 11 to the moon. Built on shared protocols and open standards, the internet itself persists as a noospheric triumph, attaining almost universal global opt-in—a substrate that remains open to all, on their own terms, even when the platforms built on it have not. AI maintains this pattern, in part because the technology itself and the culture surrounding it are comparatively diffusive by nature. So far, at least, frontier capabilities are replicated quickly, and often open-sourced, typically within months, not decades. Benchmark testing and scholarly publishing are well-established as knowledge-sharing norms. No doubt we could, and should, have even more transparency and accountability than we have now. But compared to, say, the Manhattan Project, AI development has proceeded in an open and collaborative fashion.That's likely to continue, because AI development isn't containable behind a single choke point. It rides the same open protocols, the same cloud infrastructure, and the same global talent market that has made the internet era so non-zero-sum in the first place. A country or lab that 'wins' a given capability race still has to recruit and maintain customers, publish papers to attract talent, and train on data built by the whole world. AI value arises primarily through its integration into a global system of great social complexity and mutual interdependence, with many different players playing key roles in the system's ongoing evolution. Even chip export controls end up reinforcing AI development's status as a non-zero-sum phenomenon: You wouldn't need to create policy-driven hardware bottlenecks for a technology that wasn't diffusing rapidly on its own. So will we pass the test that AI's presenting us now? Like Bob, I believe the stakes are high here. The risks are real. But I also think the odds are in our favor, in part because the elements Bob cites as the factors that make AI exceptional, also make it hard to monopolize. Between its speed, its scale, and its vast general-purpose utility, AI is generating new points of leverage faster than any single actor can consolidate them. A singleton needs at least one chokepoint it controls completely, and ideally, multiple ones. Breach one, and change becomes possible. Of course, the wild card here is recursion: If AI begins to recursively self-improve—each generation of the technology helping design the next, in a feedback loop tight and fast enough to produce a 'hard takeoff'—an ASI-equipped frontrunner could potentially engineer its way out of the dependencies that constrain today's major AI players. This is a real concern, but it still assumes a leap from 'decisive lead' to 'unassailable lead' that effectively happens in the blink of an eye.Or to put it another way, a runaway leader wouldn't just need to out-think everyone else. It would also need to out-coerce, out-manipulate, and out-operationalize everybody else, all at once, in ways that capture or bypass everything they supply, including chips, compute, cloud infrastructure, and talent, all while assuming that none of those actors notice, react, or compete back in the meantime.And nothing we've seen so far suggests capability accrues in that kind of discrete jump rather than the diffusive, catch-up-prone pattern that's held for every AI generation to date, especially given those knowledge-sharing norms I mentioned above. Consider just one data point: in 2017, when many observers believed that Google had already obtained a kind of de facto singleton status given its leading position in search, eight researchers then at Google, including an undergraduate intern, published, with the company's consent, 'Attention Is All You Need,' the paper that gave the world the transformer architecture, and with it every serious competitor Google now faces. The diffusion took a more granular path as well. Within six years of the paper's publication, five of its authors had left to found AI companies of their own—Cohere, Character.AI, Adept, and Sakana among them. So even if America's frontier labs no longer publish their architectures so freely, that post-transformer exodus matters more. Researchers leave and take their knowledge and methods with them. Capabilities get reverse-engineered and replicated from outputs within months of release. And unilateral secrecy is only as strong as its weakest participant. As long as one major lab keeps releasing frontier or near-frontier capabilities openly, those capabilities are in everyone's hands—and the labs that matter sit in different countries, under no common authority that could make them all stop. This diffusion dynamic shows up in the data: according to the non-profit research group Epoch AI, the gap between closed frontier models and open-weight models isjust four months. Still, we should be paying close attention to the concerns Bob describes so lucidly inThe God Test, especially as developers around the globe push frontier models toward new levels of autonomy and capability. In this context, speed does compound risk. We should pursue more coordination, not less. That's why, ever since deep learning breakthroughs like AlexNet and DeepMind's Atari-playing agents began to make it clear something categorically new, with world-changing implications, was underway, I've been doing everything I can to work with and support leaders, labs, and companies with humanist aspirations. To this end, I was one of the first funders of OpenAI and its quest to develop AI as 'an extension of individual human wills' with broad and accessible distribution. With DeepMind co-founder and current CEO of Microsoft AI, I co-founded Inflection AI, a pioneering effort to develop models with emotional intelligence along with raw capabilities. More recently, I co-founded Manas AI with oncologist and Pulitzer Prize-winning author Dr. Siddhartha Mukherjee and Ujjwal Singh, to accelerate discovery of new life-saving medicines. In these and similar endeavors, my goal is always the same: develop AI models and systems that work for us and with us, rather than on us. And develop them prudently, but also with a real bias to action—because, as I've said inSuperagencyand elsewhere, we'll never get the future we want simply by prohibiting the future we don't want.In 2026, as the stakes are rising, I find this to be truer than ever. Active and ambitious participation is more likely to lead to greater coordination than a more neutral or cautious posture is. Ceding the knowledge, incentives, and power that come with maintaining a leading position in AI development offers no guarantee that development will in fact decelerate across the board. Instead, it would most likely just hand the ball, or the chess piece, or the dice—insert your preferred metaphor here—to some other player. With so many opportunities and challenges ahead of us, I believe humanity will achieve the best possible outcomes if players with non-zero-sum mindsets and a people-first focus are assertively playing the game. Thanks for reading Theory of the Game! Subscribe for free to receive new posts and support my work.