September issue: The hype, peril, and promise of AI

General Dynamics X-62 VISTA (“Variable Stability In-flight Simulator Test Aircraft”) experimental plane in flight. According to the Defense Visual Information Distribution Service, one application for the X-62 and its kin is as an autonomously piloted aircraft, perhaps as a robotic wingman to a manned aircraft. Image credit: US Air Force / Kyle Brasier

By John Mecklin, September 11, 2023

General Dynamics X-62 VISTA (“Variable Stability In-flight Simulator Test Aircraft”) experimental plane in flight. According to the Defense Visual Information Distribution Service, one application for the X-62 and its kin is as an autonomously piloted aircraft, perhaps as a robotic wingman to a manned aircraft. Image credit: US Air Force / Kyle Brasier

Throughout history, advances in technology have led to changes in power relations among countries—and changes in modes of warfare. The industrial revolution that occurred in the late 19^th and early 20^th century, for example, led to the rise of Great Britain and Germany as world powers and played a large role in two world wars. World War II was fought with tanks, airplanes, and ships powered by internal combustion engines and the oil that fueled them; indeed, that war can be seen as focused significantly on access to oil. Today, the technological advance seen as most likely to fundamentally rearrange global power relations involves machines that think and the data they’re trained on (leading some media observers to call data the new oil). In his book, Four Battlegrounds: Power in the Age of Artificial Intelligence, Paul Scharre surveys in matter-of-fact detail the struggle for world leadership in AI, chiefly as it relates to US-China power dynamics.

A vice president at the Center for a New American Security think tank and an emerging military technology expert, Scharre breaks down the competition for military, economic, and political power in the nascent AI era by analyzing national capabilities in four areas that are vital to AI development. These areas (or as Scharre calls them battlegrounds) center on access to the large amounts of data on which artificial intelligence models can train; the computing hardware needed to train AI; the talent that various countries have in terms of AI research; and the institutions—educational, corporate, and governmental—needed to support AI research. The United States and its European and Asian allies are vying with China for leadership (or at least capability) in these key areas; different countries have different advantages. As Scharre explained while we talked about his book recently, successful efforts to manage and negotiate access to these inputs could be the basis for relatively peaceful and safe development of AI technology around the world. History is full of examples of the dreadful carnage that can happen when efforts to accommodate national rivalries fail. This time, however, the killing would happen with machine speed, precision, disinterest, and efficiency.

Editor’s note: This interview has been edited and condensed for readability.

John Mecklin: Your book is very wide-ranging, and it’s a little bit difficult to encapsulate in two or three sentences. So rather than me try to do that, why don’t you tell readers, a lot of whom won’t have read your book yet: What’s the overall thrust? What were you trying to explain?

Paul Scharre: The central question in the book grapples with is: How is the artificial intelligence revolution changing global power? Artificial intelligence is a general-purpose technology, much like electricity, or computer networks, or the internal combustion engine. And when we look at past industrial revolutions, nations rose and fell on the global stage based on how rapidly they industrialized.

And prior industrial revolutions even changed the key metrics of power. During the first and second industrial revolution, coal and steel became key inputs of national power. Oil became a geostrategic resource that countries are willing to fight wars over. And so what is that in an age of AI? And the book concludes that data, computing hardware, human talent, and institutions—organizations that are able to turn these raw inputs into useful applications—are the key battlegrounds of competition in an age of AI. And then the book talks through what some of the international security implications could be of the AI revolution on warfare, domestic surveillance, disinformation, and other domains of global competition.

Mecklin: The book seems to circle around sort of a central dilemma: How can the United States compete, power-wise, in that kind of a world, without engaging in some kind of race to the bottom with more authoritarian regimes? Why don’t you talk a little bit about how the United States can keep from becoming one of the AI bad guys.

Scharre: It is a core dilemma that comes up again and again throughout the book in different ways. I’m coming from a think tank based in Washington, and so I’m very much coming at this issue from a Washington-centric mindset, looking at the broader US-China geopolitical competition, of which AI is an important component. And certainly among US national security leaders, they are very concerned about China’s progress in artificial intelligence. China is a major global AI powerhouse. China’s said their intent is to be the global leader in AI by 2030. And I took them seriously with that goal. They’re certainly working hard to improve their AI capabilities in academia and the government, and in industry inside China.

But I think that this competitive dynamic has often led to perverse incentives by us policy makers. One obvious one is in the area of facial recognition. China has half of the world’s one billion surveillance cameras. And we’re increasingly using artificial intelligence for tools like facial recognition or gait recognition.

And one consequence of this widespread deployment of facial recognition is the Chinese companies have a leg up on US companies in maturing the technology. They’re getting more data. And they’re getting an opportunity to refine their systems and algorithms based on being deployed in the real world. Whereas in the United States, we’ve seen a grassroots movement against facial recognition in cities and states across the country, in many cases, banning facial recognition for law enforcement. Amazon, IBM, and Microsoft have all placed a moratorium or gotten out of the game entirely in using facial recognition for law enforcement in United States. I don’t think the conclusion of this is that we should build a dystopian surveillance state in the US with facial recognition. But that’s one example where the the fact that the Chinese Communist Party is using AI in this very troubling way internally to surveil and monitor its citizens is leading to fears in Washington of falling behind China.

One of the conclusions that come from the book is I don’t think that’s actually true when you look more broadly, not just at facial recognition but across all of the different areas of AI. I don’t even think that’s true in just the space of competition in data. But that is a perception.

And I think another area that this comes up is in the reliability of AI systems as they’re deployed. We see already that many of the leading AI companies are taking shortcuts on safety to deploy their most advanced AI systems ahead of others, whether it’s Google’s Bard and Microsoft’s Sydney, the Bing Chat AI, or OpenAI’s ChatGPT-4. And it’s quite a troubling development, when we see these competitive dynamics starting to lead to something like a race to the bottom on safety. We need to find ways to try to turn those incentives around in the private sector. And there may be a role for government regulation to do that. But I think it’s also a concern between nation states. For example, in the military domain, we wouldn’t want to see the US and Chinese military deploy AI systems before they’re ready, and then risking accidents to the process.

Mecklin: It seemed to me from the kind of access you got when you were doing the book that you’re somebody that the US military trusts to look at a lot of this stuff, and I wanted to get your assessment: Where is the US military, in terms of getting up to speed on AI, using it in responsible ways, and avoiding the irresponsible ways it might be used? You really have gotten what seems to be unparalleled access.

Scharre: Yeah, I was, I’m incredibly grateful that the Defense Department gave me quite a bit of access in doing interviews for the book—in particular, the Joint AI Center, which has now been folded into the new Chief Digital and Artificial Intelligence Office (CDAO).

But at the time, the Joint AI Center gave me a tremendous amount of access to their experts and program managers who are leading different AI programs. And I think one of the things that stands out to me, is just a huge disconnect between the way that DoD leaders talk about AI and what the DoD is actually doing. Senior defense leaders talk a lot about artificial intelligence. If you listen to their statements, you would think that DoD is doing a lot, but when you actually look at what they’re doing, it’s quite minimal, and they’re very far behind the private sector.

When he was Secretary of Defense, Mark Esper testified before Congress that AI is his number one priority. When you look at the budget, the Defense Department’s spending about 1 percent of its budget on artificial intelligence. One could argue that’s too low or too high or about right, but no matter how you slice it, that’s not the the DoD’s number one priority; that’s not even close. So I think that’s important to realize. And what was interesting to me is that, among US national security leaders, there has been this intense fear of being shut out of the technology that’s really being developed in the private sector because of a disconnect in culture and values between AI scientists and the military, or at least the perceived disconnect.

And this was an outgrowth of Google’s decision not to continue work on Project Maven several years ago, after several Google employees published an open letter protesting working with the military. And then we saw other open letters from employees at Amazon and Microsoft as well.

And that led to this real panic among US national security leaders that they’d be locked out of this game-changing technology coming out of commercial sector.

And it hasn’t been the case. In fact, we’ve seen that major tech companies, including Google, are willing to work with the military. And we’ve seen a whole crop of new defense-focused AI startups that I talk about in the book that are specifically oriented to the defense space.

The slowness comes from within the Defense Department. It’s a slowly moving bureaucracy. It’s a lot of red tape. It’s a cumbersome acquisition system. That’s where the obstacles actually come to adoption for the US military.

Mecklin: And how that could be rectified is interesting to me, in that moving fast and breaking things may be great in Silicon Valley. But you don’t really want to do that with military systems.

Scharre: Certainly the mantra of “go fast and break things” is not how the DoD procurement system works. And that’s a huge cultural disconnect from from a technology standpoint.

One of the things that comes up time and again in the US defense community is the need to move faster and take risks. But it’s worth being conscious of what we are taking risks in. And I don’t think the solution is to take risk in fielding AI systems that are likely to have accidents; that’s problematic in many ways. Depending on the AI system, the consequences could be severe—including loss of life or, for the military mission, failure.

Additionally, one consequence of deploying systems before they’re ready is that they can lose warfighter trust, and then the operating person just stops using them. And we’ve seen this in the past, where most humans often-over trust AI systems and automation. When they lose trust, that can often be quite brutal, and it’s difficult to regain it. Humans will simply say, “Well, I don’t trust this thing. I’m not going to use it” and discard it. And that’s happened in the military space, as well.

So I think taking risks in working with non-traditional companies is a place where the DoD needs to take risk. Taking risk in moving faster to cut through some of the red tape in the acquisition system makes sense. Taking risk in “Is this project going to work out? Or maybe it won’t pan out?” That’s part of the nature of competing in emerging technology. Investors understand this; startups understand this. The DOD needs to adopt that mentality, too. But that’s not taking risk on the testing side. I think that that would be unwise.

Mecklin: Let’s back up a step or two here. In the overall international security realm, you talk about past industrial revolutions. And it seems to me those have all, in some ways, played roles in war—at times, causing wars. I mean, World War II can be seen in some ways as a fight for oil. And it seems possible to me that AI could in the same way incentivize war. My question is really about US strategy in dealing with China. Right now, you see US leaders trying to have high-level talks with Chinese leaders—at the same time that we’re actively trying to keep China from getting the kind of chips you need to do high-technology things. That seems to be a real disconnect in policies. How do you view that situation

Scharre: I think that the broader US strategy towards China is one that has inherent tensions in it. Washington has made a hard pivot away from its strategy of engagement for the last several decades towards acknowledging the reality: China’s a strategic competitor. And there are areas where the United States needs to work hard to maintain a leadership position in this competition with China in technological, economic, political, and military domains.

Certainly, having a military advantage in the western Pacific along China’s periphery is likely, in my view, to stabilize the region and make it less likely that China sees an opportunity for military aggression, whether that’s in the South China Sea, or against Taiwan, or the East China Sea, or elsewhere. But, of course, the US doesn’t want a war with China, and China is not going away.

And so finding ways to manage this competition is a high priority for US policymakers. It’s one of the reasons why they have sought high-level talks with China. And every time that we see high-level engagement between the US and Chinese leaders, there is I think, by many looking at this competition, a hope that we begin to see a floor on this downward spiral in US-China relations.

And I’m not sure that we’ve seen it yet. It’s possible. We certainly have yet to see a thaw. We could be hopeful that maybe that’s coming. But I think it’s a little bit unclear how bad things [could] continue to get between the US and China, as both countries really clearly view each other as strategic competitors.

Mecklin: In the news of late, there’s been a series of open letters from AI leaders saying, sort of, “the apocalypse is upon us, we need to act now,” or whatever. Your book deals with the dark side of AI, but it does not specifically deal with the artificial-general-intelligence-taking-over-the-world, Skynet kind of thing. Why don’t you talk about your view of what is the dark side that we have to grapple with right now in the AI revolution.

Scharre: The book talks a little bit towards the end about trends we’re seeing in AI and possibilities for the future, particularly given some of the growth that we’ve seen in very large and very capable models, like ChatGPT, or its successor, GPT-4. The way that the book talks about AI risk is very much grounded in practical problems today: facial recognition, the use of AI in warfare, disinformation, and other things that we can see. There’s a huge debate in the field, about where people should focus attention on AI risks—whether people should look at these near-term practical harms that we see happening, or longer-term harms that, if AI systems continue to improve, certainly could be a concern down the road. I don’t see these as unrelated phenomena. To me, they are stemming from the reality that we are building increasingly capable machines that we really do not understand.

No one understands—even the best AI expert—how these systems actually function. We know that they train on large datasets using lots of computing hardware, and they have massive neural networks. But to ask, “Why did ChatGPT say this thing?” No one knows. And we don’t have good ways to actually interrogate the systems to understand why they exhibit certain types of behavior.

One of the real troubling phenomena in the space is, as you see AI scientists scaling up the amount of data and computing hardware that they’re using to train the most capable models, we see massive gains in performance. Not necessarily commensurate gains in reliability, or safety, or controllability of the systems. And we have no good way to predict when we’ll see new capabilities emerging.

So, for example, large language models are chatty, they tend to make things up, or what’s sometimes called hallucinate. It’s possible that this gets better, through more fine tuning with reinforcement learning, or simply scaling up to larger datasets and computing power.

[But] it’s possible that this problem proves deeply intractable. We don’t know, and I think AI experts might have a guesstimate, but no one knows; we don’t have a good way to gather data on that. There are, we’ve seen, some evidence in a few instances of AI systems that are able to engage in strategic competition with humans, such Meta’s AI system Cicero, that achieved human level performance in the game Diplomacy. There’s at least some examples with GPT-4 of it appearing to engage in deception of human users; sort of anecdotal, maybe not well established, but at least that the possibility seems there. We don’t have a good sense of, as models continue to improve, is there a risk that they may be pursuing goals that are hidden to the user, and operate in ways that are deceptive to users, or try to manipulate users? No one knows. There’s no risk of chatGPT taking over the world. That’s not what people are concerned about.

But the trajectory we’re on is concerning, I think, in a variety of respects. And I’m sympathetic to those who say we need to kind of slow down, take a breath, and maybe try to focus on making the systems safer and more reliable and more understandable before we just keep making them more powerful.

Mecklin: Do you have a view on what kind of regulation or management through the government would actually work? To slow down the research and release of technology into society in a responsible way? I have a hard time even imagining what—what the hell would that be?

Scharre: It’s a tricky one; there’s a lot of debate about this topic. I do think that there are opportunities to regulate the frontier of AI development, by focusing on hardware. The most advanced systems are very hardware-intensive and use thousands of very specialized chips running for weeks or months at a time. And that’s a physical resource that you can control.

In the 20th century, nations were able to control the spread of nuclear weapons by focusing on physical inputs to nuclear weapons, the development of weapons-grade plutonium and uranium. I think there’s an analogy, that by focusing on the physical hardware that’s needed to train the most capable systems and controlling access to it and how it’s used, gets some purchase on the ability to manage the proliferation of those capable systems and then put in place regulatory regimes about how they should be tested or deployed. That probably looks like some pre-deployment testing of systems to look for possible harms, before they’re deployed. Some kind of independent red teams that can provide an outside view of some of these systems. And frankly, better tools to just analyze the systems—which we don’t actually have today—to try to assess their performance.

In the private sector, interestingly, one of the concerns about leading AI labs cooperating to potentially manage AI risk is that doing so may run afoul antitrust regulations, which is a fascinating problem. And that’s a place where I think some government involvement might be helpful to say, “We do want companies to prioritize safety, rather than deploying systems out into the world without even a clear sense of what the possible harms might be.”

Mecklin: I assume the US military has thought of incorporating AI in military command and control systems, battlefield management systems. There seems to be both a real incentive to do that, to be able to respond more quickly, to know more about what all the sensors are sensing, supposedly leading to better decision making—but with obvious dangers. What’s your view on what seems like an inevitable march toward AI use in battlefield management

Scharre: It’s been a longstanding desire of the US military to use AI for battlefield management, dating back at least to the last 15-20 years. And for the most part, the technology is just not yet ready. Even doing very simple things like having collaborative autonomy—having instead of one person directing a drone personally, having one person direct a group or team of drones that can coordinate amongst themselves—that’s been do-able, although to my knowledge has yet to be deployed. But the kind of AI situational awareness and battlefield management has really not just been possible.

It is possible that large language models change that, and I’ve seen some early attempts to do so. I think we’re only beginning to scratch the surface of what large language models can actually do; it’s possible that they begin to give this sort of broader contextual understanding that AI systems have failed at for decades and, by using large language models, are able to take disparate pieces of information and knit them together into some coherent whole.

So instead of having image classifiers, that look at sensor data in an area, and identify “this is an airfield, this is an aircraft, this is a truck, this is a tank,” [they would] then take that data and, at a higher level, synthesize it to try to gain some contextual understanding of what’s occurring. I think it’s possible that we see in the next few years, the early development of some AI battlefield management tools that help commanders get greater situational awareness and process all of this data that’s coming through, and that look maybe a little more sophisticated than just business process automation that relies on large language models. But it does depend quite a bit on how capable these large language models will prove to be, which is a huge unknown at this point.

Mecklin: I have a last question, and it’s sort of vague, but an important one: These kinds of developments in society can be extremely stressing and upsetting to the general population, right? This is just the scariest stuff that I’ve ever seen, people say; why can’t we just stop it all? So I’m going to ask you: Are you generally optimistic or pessimistic that America and the world can manage AI?

Scharre: I’m incredibly optimistic about the future. And I think it’s worth maybe taking a step back from the day-to-day headlines, which are always very negative, and acknowledging that this is the best time to be alive by any reasonable metric. Humans have better quality of life, lifespan, healthspan, than at any point in time in human history. A lot of that is due to technology that has enabled better health and longevity and standard of living. That technology also comes with risk. And it’s one that most of us will be acutely aware of, because the power of these technologies can often be quite dangerous if they’re used for ill—whether it’s nuclear weapons, or biological weapons, or perhaps AI if it continues to develop. So I think we want to be conscious of those risks and find ways to try to guard against them. But I think the long-term trend is quite positive.

The Bulletin elevates expert voices above the noise. But as an independent, nonprofit media organization, our operations depend on the support of readers like you. Help us continue to deliver quality journalism that holds leaders accountable. Your support of our work at any level is important. In return, we promise our coverage will be understandable, influential, vigilant, solution-oriented, and fair-minded. Together we can make a difference.

Keywords: AI, China, International Security, artificial intelligence, chatbots, global power, large language models
Topics: Artificial Intelligence, Military Applications of AI

John Mecklin is the editor-in-chief of the Bulletin of the Atomic Scientists. Previously, he was editor-in-chief of Miller-McCune... Read More

Copyright © 2023 Bulletin of the Atomic Scientists. All rights reserved. Terms of Use Privacy Policy

1307 East 60th Street, Chicago, IL 60637 | 773.702.6308

Introduction