Introduction

Arms control has traditionally been preoccupied with the need to keep up with the challenges raised by new developments in science and technology. The possible development of fully autonomous weapons systems (AWS) – colloquially known as “killer robots” – has raised new concerns that technological developments and political realities will outpace normative and regulatory change with potentially harmful consequences. While both proponents and skeptics of this weapon technology tend to agree that AWS do not yet exist, there are precursors to such systems and major powers like the United States, China, Russia, and South Korea are heavily investing in advanced robotics and artificial intelligence (AI) (Haner and Garcia 2019Haner, Justin, and Denise Garcia. 2019. “The Artificial Intelligence Arms Race: Trends and World Leaders in Autonomous Weapons Development.” Global Policy 10 (3): 331–337. doi:https://doi.org/10.1111/1758-5899.12713. [Crossref], [Web of Science ®], [Google Scholar]). For those interested in keeping the genie in the bottle, the time for a preemptive ban on AWS is therefore ripe. The Campaign to Stop Killer Robots (KRC), a global coalition of over 160 non-governmental organisations (NGOs) from 65 countries currently champions the norm against the development, production and use of fully autonomous weapons. The issue has also entered the United Nation’s (UN) arms control agenda, with the Convention on Certain Conventional Weapons (CCW) as the main institutional venue for discussing the emergence of AWS. Currently, 30 state parties to the CCW have joined the Campaign’s call for a ban (KRC 2020KRC. 2020. “Who Wants to Ban Killer Robots?”, accessed 10/04/2020. https://www.stopkillerrobots.org/wp-content/uploads/2020/04/KRC_ListEndorsersApr2020_Ltr.pdf. [Google Scholar]).

For this pro-ban coalition, a major impediment to norm building has been limited knowledge about the issue which was long deemed as too futuristic and exotic to warrant further action (Carpenter 2014Carpenter, Charli. 2014. “Lost” Causes: Agenda Vetting in Global Issue Networks and the Shaping of Human Security. Ithaca, N.Y: Cornell University Press. [Crossref], [Google Scholar], 103). Under these conditions, norm building is unlikely to succeed because knowledge and scientific evidence, as Garcia (2011Garcia, Denise. 2011. Disarmament Diplomacy and Human Security: Regimes, Norms and Moral Progress in International Relations. London: Routledge. [Crossref], [Google Scholar], 23) notes, are twin conditions for successful norm emergence. With AWS, there seems to be no documentable humanitarian harm as in most previous humanitarian disarmament campaigns, no egregious incident to cite. This constitutes a genuine challenge for proponents of a norm against AWS. How, then, can an issue that is as elusive and uncertain as AWS be rendered knowable and ultimately amenable to normative change?

This article starts from the premise that the emergence of anticipatory norms, which intervene in the temporal space between the identification of a problem and its emergence as determinate, hinges on forward-looking epistemic practices that produce and sustain knowledge about future governance objects and create a demand for preventive action. International organisations (IOs) seem ideally suited for this endeavour. While IOs routinely produce and disseminate knowledge about problems that lie in the present (Bueger 2015Bueger, Christian. 2015. “Making Things Known: Epistemic Practices, the United Nations, and the Translation of Piracy.” International Political Sociology 9 (1): 1–18. doi:https://doi.org/10.1111/ips.12073. [Crossref], [Web of Science ®], [Google Scholar]; Hülsse 2007Hülsse, Rainer. 2007. “Creating Demand for Global Governance: The Making of a Global Money-Laundering Problem.” Global Society 21 (2): 155–178. doi:https://doi.org/10.1080/13600820701201731. [Taylor & Francis Online], [Google Scholar]), they also strive to be “prognostic experts” by imagining and enacting essentially uncertain und incalculable futures (Berten and Kranke 2022Berten, John, and Matthias Kranke. 2022. “Anticipatory Global Governance: International Organisations and the Politics of the Future.” Global Society 36 (2): 155–169. [Taylor & Francis Online], [Google Scholar], 156). Yet, little is known about how IOs anticipate the future (for a notable exception in this journal see Chamlian 2016Chamlian, Lucie. 2016. “The Colonisation of the Future: Power, Knowledge and Preparedness in CSDP.” Global Society 30 (3): 391–411. doi:https://doi.org/10.1080/13600826.2016.1173020. [Taylor & Francis Online], [Web of Science ®], [Google Scholar]). To fill this gap, this paper develops a theoretically guided account, grounded in practice theory and science and technology studies (STS), of how the United Nations Institute for Disarmament Research (UNIDIR) produces knowledge about AWS and creates a demand for anticipatory norms.

While UNIDIR is certainly not the only actor involved in knowledge production about AWS, it occupies a distinct position given that it is funded by states to provide expertise on the issue. As an independent research institute within the UN framework, UNIDIR is also more likely to be seen as objective than advocacy groups like the KRC and, by extension, its expertise more likely to be trusted. Small and middle-sized states, in particular, routinely consult UNIDIR to form their national positions. Any yet, UNIDIR has not received nearly as much attention in the literature. This is unfortunate because knowledge, whatever the source, is never neutral or objective but deeply impregnated with values, interests and ideologies (Jasanoff 2004Jasanoff, Sheila. 2004. States of Knowledge: The Co-Production of Science and Social Order. London: Routledge. [Crossref], [Google Scholar]). Just as the need for more data and research has been mobilised to delay progress in the CCW, knowledge can be used strategically to legitimate certain political measures (e.g. regulation of or a ban on AWS). UNIDIR is no exception in that regard as it closely aligns with the UN disarmament agenda.

While clarifying the nature, severity and impact of an emerging technology like AWS is indeed challenging, the article argues that this is no mission impossible. First, knowledge transfer from other disciplines and experts can compensate for a lack of direct or first-hand experience of AWS. Second, analogical reasoning allows us to understand AWS through the lens of existing technologies in the civilian and military realm. Third, the limits of analogies can be overcome by thinking imaginatively about AWS, for example by playing through hypothetical scenarios.

So far, scholars have studied how deliberative norms about the appropriate use of force – that is, those emerging from explicit normative-political reasoning in forums such as the CCW – are increasingly replaced by procedural norms related to the development and use of AWS (Bode and Huelss 2018Bode, Ingvild, and Hendrik Huelss. 2018. “Autonomous Weapons Systems and Changing Norms in International Relations.” Review of International Studies 44 (3): 393–413. doi:https://doi.org/10.1017/S0260210517000614. [Crossref], [Web of Science ®], [Google Scholar]), as well as by notions of normality derived from a technologically mediated reality (Huelss 2020Huelss, Hendrik. 2020. “Norms Are What Machines Make of Them: Autonomous Weapons Systems and the Normative Implications of Human-Machine Interactions.” International Political Sociology, doi:https://doi.org/10.1093/ips/olz023. [Crossref], [Google Scholar]). Others have analysed the norm-setting efforts by the KRC (Rosert and Sauer 2020Rosert, Elvira, and Frank Sauer. 2020. “How (not) to Stop the Killer Robots: A Comparative Analysis of Humanitarian Disarmament Campaign Strategies.” Contemporary Security Policy, 1–26. doi:https://doi.org/10.1080/13523260.2020.1771508. [Google Scholar]), or more generally discussed the need for “preventive security governance” to address future technological advances (Garcia 2016Garcia, Denise. 2016. “Future Arms, Technologies, and International Law: Preventive Security Governance.” European Journal of International Security 1 (1): 94–111. doi:https://doi.org/10.1017/eis.2015.7. [Crossref], [Web of Science ®], [Google Scholar]). The present paper makes both an empirical and a theoretical contribution to this evolving literature on AWS and norms. Empirically, it offers a comprehensive study of an under-researched case, namely the formative role of UNIDIR in current norm-building processes around AWS. Theory-wise, the paper offers a framework for understanding how preventive regulation may emerge by elucidating the epistemic bases of anticipatory norm-building processes. My findings suggest the following practices to be particularly conducive to anticipatory norm building: efforts to de-futurise the issue; creative ways of imagining the future through which novel concerns raised by the use of new technologies can be brought to the fore; and the articulation of a positive vision (such as the idea of meaningful human control) to give states a shared sense of direction towards a common goal.

The article proceeds as follows. The first section introduces the puzzling phenomenon of anticipatory norm-building and presents the conceptual framework to examine how future problems can be made knowable and governable. Following insights from STS, I distinguish three kinds of practices through which “objects of knowledge” like AWS are brought into being: “assembling”, “translating” and “representing” (Bueger 2015Bueger, Christian. 2015. “Making Things Known: Epistemic Practices, the United Nations, and the Translation of Piracy.” International Political Sociology 9 (1): 1–18. doi:https://doi.org/10.1111/ips.12073. [Crossref], [Web of Science ®], [Google Scholar]; Cetina and Karin 1981Cetina, Knorr, and D. Karin. 1981. The Manufacture of Knowledge: An Essay on the Constructivist and Contextual Nature of Science. Oxford: Pergamon Press. [Google Scholar]; Law 2009Law, John. 2009. “Actor Network Theory and Material-Semiotics.” In The New Blackwell Companion to Social Theory, edited by Bryan S. Turner, 141–158. Oxford: Wiley-Blackwell. [Crossref], [Google Scholar]). The second section discusses the status of a prospective norm against AWS in the light of existing weapons systems with automatic/autonomous features. The subsequent section locates UNIDIR within the framework of the UN system and the ongoing discussions about AWS in the UN-CCW, before it analyses how the Institute generates knowledge about AWS. It does so, based on extensive documentary analysis and two remote interviews with representatives of the organisation – those involved in knowledge production about AWS. It then goes on to assess UNIDIR’s relevance in the discourse. The final section concludes with a discussion of the main findings and its theoretical implications.

The epistemic foundations of anticipatory norm-building

Anticipatory norms

Most arms control-related norms have emerged as reactions to the devastating consequences imposed by new or existing weapon technologies such as nuclear weapons (Price 1998Price, Richard. 1998. “Reversing the Gun Sights: Transnational Civil Society Targets Landmines.” International Organization 52 (3): 613–644. [Crossref], [Web of Science ®], [Google Scholar]; Rosert et al. 2013Rosert, Elvira, Una Becker-Jakob, Giorgi Franceschini, and Annette Schaper. 2013. “Arms Control Norms and Technology.” In Norm Dynamics in Multilateral Arms Control: Interests, Conflicts, and Justice, edited by Carmen Wunderlich, and Harald Müller, 109–140. Athens: University of Georgia Press. [Google Scholar]; Tannenwald 2007Tannenwald, Nina. 2007. The Nuclear Taboo: The United States and the Non-Use of Nuclear Weapons Since 1945. Cambridge: Cambridge University Press. [Crossref], [Google Scholar]) and can thus be seen as efforts to regulate a relatively well-known problem. One remarkable exception is the preventive ban on blinding laser weapons, adopted at the First Review Conference of the CCW in 1995. The attempts to preemptively ban AWS are testament to another preventive or anticipatory norm in-the-making. What makes anticipatory norms special is their timing. They intervene in the temporal space between the identification of a problem and its emergence as determinate – that is, in advance of a problem taking full shape and before there is definitive evidence or proof of its harmful consequences. As such, anticipatory norms have at least two specific features that set them apart from reactive norms: their preventive thrust and their epistemic grounding. Of course, these distinctions are not always straightforward, particularly if timing is the key factor. As time goes by and potential problems move into the here and now, the status of a norm may gradually change from anticipatory to reactive. Therefore, it seems indeed more apt to think of a continuum ranging from norms with more to less anticipatory grounding.

In the IR literature norms are commonly defined as “collective expectations about appropriate behavior” (Katzenstein 1996Katzenstein, Peter J. ed. 1996. The Culture of National Security: Norms and Identity in World Politics. New York: Columbia University Press. [Google Scholar], 5). They spell out who should do what based on widely shared values or notions of appropriateness. As to anticipatory norms, the kind of behaviour that their advocates suggest takes the form of preventive or preemptive action. The very meaning of the term anticipation is “[p]rior action that meets beforehand, provides for, or precludes the action”.¹1 Oxford English Dictionary, http://www.oed.com/view/Entry/8557?redirectedFrom=anticipation#eid.View all notes Of course, one might object that the regulatory scope of any norm extends into the future with the aim of preventing further harm. And yet, reactive norms work post-facto. They may well have anticipatory elements, but in the sense of preventing a well-known problem from recurring. Those actors who promote them already have (directly or indirectly) practical experience with a particular problem. For example, a “nuclear taboo” emerged only after the destructive power of these weapons became manifestly clear. This brings us to the second feature of anticipatory norms: their epistemic foundation. Norms can have anticipatory elements in the other sense of the term, understood not only as a way of acting, but also of knowing beforehand: “The action of representing to oneself or realizing a thing before it occurs; apprehension beforehand, preconception.”²2 Oxford English Dictionary, http://www.oed.com/view/Entry/8557?redirectedFrom=anticipation#eid.View all notes If their regulative thrust is to prevent future harm to widely shared values or principles, this already implies that anticipatory norms carry with them certain ideas of how the future might unfold. Thus, while direct evidence of damage, either physically or otherwise, rarely exists, that does not mean that (future) issues are problematised as “unknown un-knowns’ (Aradau 2016Aradau, Claudia. 2016. “Risk, (In)Security and International Politics.” In Routledge Handbook of Risk Studies, edited by Adam Burgess, Alberto Alemanno, and Jens O. Zinn, 290–298. London: Routledge. [Google Scholar], 294). On the contrary, proponents of anticipatory norms usually have a clear sense that a problem exists – for example, that a future weapon technology will violate humanitarian norms – and invest a lot of time and energy in convincing other actors that this is actually the case. So, how can we know about future problems?

Practice theory offers a way forward in arguing that problems do not exist prior to specific discourses and practices. It is through performances that epistemic objects such as “climate” (Allan 2017Allan, Bentley B. 2017. “Producing the Climate: States, Scientists, and the Constitution of Global Governance Objects.” International Organization 71 (1): 131–162. doi:https://doi.org/10.1017/S0020818316000321. [Crossref], [Web of Science ®], [Google Scholar]), “migration” (Robinson 2018Robinson, Corey. 2018. “Making Migration Knowable and Governable: Benchmarking Practices as Technologies of Global Migration Governance.” International Political Sociology 12 (4): 418–437. doi:https://doi.org/10.1093/ips/oly020. [Crossref], [Web of Science ®], [Google Scholar]), “piracy” (Bueger 2015Bueger, Christian. 2015. “Making Things Known: Epistemic Practices, the United Nations, and the Translation of Piracy.” International Political Sociology 9 (1): 1–18. doi:https://doi.org/10.1111/ips.12073. [Crossref], [Web of Science ®], [Google Scholar]), or – as in the empirical case discussed here – “AWS” are brought into being, problematised and presented as demanding particular forms of intervention. The STS tradition of practice theory, in particular, directs our attention to epistemic practices to explain how scientific knowledge acquires the status of “truth” and “certainty”.

Epistemic practices

Following the STS tradition of practice theory as elaborated by Knorr Cetina and others, the paper argues that epistemic practices work in three ways: they “assemble”, “translate” and “represent” (knowledge about) objects or phenomena (Cetina and Karin 1981Cetina, Knorr, and D. Karin. 1981. The Manufacture of Knowledge: An Essay on the Constructivist and Contextual Nature of Science. Oxford: Pergamon Press. [Google Scholar]; Bueger 2015Bueger, Christian. 2015. “Making Things Known: Epistemic Practices, the United Nations, and the Translation of Piracy.” International Political Sociology 9 (1): 1–18. doi:https://doi.org/10.1111/ips.12073. [Crossref], [Web of Science ®], [Google Scholar]; Law 2009Law, John. 2009. “Actor Network Theory and Material-Semiotics.” In The New Blackwell Companion to Social Theory, edited by Bryan S. Turner, 141–158. Oxford: Wiley-Blackwell. [Crossref], [Google Scholar]). These practices serve a stabilising function. They give inherently open and malleable objects a more palpable and stable presence, even though attempts at “preserving, conserving, maintaining, and stabilizing” may never be complete (Bueger 2015Bueger, Christian. 2015. “Making Things Known: Epistemic Practices, the United Nations, and the Translation of Piracy.” International Political Sociology 9 (1): 1–18. doi:https://doi.org/10.1111/ips.12073. [Crossref], [Web of Science ®], [Google Scholar], 4). While assembling and translation refer to the form of knowledge – the design or presentation of information – representation is about its content. Epistemic forms play an important role in the construction of futures. As Hodžić (2013Hodžić, Saida. 2013. “Ascertaining Deadly Harms: Aesthetics and Politics of Global Evidence.” Cultural Anthropology 28 (1): 86–109. doi:https://doi.org/10.1111/j.1548-1360.2012.01174.x. [Crossref], [Google Scholar]) reminds us, for knowledge to be seen as certain what matters is not only what is presented, but also how (the form). She argues that certain modes of representing knowledge rather than others are regarded as more authoritative, for example those that have underwent smoothing and the veiling of doubt in scientific publications.

First, epistemic practices assemble by “making generalizable claims” about potentially contested objects (Bueger 2015Bueger, Christian. 2015. “Making Things Known: Epistemic Practices, the United Nations, and the Translation of Piracy.” International Political Sociology 9 (1): 1–18. doi:https://doi.org/10.1111/ips.12073. [Crossref], [Web of Science ®], [Google Scholar], 6). They bring together various types of knowledge and construct a larger picture out of seemingly disparate details. This is not necessarily a neutral practice of listing what is out there, but may involve the selection and omission of certain voices, privileging some ways of understanding an issue (Weaver and Leander 2019Weaver, Ole, and Anna Leander. 2019. Assembling Exclusive Expertise: Knowledge, Ignorance and Conflict Resolution in the Global South. Abingdon: Routledge. [Google Scholar]). For the following analysis, the process of “data gathering” is therefore key. How and from what sources does UNIDIR collect information about AWS?

Second, translation is referred to as the process through which complex social or scientific phenomena are converted into a concrete and tangible form such as reports, benchmarks or concepts (Robinson 2018Robinson, Corey. 2018. “Making Migration Knowable and Governable: Benchmarking Practices as Technologies of Global Migration Governance.” International Political Sociology 12 (4): 418–437. doi:https://doi.org/10.1093/ips/oly020. [Crossref], [Web of Science ®], [Google Scholar], 426; Callon 1981Callon, Michel. 1981. “Struggles and Negotiations to Define What is Problematic and What is Not.” In The Social Process of Scientific Investigation, edited by Karin D. Knorr, Roger Krohn, and Richard Whitley, 197–219. Dordrecht: Springer Netherlands. [Google Scholar]). It involves the formidable task of mediating between the known (object) and the knower (subject) (Bueger 2015Bueger, Christian. 2015. “Making Things Known: Epistemic Practices, the United Nations, and the Translation of Piracy.” International Political Sociology 9 (1): 1–18. doi:https://doi.org/10.1111/ips.12073. [Crossref], [Web of Science ®], [Google Scholar], 7). With future object, the additional challenge is to literally make present “what has not happened and may never happen” (Massumi 2007Massumi, Brian. 2007. “Potential Politics and the Primacy of Preemption.” Theory & Event 10 (2), doi:https://doi.org/10.1353/tae.2007.0066. [Crossref], [Google Scholar]: n.p.). According to Anderson (2010Anderson, Ben. 2010. “Preemption, Precaution, Preparedness: Anticipatory Action and Future Geographies.” Progress in Human Geography 34 (6): 777–798. doi:https://doi.org/10.1177/0309132510362600. [Crossref], [Web of Science ®], [Google Scholar]), this passage between ontological modes involves processes of materialisation, articulation and embodiment. The future, the “abstract and non-existent”, can be translated into something “visible” and “intelligible” (Chamlian 2016Chamlian, Lucie. 2016. “The Colonisation of the Future: Power, Knowledge and Preparedness in CSDP.” Global Society 30 (3): 391–411. doi:https://doi.org/10.1080/13600826.2016.1173020. [Taylor & Francis Online], [Web of Science ®], [Google Scholar], 405), for example through reports, oral briefings, or collective practices like simulations and webinars. In the following, I will analyse how UNIDIR communicates and disseminates knowledge to its main audience, the state parties to the CCW. This includes not only written, oral, graphic and embodied ways of rendering the Institute’s research findings legible, but also practices of classification through which the organisation seeks to organise and repackage knowledge about AWS.

Finally, epistemic practices represent that “which is absent, and hence needs to be introduced into the present situation, to be made (con)textually visible and available” (Pels 2000Pels, Dick. 2000. The Intellectual as Stranger. Studies in Spokespersonship. London: Routledge. [Google Scholar], 1). They, too, have a stabilising force in the sense of filling the somewhat open space between the future and the present. Representations of the future can be distinguished based on whether they articulate robust visions or speculative/creative visions. Robust visions are characterised by a probabilistic or deterministic way of reasoning about the future (Berenskoetter 2011Berenskoetter, Felix. 2011. “Reclaiming the Vision Thing: Constructivists as Students of the Future1.” International Studies Quarterly 55 (3): 647–668. doi:https://doi.org/10.1111/j.1468-2478.2011.00669.x. [Crossref], [Web of Science ®], [Google Scholar]; Berten and Kranke 2022Berten, John, and Matthias Kranke. 2022. “Anticipatory Global Governance: International Organisations and the Politics of the Future.” Global Society 36 (2): 155–169. [Taylor & Francis Online], [Google Scholar], 160). They allow for predictive claims based on induction from past or present conditions. Presenting future technological trajectories as if they belong to the realm of the probable, inevitable and certain can instil a sense of urgency in anticipatory norm-building processes. Creative visions, in contrast, are “looser, more tentative and open to alternative possibilities” (Johnson 2015Johnson, Deborah G. 2015. “Technology with No Human Responsibility?” Journal of Business Ethics 127 (4): 707–715. [Crossref], [Web of Science ®], [Google Scholar], 5). Even if they may never fully realise, creative visions may become a source of inspiration for breaking with the prevailing order of things and thus open up the possibility of (normative) change (Berenskoetter 2011Berenskoetter, Felix. 2011. “Reclaiming the Vision Thing: Constructivists as Students of the Future1.” International Studies Quarterly 55 (3): 647–668. doi:https://doi.org/10.1111/j.1468-2478.2011.00669.x. [Crossref], [Web of Science ®], [Google Scholar], 660). By encouraging imaginative thinking, creative visions also allow to explore novel concerns raised by a new technology. Robust and creative representations are not mutually exclusive, but can be used in a complementary fashion. Whereas creative visions articulate broad aspirations that are intended to move people to action, “robust visions render big-picture narratives more credible” by anchoring them in real-world conditions (Berten and Kranke 2022Berten, John, and Matthias Kranke. 2022. “Anticipatory Global Governance: International Organisations and the Politics of the Future.” Global Society 36 (2): 155–169. [Taylor & Francis Online], [Google Scholar], 160).

A further distinction relates to whether future objects are represented as desirable or not (Berten and Kranke 2022Berten, John, and Matthias Kranke. 2022. “Anticipatory Global Governance: International Organisations and the Politics of the Future.” Global Society 36 (2): 155–169. [Taylor & Francis Online], [Google Scholar], 156). With emerging technologies such as geo-engineering or AWS, people are likely to disagree over their risks and benefits. How well these risks and benefits are articulated can have a huge impact on whether or not such technologies are seen as politically, morally and legally acceptable. With regard to anticipatory norm-building, representations of the future can be particularly powerful if they formulate clear and unambiguous visions in terms of “good” or “bad”. Presenting the future in such a way, as either desirable/comforting or undesirable/threatening, suggests that states ought to take action in the present to prevent an unwanted and enact a desirable future.

UNIDIR’s reports offer a good entry point as to how AWS are represented because they have by far the widest outreach. Since the launch of the project in 2013 UNIDIR has produced a total of 16 reports. Based on a qualitative content analysis, I have looked for elements that fit into either the categories robust/creative and desirable/undesirable. Statements are coded as “robust” if they base their projections on past or present conditions and/or articulate the future in terms of the inevitable, necessary, and probable. The qualifier “creative”, in turn, is reserved for statements that are somehow detached from reality and/or stress the contingency of the future. I have also coded the evaluative slant of UNIDIR’s claims about AWS (good/bad, desirable/undesirable), the degree to which doubt is expressed in such assessments and the normative grounds upon which these claims are based.³3 For example, objections related to AWS can be based on international humanitarian law (IHL) or moral principles that are not (yet) codified in law.View all notes To put my findings into perspective, I also draw on insights from a simulation run by the Institute in 2018, written statements produced for the CCW meetings, and two semi-structured interviews that I have conducted with researchers from UNIDIR’s project on AWS.

Are AWS weapons of the future?

According to a recent UN report, the world might have witnessed the first use of a fully autonomous weapon – the Turkish-made Kargu-2 drone or loitering munition – in Libya (Kallenborn 2021Kallenborn, Zachary. 2021. “Was a Flying Killer Robot Used in Libya? Quite Possibly.” Bulletin of the Atomic Scientists, May 20. https://thebulletin.org/2021/05/was-a-flying-killer-robot-used-in-libya-quite-possibly/. [Google Scholar]). The instant media attention that followed has surprised many observers who noted that weapons with automated or autonomous features have been around for years, including other loitering weapons such as the Israeli Harpy drone, certain air defense systems and robotic sentry guns (Bode and Watts 2021Bode, Ingvild, and Tom Watts. 2021. “Meaning-less Human Control: Lessons from Air Defence Systems on Meaningful Human Control for the Debate on AWS.” Centre for War Studies, University of Southern Denmark & Drone Wars UK. [Google Scholar]; Boulanin and Verbruggen 2017Boulanin, Vincent, and Maaike Verbruggen. 2017. Mapping the Development of Autonomy in Weapon Systems. Solna: SIPRI. [Google Scholar]). What is the status of a prospective norm against AWS if precursors to such systems have already been fielded? Can it still be considered anticipatory? Despite there being evidence of systems that somehow liken AWS, discussions under the auspices of the CCW have proceeded under the assumption “that such systems do not currently exist” (CCW 2015CCW. 2015. “Report of the 2015 Informal Meeting of Experts on Lethal Autonomous Weapons Systems (LAWS). 12–13 November.” https://undocs.org/pdf?symbol = en/ccw/msp/2015/3. [Google Scholar]). This seems to suggest that today’s weapons systems fall below the threshold of what is considered “truly” autonomous. Where exactly one locates a norm against AWS on a continuum between anticipatory and reactive is therefore largely a definitional matter.

Autonomy has been approached from quite different angles. From a functionalist point of view, a system is considered autonomous if “it can perform some task or function on its own” (Boulanin and Verbruggen 2017Boulanin, Vincent, and Maaike Verbruggen. 2017. Mapping the Development of Autonomy in Weapon Systems. Solna: SIPRI. [Google Scholar], 5). What is at stake in the debate about AWS is autonomy over certain critical functions: selecting and engaging targets without human intervention. AWS, then, are defined as “systems [that] can select and engage targets without further human intervention” (Heyns 2016Heyns, Christof. 2016. “Autonomous Weapons Systems: Living a Dignified Life and Dying a Dignified Death.” In Autonomous Weapons Systems: Law, Ethics, Policy, edited by Nehal Bhuta, Susanne Beck, and Claus Kreβ, 3–19. Cambridge: Cambridge University Press. [Google Scholar], 4). This definition would exclude most existing systems because they are assigned targets in advance by a human operator (guided missiles) (Boulanin and Verbruggen 2017Boulanin, Vincent, and Maaike Verbruggen. 2017. Mapping the Development of Autonomy in Weapon Systems. Solna: SIPRI. [Google Scholar], 47) or require human approval before initiating an attack (air defense systems) (Bode and Watts 2021Bode, Ingvild, and Tom Watts. 2021. “Meaning-less Human Control: Lessons from Air Defence Systems on Meaningful Human Control for the Debate on AWS.” Centre for War Studies, University of Southern Denmark & Drone Wars UK. [Google Scholar], 27).

Some draw a line between automated weapons systems, which have been in use for decades, and autonomous ones. Here, the key distinguishing feature is whether a weapon system “merely” reacts to triggers or independently determines a specific course of action (that is to strike or not a specific target) based on its sensory input. On the automatic end of the spectrum, we have systems that “mechanically respond to sensory input and step through predefined procedures, and whose functioning cannot accommodate uncertainties in the operating environment” (Boulanin and Verbruggen 2017Boulanin, Vincent, and Maaike Verbruggen. 2017. Mapping the Development of Autonomy in Weapon Systems. Solna: SIPRI. [Google Scholar], 6). A landmine, for example, detonates automatically if the weight exerted on the mine exceeds a certain threshold. An AWS, in contrast, does not merely react to its sensory input by following a predetermined sequence of actions, but would be able to further process the information received by its sensors (e.g. making probabilistic calculations about whether the trigger was due to a child or a tank, if it features machine learning capabilities) and, based on these calculations, “decide” how to act (Crootof 2015Crootof, Rebecca. 2015. “The Killer Robots Are Here: Legal and Policy Implications.” Cardozo Law Review 36 (4): 1837–1915. [Google Scholar], 1855). Most of today’s systems lack these abilities.

Others argue that it is less the intrinsic features of any weapon that makes it truly autonomous, but the way it is used (Horowitz 2016Horowitz, Michael C. 2016. “Why Words Matter: The Real World Consequences of Defining Autonomous Weapons Systems.” Temple International and Comparative Law Journal 30 (1): 85–98. [Google Scholar], 93). The reasons why many of the existing systems are likely to be exempt from a prohibitionary norm is that human agents have set narrow parameters of use, including limits on target type or where and when a weapon is used (for example, the SGR-A1 sentry robot has been developed for potential deployment in the Demilitarised Zone where no civilian presence is expected). This implies that those AWS that most trouble the international community have yet to emerge (with the likely exception of loitering weapons such as the Kargu-2 drone), making current norm setting efforts largely anticipatory. However, the more we see cases like the Kargu-2 incident, the further we may end up at the reactive end of the norm spectrum.

UNIDIR: making autonomous weapons known

Broome and Seabrooke (2012Broome, André, and Leonard Seabrooke. 2012. “Seeing Like an International Organisation.” New Political Economy 17 (1): 1–16. doi:https://doi.org/10.1080/13563467.2011.569019. [Taylor & Francis Online], [Web of Science ®], [Google Scholar], 3) argue that IOs make sense of their social environment primarily through their analytic institutions, defined as “the specialist units, departments, committees, adjudicatory bodies and others housed by or linked to IOs that develop the cognitive framework for understanding and solving policy problems”. UNIDIR is an emblematic analytic institution within the framework of the UN. Established by the General Assembly Resolution A/RES/34/83M in 1979, it is mandated to assist the international community with independent research on disarmament and international security. It works in close relationship with other UN bodies such the United Nations Office for Disarmament Affairs (UNODA) which frequently consults the Institute on drafts or commissions studies. As an autonomous research institute, however, the Institute enjoys a certain degree of independence from the larger UN system, which is primarily secured through the mixed nature of its funding. While it receives a small subvention from the regular UN budget, UNIDIR is mainly funded by project-related contributions from member States and grants from public and private foundations.

That said, UNIDIR’s research is geared to the needs and principles of the UN and its member states. In this sense, the Institute is not a completely autonomous entity, but closely aligns with the UN disarmament agenda, as one of my interview partners confirmed:

… we are not neutral because the United Nations is not a neutral organisation. You read the charter and its principles include the reduction of armaments to the least diversion from social and economic development. That’s not a neutral statement. … We do believe that everyone deserves security but there are other ways to have security than through, quantitatively, more weapons or improved weapon systems.⁴4 Interview, 2 July 2021.View all notes

Given this strong normative alignment, it is not surprising that UNIDIR’s uptake of the AWS issue has closely followed that of other UN bodies and its member states. Within the UN framework, the issue got an important push in 2013 when the former UN Special Rapporteur on extrajudicial, summary or arbitrary executions Christof Heyns brought the topic of “lethal autonomous robotics” to the attention of the Human Rights Council where, for the first time, governments had the opportunity to discuss the issue in a multilateral setting (Heyns 2013Heyns, Christof. 2013. “Report of the Special Rapporteur on Extrajudicial, Summary or Arbitrary Executions.” A/HRC/23/47. 9 April. https://daccess-ods.un.org/TMP/2689446.50888443.html. [Google Scholar]). And yet, the staging in the Human Rights Council promised only limited outreach beyond the human rights context. In particular, the wider arms control community has been largely unaware of the topic. With close links to Christoph Heyns and the arms control community, UNIDIR was well positioned to link both rather siloed networks. Shortly after the Heyns report was released, the Institute organised a briefing for the ambassadors of the Conference on Disarmament with Heyns in Geneva, which “blew things up in a big way because people were really shocked”.⁵5 Interview, 2 July 2021.View all notes The issue was subsequently taken up at the November Meeting of the state parties to the CCW in Geneva where state delegates agreed to place the issue on their agenda in May 2014. The increased attention that followed meant not only a heightened demand for expertise on AWS but also a growing readiness to fund work on this issue. UNIDIR heeded that call and launched the project “The Weaponization of Increasingly Autonomous Technologies” in 2013. The project was intended to advance the nascent multilateral discussion by delivering “insights and conceptual frameworks that will enable policy-makers to better think, discuss and make informed decisions about autonomy in weapon systems” (UNIDIR 2014aUNIDIR. 2014a. Framing Discussions on the Weaponization of Increasingly Autonomous Technologies. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/files/publications/pdfs/framing-discussions-on-the-weaponization-of-increasingly-autonomous-technologies-en-606.pdf. [Google Scholar], 2).

Since 2014, state parties to the CCW have held three informal meetings of experts on AWS. After little initial progress, they agreed to formalise deliberations on AWS by creating a Group of Governmental Experts (GGE), which has held its first session in November 2017. Despite all the criticism of the slow pace at which the CCW discussions are proceeding, states have experienced “a steep learning curve since 2014” (Rosert and Sauer 2020Rosert, Elvira, and Frank Sauer. 2020. “How (not) to Stop the Killer Robots: A Comparative Analysis of Humanitarian Disarmament Campaign Strategies.” Contemporary Security Policy, 1–26. doi:https://doi.org/10.1080/13523260.2020.1771508. [Google Scholar], 16; Renic 2019Renic, Neil C. 2019. “Death of Efforts to Regulate Autonomous Weapons Has Been Greatly Exaggerated.” Bulletin of the Atomic Scientists, December 18. https://thebulletin.org/2019/12/death-of-efforts-to-regulate-autonomous-weapons-has-been-greatly-exaggerated/. [Google Scholar]). This is not only testament to the role of seasoned campaigners such as the KRC, but also to the efforts of knowledge-hubs like UNIDIR which have gone a long way to educate decision-makers about the nature and consequences of increasing autonomy in weapons systems. Small and middle-power states, in particular, have a high demand and appreciation for UNIDIR’s expertise.⁶6 Interview, 23 May 2021.View all notes Lacking the kind of “in-house expertise” that makes states like, say, the United States quasi self-sufficient in terms of knowledge production, they have routinely asked UNIDIR for input and background information and keep funding its research on AWS. How, then, does UNIDIR render AWS knowable and governable?

Assembling

A common observation running through my interviews is that the issue of AWS does not lend itself to traditional humanitarian disarmament approaches. Given the early stages of research and development and the limited or simply unknown (prototype) uses of AWS, gaining first-hand knowledge of these systems (e.g. by studying operational practices) has been no option (UNIDIR 2020UNIDIR. 2020. Swarm Robotics: Technical and Operational Overview of the Next Generation of Autonomous Systems. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/swarm-robotics-technical-and-operational-overview-next-generation-autonomous-systems. [Google Scholar], 7). Instead, UNIDIR produces a type of authoritative knowledge primarily based on literature reviews, multi-disciplinary expert group meetings and conferences, focus group discussions, and interviews with stakeholders from a range of fields and disciplines, including human rights, IHL, ethics, artificial intelligence, defense, military strategy, robotics, computer science and engineering. Even if it lacks the scientific credentials of first-hand observations, this way of knowing is legitimised by reference to other authoritative sources, e.g. experts in the field. To the extent that it draws on diverse viewpoints and interpretations of AWS, it commands a high degree of representativeness. It suggests a consensual knowledge base and a sense of objectivity, which is particularly important in the consensus-based negotiations within the CCW.

This multi-disciplinary approach also strongly encourages out-of-the box thinking. For example, UNIDIR was careful not (only) to involve the usual suspects when setting up the series of expert meetings, but also.

people who are not the normal experts that we think of in our very close siloed community. I want to talk to people who have worked on autonomous cars and I want to talk to people who are thinking about responsibility in insurance. Because they’re crunching numbers about probability and what are the costs and where should responsibility lie when things go wrong, they’re light years ahead of us in thinking about how to distribute responsibility in increasingly autonomous systems.⁷7 Interview, 2 July 2021.View all notes

As we can infer from this statement, the meetings were intended to provide new insights by coordinating and assembling knowledge from neighbouring or related fields, combining it (in the sense of letting the different experts from otherwise “siloed” communities speak to each other), and (re-)considering it in a new context: autonomy in weapons systems.

Translation

UNIDIR plays an important role as knowledge broker or mediator by assembling existing knowledge, transferring it to a very specific case (AWS), and translating the results for UNIDIR’s main audiences: the parties to the CCW. They stand at the receiving end of the translation process, which ultimately seeks to make the issue legible for “non-technical audiences” (UNIDIR 2018UNIDIR. 2018. The Weaponization of Increasingly Autonomous Technologies: Artificial Intelligence. Geneva: United Nations Institute for Disarmament Research. http://www.unidir.org/files/publications/pdfs/the-weaponization-of-increasingly-autonomous-technologies-artificial-intelligence-en-700.pdf. [Google Scholar], 1). In this sense, translation can be understood in the literal sense: to convert or render the complex and often technical details of the expert discussion into a language and format that are comprehensible to less tech-savvy policy-makers. UNIDIR’s observation reports are the most visible outcome of this process; they are the public-facing side of the Institute’s research activities. Hard copies of these reports are distributed at the GGE meetings and UNIDIR regularly organises events alongside the GGE deliberations, where the studies’ main findings are disseminated and discussed. In collaboration with UNODA, the Institute has also hosted a series of webinars, which explored the technological, military and legal aspects of AWS.

Through its various outreach activities, UNIDIR seeks to classify and organise knowledge about AWS, most notably by reframing existing definitions and characterisations of AWS. In departure from categorical distinctions of AWS (e.g. in terms of automatic/semi-automonous/autonomous), UNIDIR suggests to reframe the discussion in terms of autonomy in weapons systems (Vignard 2016Vignard, Kerstin. 2016. “Statement of the UN Institute for Disarmament Research at the CCW Informal Meeting of Experts on Lethal Autonomous Weapon Systems.” 12 April 2016. Accessed 07/16/2021. https://www.unidir.org/files/medias/pdfs/unidir-s-statement-to-the-ccw-informal-meeting-of-experts-on-lethal-autonomous-weapon-systems-eng-0-648.pdf. [Crossref], [Google Scholar]): focusing on the “the key/critical features or tasks” that are considered problematic when autonomy is applied to them (UNIDIR 2017UNIDIR. 2017. The Weaponization of Increasingly Autonomous Technologies: Concerns, Characteristics and Definitional Approaches. Geneva: United Nations Institute for Disarmament Research. https://www.unidir.org/files/publications/pdfs/the-weaponization-of-increasingly-autonomous-technologies-concerns-characteristics-and-definitional-approaches-en-689.pdf. [Google Scholar], 21), particularly the selection and engagement of targets, and identifying further variables that make up autonomy (for example, communication links to a human operator or the operating environment). Such an approach is agnostic about the degree of technological sophistication of a weapons system and therefore evades the difficult question of how to define what is not yet in use; it is accessible to a broad range of governments and publics regardless of their degree of technical knowledge (UNIDIR 2017UNIDIR. 2017. The Weaponization of Increasingly Autonomous Technologies: Concerns, Characteristics and Definitional Approaches. Geneva: United Nations Institute for Disarmament Research. https://www.unidir.org/files/publications/pdfs/the-weaponization-of-increasingly-autonomous-technologies-concerns-characteristics-and-definitional-approaches-en-689.pdf. [Google Scholar], 20); and it recognises that autonomy is a spectrum and that incremental changes in some aspect or function of a system might lead to autonomous weapons without ever crossing a clear, bright line (UNIDIR 2017UNIDIR. 2017. The Weaponization of Increasingly Autonomous Technologies: Concerns, Characteristics and Definitional Approaches. Geneva: United Nations Institute for Disarmament Research. https://www.unidir.org/files/publications/pdfs/the-weaponization-of-increasingly-autonomous-technologies-concerns-characteristics-and-definitional-approaches-en-689.pdf. [Google Scholar], 9). This means that even marginal technological advancements or applications of autonomy to some aspects of a system might lead to autonomous weapons well below the threshold of “actual intelligent robotic soldiers” (UNIDIR 2017UNIDIR. 2017. The Weaponization of Increasingly Autonomous Technologies: Concerns, Characteristics and Definitional Approaches. Geneva: United Nations Institute for Disarmament Research. https://www.unidir.org/files/publications/pdfs/the-weaponization-of-increasingly-autonomous-technologies-concerns-characteristics-and-definitional-approaches-en-689.pdf. [Google Scholar], 8) – i.e. the Terminator-kind of weapons. This makes the prospect of AWS seem very real. Speaking of “increasing autonomy in weapons systems” suggests that autonomy is a process that is well underway:

To me, this isn’t a future conversation. This is very much a conversation about now. I think it’s a red herring to be debating, “Do these systems already exist, not exist?” depending on how you define it, they existed for 30 years or they may never exist depending on how you define them. Again, completely unproductive to have semantic debates about this. There are facts. We have increasing autonomy in weapon systems.⁸8 Interview, 2 July 2021.View all notes

That said, much of UNIDIR’s outreach to decision-makers has been behind the scenes. UNIDIR works closely with state delegations to help them prepare their national positions in the ongoing international or domestic discussion. Moreover, UNIDIR has set up a number of tabletop exercises with small groups of diplomats, military attachés and other governmental representatives to consider distinct scenarios of AWS use. For example, in 2018 UNIDIR held a day-long “technology game” exercise in which participants were exploring different scenarios in which existing weapons systems evolved from present-day to five, ten and then fifteen years on (Vignard 2018Vignard, Kerstin. 2018. “Statement of the UN Institute for Disarmament Research on Lethal Autonomous Weapon Systems.” 2018 Group of Governmental Experts on LAWS, 9 April 2018. Accessed 07/01/2021. https://unidir.org/files/medias/pdfs/unidir-statement-ccw-expert-meeting-laws-april-2018-eng-0-746.pdf. [Google Scholar]). At each of three steps, participants were asked to reflect on how increased autonomy in some aspect of the system (e.g. navigation or weapons release) affected perceptions of their legality and acceptability. The various scenarios also included different operational settings such as the type of target, the nature of the area of deployment, the complexity of the system, and its time operating autonomously.

Simulations differ from other translation practices in their evidentiary base. Unlike reports, written statements or webinars, they do not rely on the codified knowledge of external or in-house experts but on the (bodily) experience of participants. Simulations translate the absent future into acts of performance or play. Engaging in scenario-based exercises has at least two important effects. First, governments that otherwise lack familiarity with the issue are able to gain what Anderson calls “experiential knowledge” (2010Anderson, Ben. 2010. “Preemption, Precaution, Preparedness: Anticipatory Action and Future Geographies.” Progress in Human Geography 34 (6): 777–798. doi:https://doi.org/10.1177/0309132510362600. [Crossref], [Web of Science ®], [Google Scholar], 786) about the kind of concerns that AWS may give raise to. By playing through the different scenarios, participants were able to experience how a specific future might feel like. Second, simulations can introduce a certain element of surprise that is necessary to galvanise recalcitrant states into action. According to one of my interview partners,

they [simulations] are useful because sometimes … your audience feel they have a certain level of knowledge that makes them complacent about things and gaming something else can show people that … things aren’t necessarily going to occur as they expected they will.⁹9 Interview, 25 May 2021.View all notes

Simulations open up the future for alternative trajectories that challenge preexisting knowledge, particularly the belief that autonomy and AI in weapons systems is a military necessity: “They [government] are … being told every day in the media that AI is the best thing ever so regulation is against the grain of that.”¹⁰10 Interview, 2 July 2021.View all notes

Representation

As mentioned earlier, anticipatory norm building presupposes an understanding of what is to come. One of the major questions that states were (and to some extent still are) grappling with is the expected timeline for the development of AWS. For UNIDIR, the discussion about AWS (whether in the CCW or the media) has overly focused on the far end of the autonomy spectrum: the Terminator-like or killer robots scenarios (UNIDIR 2017UNIDIR. 2017. The Weaponization of Increasingly Autonomous Technologies: Concerns, Characteristics and Definitional Approaches. Geneva: United Nations Institute for Disarmament Research. https://www.unidir.org/files/publications/pdfs/the-weaponization-of-increasingly-autonomous-technologies-concerns-characteristics-and-definitional-approaches-en-689.pdf. [Google Scholar], 4, 8, 2014aUNIDIR. 2014a. Framing Discussions on the Weaponization of Increasingly Autonomous Technologies. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/files/publications/pdfs/framing-discussions-on-the-weaponization-of-increasingly-autonomous-technologies-en-606.pdf. [Google Scholar]). While certainly helpful to draw attention to the issue, the “killer robots” meme reinforces the impression that AWS are a far-off phenomenon. It downplays the urgency of the issue and plays into the hands of those who declare fears of AWS as overblown. Against such futurism, UNIDIR tries to ground the debate in “robust” – that is, observable, real-world – trends that point towards increasingly autonomous weapons: states’ expressed interest in moving further along the autonomy spectrum (which is mostly predicated on the expected benefits of AWS) and innovations in the civilian technology sector (for example in AI, sensor technology and processing power) (UNIDIR 2014aUNIDIR. 2014a. Framing Discussions on the Weaponization of Increasingly Autonomous Technologies. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/files/publications/pdfs/framing-discussions-on-the-weaponization-of-increasingly-autonomous-technologies-en-606.pdf. [Google Scholar]). From this perspective, the future development and use of AWS seems not only necessary but inevitable, driven by technological possibilities, hopes and military interests in these systems.

However, UNIDIR warns that the agenda for discussing the future of AWS be set solely by technological capabilities or (perceived) military necessity. Such a tech-centered frame mutes the possibility of human choice and renders alternatives to particular innovations invisible (UNIDIR 2014aUNIDIR. 2014a. Framing Discussions on the Weaponization of Increasingly Autonomous Technologies. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/files/publications/pdfs/framing-discussions-on-the-weaponization-of-increasingly-autonomous-technologies-en-606.pdf. [Google Scholar], 7). This is all the more worrisome because not everything that is feasible from a technical point of view is also normatively desirable. Instead, UNIDIR suggests to “shift to a proactive conversation” on what we expect from technology (Vignard 2016Vignard, Kerstin. 2016. “Statement of the UN Institute for Disarmament Research at the CCW Informal Meeting of Experts on Lethal Autonomous Weapon Systems.” 12 April 2016. Accessed 07/16/2021. https://www.unidir.org/files/medias/pdfs/unidir-s-statement-to-the-ccw-informal-meeting-of-experts-on-lethal-autonomous-weapon-systems-eng-0-648.pdf. [Crossref], [Google Scholar]). Imagining the future, then, means staking out desirable (and undesirable) paths of autonomy in weapons systems. This includes the identification of broad organising principles that could “form a basis for normative development” (UNIDIR 2014bUNIDIR. 2014b. The Weaponization of Increasingly Autonomous Technologies: Considering how Meaningful Human Control Might Move the Discussion Forward. Geneva: United Nations Institute for Disarmament Research. http://www.unidir.org/files/publications/pdfs/considering-how-meaningful-human-control-might-move-the-discussion-forward-en-615.pdf. [Google Scholar], 7): compliance with IHL, human dignity and human control.

Compliance with IHL, which has already codified well known and universally accepted principles on the use of weapons in armed conflict, is one obvious source for a prospective AWS-norm (UNIDIR 2014bUNIDIR. 2014b. The Weaponization of Increasingly Autonomous Technologies: Considering how Meaningful Human Control Might Move the Discussion Forward. Geneva: United Nations Institute for Disarmament Research. http://www.unidir.org/files/publications/pdfs/considering-how-meaningful-human-control-might-move-the-discussion-forward-en-615.pdf. [Google Scholar], 7). The central question here is one of performance, or more precisely, whether one can reliably predict if any given system would be able to comply with IHL requirements, notably the rules of distinction, proportionality, and precautions in attack. While it has been argued that AWS could possibly adhere to these rules better than a human operator, this is very much an argument about future technological capabilities. To avoid such speculations, UNIDIR shifts the focus to those characteristics that are intrinsic to all autonomous systems and that could become a problem from a IHL-point of view, particularly their inherent unpredictability. Like all autonomous systems, AWS are likely to acquire new, anticipated behaviours when they navigate situations that deviate from the environments for which they were originally trained; can adjust in real time to data inputs if endowed with self-learning algorithms; or are confronted with low quality data (Holland Michel 2021Holland Michel, Arthur. 2021. Known Unknowns: Data Issues and Military Autonomous Systems. Geneva: United Nations Institute for Disarmament Research. doi:https://doi.org/10.37559/SecTec/21/AI1. [Crossref], [Google Scholar], 2020Holland Michel, Arthur. 2020. The Black Box, Unlocked: Predictability and Understandability in Military AI. Geneva: United Nations Institute for Disarmament Research. doi:https://doi.org/10.37559/SecTec/20/AI1. [Crossref], [Google Scholar]). These problems are well known from civilian applications of autonomy and AI such as in self-driving cars; they can result in autonomous systems failures that “could cause unintended harm” (Holland Michel 2021Holland Michel, Arthur. 2021. Known Unknowns: Data Issues and Military Autonomous Systems. Geneva: United Nations Institute for Disarmament Research. doi:https://doi.org/10.37559/SecTec/21/AI1. [Crossref], [Google Scholar], 2). Even if this way of representing AWS is tentative, as the careful wording suggests, the focus on inherent features of AWS – their unpredictability – puts it on a robust foundation.

Assessing the desirability of AWS in terms of compliance with IHL has a big advantage: it foregrounds the familiarity of concerns that AWS might give rise to (e.g. harming civilians). And yet, it does now allow to address novel ethical considerations that might arise from new technologies and that are not (yet) codified by international law (UNIDIR 2015bUNIDIR. 2015b. The Weaponization of Increasingly Autonomous Technologies: Considering Ethics and Social Values. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/weaponization-increasingly-autonomous-technologies-considering-ethics-and-social-values. [Google Scholar], 2). In particular, it sidelines concerns about human control and human dignity which are often said to be “at the core of concerns raised about fully autonomous weapons” (UNIDIR 2015bUNIDIR. 2015b. The Weaponization of Increasingly Autonomous Technologies: Considering Ethics and Social Values. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/weaponization-increasingly-autonomous-technologies-considering-ethics-and-social-values. [Google Scholar], 7). Human dignity-related concerns capture fears about the dehuminsation of warfare, including scenarios of “machines [taking] life and death decisions, without any human intent involved” (UNIDIR 2014bUNIDIR. 2014b. The Weaponization of Increasingly Autonomous Technologies: Considering how Meaningful Human Control Might Move the Discussion Forward. Geneva: United Nations Institute for Disarmament Research. http://www.unidir.org/files/publications/pdfs/considering-how-meaningful-human-control-might-move-the-discussion-forward-en-615.pdf. [Google Scholar], 8). This vision of the future is anticipated or known – made “real” – through collective emotions and intuitions tied to the use of these systems. The Institute notes an “instinctive unease” (2015bUNIDIR. 2015b. The Weaponization of Increasingly Autonomous Technologies: Considering Ethics and Social Values. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/weaponization-increasingly-autonomous-technologies-considering-ethics-and-social-values. [Google Scholar], 8), “revulsion” (2015bUNIDIR. 2015b. The Weaponization of Increasingly Autonomous Technologies: Considering Ethics and Social Values. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/weaponization-increasingly-autonomous-technologies-considering-ethics-and-social-values. [Google Scholar], 7), “deep aversion” and “a generalized feeling that the automated taking of human life by machines is somehow wrong” (2014bUNIDIR. 2014b. The Weaponization of Increasingly Autonomous Technologies: Considering how Meaningful Human Control Might Move the Discussion Forward. Geneva: United Nations Institute for Disarmament Research. http://www.unidir.org/files/publications/pdfs/considering-how-meaningful-human-control-might-move-the-discussion-forward-en-615.pdf. [Google Scholar], 8). Such feelings of abhorrence may be based on contingent interpretations of the future, but they can nevertheless form the basis for norm-building. They are indicative of “potentially emergent norms of public conscience regarding fully autonomous weapons”, as UNIDIR explains (2015bUNIDIR. 2015b. The Weaponization of Increasingly Autonomous Technologies: Considering Ethics and Social Values. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/weaponization-increasingly-autonomous-technologies-considering-ethics-and-social-values. [Google Scholar], 6). The so-called Martens Clause in IHL explicitly recognises “the dictates of public conscience” as legitimate sources for new norms to consider (UNIDIR 2015bUNIDIR. 2015b. The Weaponization of Increasingly Autonomous Technologies: Considering Ethics and Social Values. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/weaponization-increasingly-autonomous-technologies-considering-ethics-and-social-values. [Google Scholar], 5).

According to UNIDIR, such new norms are not only found in the shared belief and sentiment that AWS are intrinsically wrong; they have also been more explicitly articulated in public deliberations. What has emerged from the discussions at the GGE meetings so far is a consensus that “weapons systems lacking meaningful human control are unacceptable” (UNIDIR 2015bUNIDIR. 2015b. The Weaponization of Increasingly Autonomous Technologies: Considering Ethics and Social Values. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/weaponization-increasingly-autonomous-technologies-considering-ethics-and-social-values. [Google Scholar], 7). Originally introduced by the NGO Article 36, the concept of meaningful human control spells out an expectation that humans should remain in control of AWS or certain critical functions (e.g. identifying, selecting and applying force to targets) and that the quality of this control be meaningful. While the concept itself remains vague, it is broad enough to integrate different concerns about AWS – including human dignity and compliance with IHL (UNIDIR 2014bUNIDIR. 2014b. The Weaponization of Increasingly Autonomous Technologies: Considering how Meaningful Human Control Might Move the Discussion Forward. Geneva: United Nations Institute for Disarmament Research. http://www.unidir.org/files/publications/pdfs/considering-how-meaningful-human-control-might-move-the-discussion-forward-en-615.pdf. [Google Scholar], 9) – and rejects the notion that AWS will be an unavoidable feature of future warfare. The idea of meaningful human control injects agency and choice into the debate and turns AWS into an admittedly unpredictable but still controllable governance object. It carries hope and confidence that the development of technology is shaped by human deliberation and choice, and not the other way round (UNIDIR 2014aUNIDIR. 2014a. Framing Discussions on the Weaponization of Increasingly Autonomous Technologies. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/files/publications/pdfs/framing-discussions-on-the-weaponization-of-increasingly-autonomous-technologies-en-606.pdf. [Google Scholar], 11).

The question of meaningful human control is also about a very concrete and tangible problem. Even if it is difficult to anticipate what form AWS will take in the future, it is not hard to imagine the lack of human control and its likely consequences. To highlight the potential dangers of losing human control, UNIDIR draws on analogies from other technologies – nuclear power plants and flight control technologies in the civilian aviation industry – that share increasing levels of complexity and automation (UNIDIR 2016UNIDIR. 2016. Safety, Unintentional Risk and Accidents in the Weaponization of Increasingly Autonomous Technologies. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/safety-unintentional-risk-and-accidents-weaponization-increasingly-autonomous. [Google Scholar]). These examples show that adding new computerised and automated devices “can make the user’s job more complicated” as the human operator is struggling to understand why the machine does what it does (UNIDIR 2016UNIDIR. 2016. Safety, Unintentional Risk and Accidents in the Weaponization of Increasingly Autonomous Technologies. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/safety-unintentional-risk-and-accidents-weaponization-increasingly-autonomous. [Google Scholar], 3). This significantly curtails the human capacity to supervise and intervene in time-critical situations with severe consequences. “Accidents” and “catastrophic failures” resulting from a lack of human control (i.e. a human operator’s misunderstanding of the system’s behaviour) are a case in point (UNIDIR 2016UNIDIR. 2016. Safety, Unintentional Risk and Accidents in the Weaponization of Increasingly Autonomous Technologies. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/safety-unintentional-risk-and-accidents-weaponization-increasingly-autonomous. [Google Scholar], 6). By foregrounding similarities between AWS and existing hazardous technologies, UNIDIR suggests that AWS could turn out equally harmful in the future. Because AWS, too, are complex systems, they “will be prone to inevitable accidents” (Holland Michel 2021Holland Michel, Arthur. 2021. Known Unknowns: Data Issues and Military Autonomous Systems. Geneva: United Nations Institute for Disarmament Research. doi:https://doi.org/10.37559/SecTec/21/AI1. [Crossref], [Google Scholar], 1). At the same time, UNIDIR draws attention to the novel dimensions of AWS and how they could undermine human control and harm civilians in unprecedented ways. For one, the likely integration of machine learning in the design of AWS will make the behaviour of these systems even less understandable to a human user (Holland Michel 2020Holland Michel, Arthur. 2020. The Black Box, Unlocked: Predictability and Understandability in Military AI. Geneva: United Nations Institute for Disarmament Research. doi:https://doi.org/10.37559/SecTec/20/AI1. [Crossref], [Google Scholar], 9; UNIDIR 2016UNIDIR. 2016. Safety, Unintentional Risk and Accidents in the Weaponization of Increasingly Autonomous Technologies. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/safety-unintentional-risk-and-accidents-weaponization-increasingly-autonomous. [Google Scholar], 7). For another, the repercussions of even a minor failure of AWS are expected to be of a different magnitude because these systems are designed to inflict damage or kill. This would “worsen the lethal consequences of accidents … The outcomes of AWS failures could include fratricide, civilian casualties, or unintended escalation in a crisis.” (UNIDIR 2016UNIDIR. 2016. Safety, Unintentional Risk and Accidents in the Weaponization of Increasingly Autonomous Technologies. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/safety-unintentional-risk-and-accidents-weaponization-increasingly-autonomous. [Google Scholar], 9; Holland Michel 2020Holland Michel, Arthur. 2020. The Black Box, Unlocked: Predictability and Understandability in Military AI. Geneva: United Nations Institute for Disarmament Research. doi:https://doi.org/10.37559/SecTec/20/AI1. [Crossref], [Google Scholar], 11) Such a scenario is more than creative speculation because it is firmly rooted in real-world examples that involved observable consequences. Thus, analogical reasoning can help render certain future scenarios more robust than others.

Another way to think about AWS and their likely or possible consequences is through existing military systems that “already operate with extremely limited or no real-time human control” (UNIDIR 2014aUNIDIR. 2014a. Framing Discussions on the Weaponization of Increasingly Autonomous Technologies. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/files/publications/pdfs/framing-discussions-on-the-weaponization-of-increasingly-autonomous-technologies-en-606.pdf. [Google Scholar], 5), including anti-materiel, defensive weapons systems such as C-RAM13 or the Israeli Iron Dome (UNIDIR 2014aUNIDIR. 2014a. Framing Discussions on the Weaponization of Increasingly Autonomous Technologies. Geneva: United Nations Institute for Disarmament Research. https://unidir.org/files/publications/pdfs/framing-discussions-on-the-weaponization-of-increasingly-autonomous-technologies-en-606.pdf. [Google Scholar], 5), autonomous surface or underwater vehicles used for tasks such as surveillance, mapping and mine detection (UNIDIR 2015aUNIDIR. 2015a. “The Weaponization of Increasingly Autonomous Technologies in the Maritime Environment: Testing the Waters.” Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/weaponization-increasingly-autonomous-technologies-maritime-environment-testing-waters. [Google Scholar]), and loitering munitions like the Harpy drone (UNIDIR 2014bUNIDIR. 2014b. The Weaponization of Increasingly Autonomous Technologies: Considering how Meaningful Human Control Might Move the Discussion Forward. Geneva: United Nations Institute for Disarmament Research. http://www.unidir.org/files/publications/pdfs/considering-how-meaningful-human-control-might-move-the-discussion-forward-en-615.pdf. [Google Scholar], 6). Even if the international discussion on AWS has not been about these sorts of existing, long-established systems, they help state delegates to comprehend real-life implications of autonomy (in weapons systems) without drifting off in extreme futuristic scenarios. For example, the limitations on communications underwater for increasingly autonomous technologies in the marine environment offers a very concrete opportunity to scrutinise what level of comfort states have with weaponised objects operating outside their control (UNIDIR 2015aUNIDIR. 2015a. “The Weaponization of Increasingly Autonomous Technologies in the Maritime Environment: Testing the Waters.” Geneva: United Nations Institute for Disarmament Research. https://unidir.org/publication/weaponization-increasingly-autonomous-technologies-maritime-environment-testing-waters. [Google Scholar], 11). Table top exercises like the one described earlier serve a similar purpose. They take existing technologies as a starting point and invite state delegates to imagine what specific future scenarios could be problematic, if we move up the ladder of autonomy. For example, participants noted that it was not necessarily the “intelligence” of any weapon that was the primary issue of concern but its specific parameters of use: the latitude of a weapon over time, offensive applications, the use of increasingly autonomous weapons in unbounded and cluttered geographical areas, and systems used against persons (rather than anti-material systems) (Vignard 2018Vignard, Kerstin. 2018. “Statement of the UN Institute for Disarmament Research on Lethal Autonomous Weapon Systems.” 2018 Group of Governmental Experts on LAWS, 9 April 2018. Accessed 07/01/2021. https://unidir.org/files/medias/pdfs/unidir-statement-ccw-expert-meeting-laws-april-2018-eng-0-746.pdf. [Google Scholar]). By interweaving hypothetical (creative) scenarios with elements from the present, the future thus envisioned may not only seem more plausible, but the concerns raised by UNIDIR (and others) become also more urgent. If it is not the technical sophistication of AWS that most worries decision-makers, it is more difficult to dismiss concerns about humanity, dignity or proportionality as inflated.

Assessing UNIDIR’s agency

Although UNIDIR is hardly the most vociferous and assertive actor in current norm building efforts, it occupies a distinct position in the debate for a number of reasons. One lies in its attempts to “de-science-fictionalise” the AWS issue. Rather than sticking to “robocalyptic” scenarios and scare tactics, the Institute builds its representations of the future largely (through not exclusively) on solid (or robust) grounds, by extrapolating from current trends and existing technologies. This anchoring in real-world conditions makes it more difficult to dismiss UNIDIR’s concerns as speculative. The other reason is that the Institute refrains from being openly judgmental and prescriptive/proscriptive. Rather than telling governments what to think or what to do about AWS, UNIDIR gives them tools of how to think about these things. This approach stands in stark contrast to the advocacy strategy of the campaign and has caused some frictions among UNIDIR and the latter. Moreover, it sets UNIDIR apart from other UN bodies that have dealt with the issue and can be considered active supporters of a ban on AWS, including the UN Secretary-General Antonio Guterres and the Human Rights Council. And yet, UNIDIR is far from being a neutral arbiter of information, as the preceding analysis has shown, but shares the concerns raised by the Secretary-General and the campaign.

It’s not like we don’t have a dog in the fight, but we’re not an advocacy organization either. … We have to sometimes move a little bit slower or be less explicit. We find we get very good results when we don’t tell people what to think. … When people are asked to reflect on something rather than told what to think about it, they are more likely to believe that they came to this conclusion on their own and therefore they embrace it better than, “So-and-so told me I should think this.” … That’s been one of the problems, again, with a lot of the advocacy work on everything cyber or autonomy that comes out with, “You should” or “You shouldn’t”, “You must”, and “This is what you need to do.”¹¹11 Interview, 2 July 2021.View all notes

As we can infer from this statement, UNIDIR’s seemingly non-judgmental style of reasoning may open up new venues for influence. The Institute’s initial framing paper, for instance, has served as the basis for an inter-ministerial conversation about autonomy in one Western state, which ultimately ended up turning into a whole government approach to AI and autonomy.¹²12 Interview, 2 July 2021.View all notes While few routes of influence may be as clear-cut, states frequently quote or reflect upon key concepts or framings from UNIDIR’s publications in their national statements at the CCW. This includes key concepts such as UNIDIR’s understanding of autonomy and its suggestion to put human control over technical criteria in characterising AWS (Canadian Delegation 2016Canadian Delegation. 2016. “Canadian Food for Thought Paper: Mapping Autonomy.” Accessed 7/04/2021. https://www.reachingcriticalwill.org/images/documents/Disarmament-fora/ccw/2016/meeting-experts-laws/documents/Canada-mapping-autonomy.pdf. [Crossref], [Google Scholar]; CCW 2018CCW. 2018. “Report of the 2018 Session of the Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems.” Geneva, 9–13 April 2018 and 27-31 August 2018. Accessed 07/04/2021. https://undocs.org/pdf?symbol = en/CCW/GGE.1/2018/3. [Google Scholar]). Some states like Sri Lanka also explicitly reference the scenarios raised by UNIDIR, including “the possibility of accidental catastrophic consequences” (Aryasinha and Ravinatha 2016Aryasinha, Mr, and P. Ravinatha. 2016. “Statement by Ambassador/Permanent Representative of Sri Lanka.” Informal Expert Meeting on Lethal Autonomous Weapon Systems (LAWS), 11th April 2016. Accessed 7/04/2021. https://www.reachingcriticalwill.org/images/documents/Disarmament-fora/ccw/2016/meeting-experts-laws/statements/11April_SriLanka.pdf. [Crossref], [Google Scholar]). While discussions in the CCW are still plagued with persistent ambiguity over the technical, legal and moral aspects of AWS, the work by UNIDIR and other knowledge hubs means that it will become more difficult for states to avert action on the grounds that “we don’t know enough yet”.

Conclusion

How does norm-building proceed in advance of problems taking full shape? States are unlikely to take preventive action unless they have a clear sense of the problem, the nature, scale and urgency of the challenge ahead. AWS are a case in point as the futuristic vibe of the issue and the need for more data and research have served as a pretext for delaying action. In this article I have argued that the emergence of anticipatory norms depends upon forward-looking epistemic practices that produce knowledge about future governance objects and create a normative demand for preemptive action. Focusing on the knowledge work of UNIDIR, the article has analysed three epistemic practices: assembling, translating and representation.

Without the possibility to study AWS in practice, UNIDIR assembles knowledge from a representative set of experts and disciplines, which is then applied to AWS. Assembling both stabilises future objects, by creating consensus, and enables a broadening of the debate, which is particularly important in the consensus-based CCW. By translating the complex details of the expert discussion into a language and form that is accessible to less technically-versed decision-makers, UNIDIR has firmly established itself as a trusted partner of the state parties to the CCW. In representing AWS, the Institute combines creative and robust elements. It invites governments to think imaginatively (creatively) about how AWS could affect legal and ethical principles and how to realise a better future. This way of imagining the future allows to explore novel concerns raised by this weapon technology which have not (yet) been addressed by law, especially concerns about human dignity. Moreover, creative visions underwrite the inherent openness and malleability of technological innovation and progress, which is, in fact, an important prerequisite for anticipatory norms to emerge. For the future to become a site for preemptive/preemptive action in the present, it needs to be seen as open and susceptible to normative change. Since creative visions, on their own, are unlikely to move states into action, they are also infused with robust elements that convey a sense of certainty and urgency. By extrapolating from current trends, UNIDIR depicts the development and fielding of these weapons as a likely and even near-term scenario. Likewise, the Institute foregrounds the familiarity and immediacy of concerns that might arise from the use of AWS by analogising them to existing technologies.

Focusing on the knowledge work of UNIDIR is instructive insofar as its status as independent research institute and its non-judgmental style of reasoning increases its chances of influencing the CCW process. Yet, upon closer investigation, it becomes apparent that UNIDIR is far from being a neutral knowledge broker. Even if it refrains from being prescriptive or proscriptive, its educational material it is infused with normative expectations about how society should (not) be in the future. These representations boil down to two warring visions: one in which humans remain in control over the use of force, and one in which technological innovation proceeds unchecked with potentially grave consequences for human dignity and humanitarian principles. Presenting the future in such a way, as either desirable/comforting or undesirable/threatening, suggests that states ought to take action in the present to prevent an unwanted and enact a desirable future.

Further research can investigate other sites of knowledge production, including SIPRI, the ICRC and the KRB, and the way they construct AWS. Their practices of assembling, translating and representing are likely to differ from UNIDIR’s epistemic practices, not least because they are less constrained by states’ demand for neutral and unbiased input. That does not mean that their approaches are at odds with one other. Rather, they should be seen as complementary efforts in creating a demand for anticipatory norms.

The focus on anticipatory norm-building adopted in this paper seems all the more urgent because research and development, and uses of AI-driven features such as machine-learning are already underway. These practices create precedents and propel the emergence of de facto norms about the appropriate use of force, which takes largely place outside of formal, deliberative law-settings such as the CCW (Bode and Huelss 2018Bode, Ingvild, and Hendrik Huelss. 2018. “Autonomous Weapons Systems and Changing Norms in International Relations.” Review of International Studies 44 (3): 393–413. doi:https://doi.org/10.1017/S0260210517000614. [Crossref], [Web of Science ®], [Google Scholar]; Huelss 2020Huelss, Hendrik. 2020. “Norms Are What Machines Make of Them: Autonomous Weapons Systems and the Normative Implications of Human-Machine Interactions.” International Political Sociology, doi:https://doi.org/10.1093/ips/olz023. [Crossref], [Google Scholar]). The more control shifts from humans to machines in terms of algorithms or machine learning, the less the definition of what use of force is acceptable is subject to open deliberations and human choice. In this regard, processes of anticipatory norm building should be of vital interest to practitioners and scholars who have embraced the idea that the development and use of AWS should be shaped by definitions of what is acceptable from a normative point of view, and not the other way round.