Abstract

This article discusses an important limitation on the degree of autonomy that may permissibly be afforded to autonomous weapon systems (AWS) in the context of an armed conflict: the extent to which international humanitarian law (IHL) requires that human beings be able to intervene directly in the operation of weapon systems in the course of an attack. As there is currently no conventional or customary law directed specifically at AWS, limits on use of autonomous capabilities in weapons, if any exist, must be inferred from the principles, rules and goals of general IHL. The process adopted herein is to look for two broad types of limitations: those which take the form of maximum permissible degrees of machine involvement in regulated activities, and those which take the form of minimum permissible degrees of human involvement. The article’s main finding is that while existing law does not impose limits of the first type, it does impose some of the second type. Specifically, legal obligations borne by individuals (commanders in charge of AWS operations, weapon system operators and others) determine the required minimum capacity for direct human intervention. The article further suggests means by which the required degree of human intervention may be determined in specific circumstances.

Issue Section:

Article

For approximately 8 years, States parties to the Convention on Certain Conventional Weapons¹ (CCW) have been debating whether and how to mount a regulatory response to one of the most significant recent developments in military affairs. Efforts by various armed forces to design and build increasingly autonomous weapon systems (AWS)² for use in armed conflict promise to have a significant impact on the conduct of hostilities.³ Accordingly, those development efforts have generated significant controversy and concern among States, non-governmental organizations and commentators.⁴ Unfortunately, little progress has been made over the course of the debate and there is, as yet, no general agreement about the necessity, much less the substance, of specific regulatory measures to be directed at AWS.

There is, however, some consensus that a critical regulatory consideration is the ‘type and extent of human involvement or control necessary to ensure compliance with applicable law’.⁵ This view is based on the centrality of the human actor as the bearer of legal and moral responsibilities in relation to the conduct of armed conflict. The Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons System (GGE) articulated this position in guiding principle (c) of its 2019 Guiding Principles:

Human-machine interaction, which may take various forms and be implemented at various stages of the life cycle of a weapon, should ensure that the potential use of [AWS] is in compliance with applicable international law, in particular [international humanitarian law]. In determining the quality and extent of human-machine interaction, a range of factors should be considered including the operational context, and the characteristics and capabilities of the weapons system as a whole[.]⁶

This article is concerned with the form of human–machine interaction which is most directly affected by autonomous capabilities in weapons and which is arguably the source of most controversy in discussions about regulating AWS: direct human intervention in the operation of an AWS during an attack. Such an intervention would encompass a range of activities by a human operator which are traditionally associated with operating a weapon, including manually selecting a target, firing a weapon, aborting an attack, or some other legally significant activity. To take a present-day example, Israel Aerospace Industries (IAI)’s ‘Harpy’ loitering munition automates several tasks which would otherwise require direct human intervention, including evaluating possible enemy radiation signatures, aiming the munition at the source of the radiation and aborting an attack if conditions change. Those tasks are still performed as part of the attack process by the operating State via the Harpy’s control system software, but not directly by a human operator. Generally, the article is concerned with identifying whether there is any particular extent to which international humanitarian law (IHL) requires that a human operator be able to directly monitor the operation of an AWS and, where feasible, bypass or override its autonomous control in the course of an operation.

Many objections to AWS development that have been raised in the GGE and elsewhere have been framed as consequences of reducing the capacity for human intervention below some acceptable level: that AWS, which lack the ability to employ human judgement, may behave indiscriminately in complex battlefield conditions⁷; that their use would prevent attribution of responsibility for legal transgressions⁸; that they would be likely to interact with the environment or each other in ways humans could not anticipate, potentially triggering or escalating conflict⁹; and so on.

Human intervention in autonomous operations is not solely a legal concern, even though the question has arisen in the context of several bodies of law, including IHL, human rights law, jus ad bellum and so on. Arguments for some minimum level of human intervention can be, and have been, made on many other grounds, such as ethics, politics and international and regional security.

It is not feasible to address all of those considerations in one paper, though. Instead, this article focuses on ways in which the need for IHL compliance might require that some minimum capacity for human intervention be maintained in AWS operations.¹⁰ In other words, the ways in which IHL requires that human beings be kept in or on the ‘loop’,¹¹ thus limiting the permissible degree of reliance on autonomous systems. The argument is presented in six sections. Section 1 briefly outlines the conception of autonomy which underlies the legal discussion to follow. Sections 2, 3 and 4 investigate the ways in which IHL currently does and does not require direct human intervention in operation of weapon systems. The focus of these sections is rather narrowly on the lex lata; the aim is to discover what type and degree of human intervention is required in order to ensure that AWS use remains strictly in compliance with IHL as it is interpreted today. The conventional framework adopted for this part of the discussion is the First Additional Protocol to the 1949 Geneva Conventions (API).¹² Section 5 offers some thoughts about how guidelines for ensuring a legally acceptable minimum level of human intervention in AWS operations might be formulated. Section 6 offers a tentative conclusion: essentially those persons who are responsible for the operation of AWS must retain the practical ability to meet their IHL obligations, which in turn requires sufficient information about the activities of an AWS as well as the ability to act on that information.

This is a forward-looking article. It does not suggest that any existing AWS, or any which are likely to be fielded within the next few years, test the lower limits of legally acceptable direct human intervention, assuming that such limits exist. However, given the discrepancy between the slow pace of regulatory discussions and the comparatively rapid progress being achieved in the technologies of autonomous systems, it seems prudent to prepare now for situations that could plausibly arise further into the future.

Additionally, the article focuses on special considerations that arise in respect of AWS, rather than on general questions about human involvement and control under IHL.¹³

1. Relevant concepts and terminology

The regulatory debate about AWS is notable for the persistence of disagreements on two fundamental matters.

One disagreement concerns views about the scope of the set of weapon systems that are of interest. Specifically, what capabilities will be (or are) exhibited by weapon systems that should be considered to be ‘autonomous’ and which, if any, of those capabilities raise legal concerns, beyond the general concerns raised by development of any new military technology.

The mandate of the GGE process underway in Geneva confines that group’s discussions to ‘emerging’ technologies and their ‘potential challenges’.¹⁴ That has focused participants’ attention on capabilities which are generally more advanced and potentially present greater operational and legal risk than those provided by weapon systems which are currently used in armed conflict. The types of AWS most discussed in the GGE meetings generally exhibit high degrees of operational independence, heavy reliance on artificial intelligence (AI), utilization of advanced sensing modes and anti-personnel applications.¹⁵ Some contributions to the accompanying academic debate have followed suit, either framing weapon system autonomy as an advanced capability that is yet to fully emerge or defining AWS of legal concern in ways that tend to exclude existing systems.¹⁶

On the other hand, some participants in the GGE process and a number of academic commentators appear to have adopted a broader view of weapon system autonomy. Their contributions approach it as simply a capacity to perform some function without human intervention, a capacity which is already present to some limited extent in many existing weapon systems that have been in use, generally without legal objection, for years or decades.¹⁷

No doubt this disagreement is due, at least in part, to the somewhat amorphous nature of machine autonomy. It does not represent a specific, well-defined technology or physical effect, but is a property that may be realised by various means to various degrees in a range of systems. It represents both a means of control and a characteristic of devices which are to be controlled and so is difficult to discuss in relation to a legal framework which has evolved to regulate the use of simple, human-operated weapons.

For the purposes of this article, autonomy is used in a broad sense to refer to the capacity of a weapon system to operate to some degree without human intervention.¹⁸ Thus, weapon systems that allow but do not require human intervention are considered to exhibit a (lower) level of autonomy, as do weapon systems that can perform only parts of their assigned tasks autonomously. Most legal interest is, of course, in weapons that exhibit autonomy in their ‘critical’ functions of selecting and engaging targets, and that is also the focus of this article.

More specifically, the legally significant quality of an AWS for present purposes is that it partly or fully relieves a human operator of the need to, and perhaps the ability to, directly intervene in operating the weapon during parts of a military operation. It generally does so by means of a software-based control system (essentially a special-purpose computer) which can receive information about the weapon system’s environment and the operation it is undertaking via sensors and/or data links and respond by applying control actions directly to the weapon in place of those which would otherwise be applied by a human operator. Thus, the human operator is partly or fully removed from the ‘loop’.

In other words, autonomy in weapon systems represents a transfer of direct performance of specified combat functions from humans to machines. That transfer may be partial or complete, temporary or permanent (in the context of the operation). The argument below separately examines each ‘end’ of that transfer for its compliance with IHL rules: the legal effects of having a machine perform combat functions which are traditionally done by humans and the legal effects of removing humans from the role of performing those functions.

The second matter concerns terminology relating to control over the behaviour of weapon systems. For the purposes of this article, a distinction may be drawn between the narrow set of ‘direct’ control measures traditionally associated with operating a weapon (aiming, pulling a trigger and so on), herein referred to as ‘human intervention’ and the much broader set of measures that States and armed forces take to ensure that lethal force is applied in accordance with legal obligations (formulation of strategic goals, rules of engagement and so on, in addition to the direct measures referred to here as ‘intervention’). That broader set of measures is described here as ‘control’.

This distinction is in contrast to the rather less structured, and occasionally inconsistent, usage of terms such as ‘control’ throughout the regulatory debate. At various times, ‘control’ has been used with a range of qualifiers to refer to either or both of the narrow and broad aspects of control over weapon systems or used in ways that leave it unclear which aspect is being discussed. For example, the concept of ‘meaningful human control’ has been employed by a range of actors to help guide the global debate about regulation of AWS for several years, but its precise meaning is still unclear.¹⁹ In the first session of the 2021 GGE meetings in Geneva, it has been used in a quite narrow sense, similar to ‘human–machine interaction’²⁰, and in a sense that could be read more broadly.²¹

Here, control, as it applies to weapons, refers to the totality of measures taken to ensure that a weapon behaves in a certain way in a certain situation. Direct human intervention (such as manually aiming, pulling a trigger, etc.) is one means of exercising control over a weapon. The design, testing and implementation of an autonomous control system is another means, in that the purpose of autonomous control is still to ensure that the weapon behaves in a specific, desired manner. Both of those are, ultimately, means of human control over the weapon. More broadly still, control over each individual weapon system forms part of a State’s overall system of control over its use of military force. That broader system of control supports and works in concert with the specific control measures applied to individual weapon systems.²²

To speak of autonomous control as being an alternative to human control can be misleading and this article avoids doing so. Unquestionably, some form of human control must be applied to a weapon in order to satisfy the requirements of IHL. The issue being addressed here is the degree of direct human intervention which must form part of that overall system of control.

2. No explicit limits on autonomy

There are currently no explicit references to autonomous systems in any IHL convention. Nor can it be said that any practices relating specifically to autonomous systems have risen to the status of customary law. A legal instrument focusing on autonomy in weapon systems is one potential outcome of the ongoing GGE process, but this possibility seems remote owing to fundamental disagreements between States as to the need for further regulation.²³

Autonomous weapons per se do not fall within the scope of any existing weapon-specific restriction or prohibition, although there is no reason to suppose that the addition of autonomous capabilities to an existing weapon which is within such scope would affect the applicability of those restrictions. As the focus of this article is only on autonomous capabilities and general IHL, their interaction with other weapon-specific rules will not be considered.

Indeed, IHL as it stands today contains only minimal references to any specific means of controlling or activating weapons. Other than some references to automatic contact mines in the Hague Convention VIII on sea mines,²⁴ to mines being detonated by the presence of persons in the Ottawa Convention on anti-personnel mines,²⁵ and likewise in CCW Protocol 2 on mines, booby traps and other devices,²⁶ the law appears to be generally silent on the technical means by which weapons are or are not to be operated. Beyond those exceptions, reference is made only to the types of behaviours that must result from whatever means of control is used: that weapons must not be directed against civilians or civilian objects, for example.

It follows that specific limits on use of autonomous capabilities in weapons, if any exist, must be inferred from the principles, rules and goals of general IHL. IHL undoubtedly applies to weapon systems which employ some degree of autonomous control as it does to all weapons,²⁷ so the limits in question are those which are necessary in order to respect those general constraints.

Given the lack of explicit legal restrictions on the means by which weapons are to be controlled, though, is there a reason to believe that any particular requirement for direct human intervention exists? Autonomous control aims to reduce or remove the need for direct human intervention in operating a weapon system, but like all technologies, it is limited in the capabilities it can provide, which suggests that at least a de facto requirement exists for some degree of human intervention. At a minimum, baseline standards imposed by IHL, such as the prohibition on the use of inherently indiscriminate weapons,²⁸ seem to require some degree of human intervention in autonomous operations where the capabilities of a weapon’s control system and any other control measures taken are unable to sufficiently ensure that only combatants and military targets will be selected. Beyond that, the unique nature of autonomous control justifies a more in-depth examination. It promises to partly displace some personnel from the roles which the law assumes them to play and delves into areas which weapon technologies do not typically touch (specifically, the decision-making processes leading to an act of violence). Autonomous control of weapons could, therefore, conceivably require context-specific constraints on its use which differ from those that are at issue with weapons generally. The possibility of such contextual constraints is discussed in the next two sections.

3. Preclusion of machine performance of regulated activities

This section examines arguments for minimum levels of human involvement with autonomous weapon operations based on the notion that tipping the balance of direct performance of combat functions too far away from humans and towards machines is impermissible due to some aspect or consequence of machine performance which is seen to be problematic.

There do not appear to be any rules of IHL which necessarily preclude a degree of reliance on machines to perform tasks that are the subject of legal obligations. The success of such reliance is, however, in the case of AWS, heavily dependent on the standards of AI performance in military decision-making that are achievable.²⁹ The standards that are achievable today and are expected to be achievable in the near future are limited by the fact that the types of decision-making processes that are required in order to abide by many of the rules of IHL are often ill suited for computers and software.³⁰ Schuller identifies the difficulty of establishing quantifiable standards of decision-making performance as one important factor contributing to this limitation:

For at least the foreseeable future, AI will require these standards to be described in a numerically quantifiable manner. This means that the goals established for the AI are amenable to quantitative description. For example, “do not kill more than five civilians” is a quantifiable standard, discounting, of course, the question of whether the system is capable of distinguishing between the status of people on the battlefield. On the other hand, “do the right thing” is inherently not a quantitative standard.³¹

AI’s reliance on objective, quantifiable standards is particularly limiting in respect of decisions which directly import value judgements. Perhaps, the most prominent example would be proportionality assessments, which aim to determine whether the civilian harm that is expected to result from a proposed attack would be excessive compared to the anticipated military advantage³²:

In the past, there was no imperative to set clear standards because the task of deciding what was “excessive” was left to military commanders and rarely second-guessed. Observers have lamented that machines could never possibly be able to replicate human decision-making in this realm. There are many conundrums to consider in answering this complicated question, but a key one amongst them is whether the decision a machine is tasked with is the type of decision that machines are adept at making on our behalf. An important aspect of the inquiry is whether the goal established for an Al is numerically quantifiable.³³

More specific scenarios are also sometimes proposed to illustrate a perceived shortcoming of AI and autonomous systems. For example, in relation to the principle of distinction and the difficulty of determining the combatant status of individuals in situations where civilians and combatants are intermingled:

…a frightened mother may run after her two children and yell at them to stop playing with toy guns near a soldier. A human soldier could identify with the mother’s fear and the children’s game and thus recognize their intentions as harmless, while a fully autonomous weapon might see only a person running toward it and two armed individuals.³⁴

Although these and similar observations about the capabilities of specific AI systems may currently be accurate in specific operational contexts and the concerns they raise contextually valid, it is doubtful that they effectively establish a need for a general limitation on autonomous capabilities, or assist in formulating a universal prescription for a minimum capacity for human intervention in autonomous operations.

The problem stems from the need to reason inductively from specific problematic scenarios such as those cited above to general rules about autonomy and human intervention and the errors that process inevitably involves. It may well be that the identified problems merely reflect the current state of the art in AI and autonomous systems: temporary challenges which will be overcome rather than inherent limitations which would reliably ground a legal requirement for a degree of direct human intervention. Capabilities required for adherence to the principle of distinction and other legal principles are likely to improve with further research, and AI’s reliance on readily quantifiable standards might also be eased as the relevant technologies advance. If that is the case, these current limitations are appropriately dealt with by relying on existing preventative measures. Those measures include the requirement to conduct weapon reviews which identify the types of operations in which a specific weapon system may be employed in compliance with IHL and other law,³⁵ and the requirement for commanders in charge of attacks to select means and methods which are expected to minimise civilian harm.³⁶

Even if these or other barriers to AI capabilities appear to be more long lasting, it is doubtful that any special regulation is necessary on that basis. All weapon-related technologies offer finite capabilities and consequently the use of any weapon system must be avoided in circumstances in which it is reasonably expected to result in a proscribed act. Diligent application of the existing rules of general IHL restrains the use of weapons so as to avoid such acts. There does not appear to be a reason to believe that the addition of an autonomous control system, essentially a computer, to a weapon would undermine that process such that special requirements for human intervention must be mandated.³⁷

4. Positive requirements for human intervention

The previous sections showed that IHL does not explicitly limit reliance on autonomous systems, nor does machine performance of attack-related tasks necessarily breach legal rules. This section focuses on the ‘human’ side of the human–machine task allocation spectrum. It examines arguments for a minimum capacity for human intervention on the basis that a certain type or degree of human intervention in and of itself is necessary for compliance with IHL. Arguments of this form are perhaps more often made on bases other than IHL compliance: that too little human intervention in a decision to use lethal force violates a targeted person’s right to dignity,³⁸ that allowing a machine to ‘make’ the decision to kill is mala in se,³⁹ and so on. Nevertheless, a strong argument can be made that the lex lata requires that a sufficient capacity be retained for individuals to oversee and intervene in the operations of AWS.

The burden of complying with the rules of IHL ultimately rests with people.⁴⁰ In some instances, that burden is expressed as an obligation formally assigned to individuals. Most relevant to the current discussion, Article 57(2)(a) of API charges ‘[t]hose who plan or decide upon an attack’ with undertaking a range of precautionary measures aimed at avoiding or minimising civilian harm when preparing to conduct attacks. By Article 31(1) of the Vienna Convention on the Law of Treaties’⁴¹ (VCLT) requirement that ‘[a] treaty shall be interpreted in good faith in accordance with the ordinary meaning to be given to the terms of the treaty in their context and in the light of its object and purpose’, ‘those who plan or decide upon an attack’ must be read as referring to human beings. State practice⁴² establishes that, at the least, it refers to commanding officers in charge of attacks,⁴³ and perhaps more broadly to ‘all personnel who have the authority and practical possibility to affect the course of an attack’.⁴⁴

Beyond that, ‘Parties to the conflict’, being States and organised armed groups, are the primary bearers of obligations under IHL. Parties to a conflict act through their organs and agents, though: most significantly, in this context, their armed forces and the personnel who serve in those forces.⁴⁵ Additionally, States in fact bear responsibility for the acts of a much broader circle of people who may be involved in a conflict, specifically those whom it empowers to exercise its authority, those who act on its instructions and those whose conduct it adopts as its own.⁴⁶ Regardless of the precise scope, though, the important point is simply that it is responsible individuals, in both a formal and a practical sense, who bear the burden of ensuring that the law is respected.

Responsible persons must, of course, retain the practical ability to meet their legal obligations. That is required by the customary law principle of pacta sunt servanda and the corresponding treaty obligation in VCLT Article 26: ‘Every treaty in force is binding upon the parties to it and must be performed by them in good faith.’ The principle’s application to the current subject matter is reinforced by API Article 1(1) (‘The High Contracting Parties undertake to respect and to ensure respect for this Protocol in all circumstances’) and the accompanying text in the Commentary by the International Comittee of the Red Cross (ICRC).⁴⁷

Where the obligation relates to the operation of a weapon system, or oversight of an attack, it means those persons must retain the capability to ensure that weapon systems for which they are responsible behave consistently with the constraints imposed by IHL. In modern armed forces, that capability is derived from the collective efforts of everyone involved in the design and operation of weapon systems. As one example of the breadth of this effort, in the 2019 GGE meeting, Australia described its ‘system of control’ over weapon systems including AWS:

A system of control is an incremental, layered approach to applying control, covering all aspects of a weapon system from design through to engagement. This system is applicable to the full spectrum of weapon systems, including those without, with limited and with near-fully autonomous functionalities. Through a system of control, militaries are able to appropriately ensure that all capabilities, including AWS, operate in a lawful and regulated manner. Many militaries already employ systems of control, in one form or another.⁴⁸

In Australia’s case, the system of control comprises a range of measures applied in multiple stages: broad application of the legal and policy framework to guide all subsequent stages; weapon system design and development; weapon system testing, evaluation and review; weapon system acceptance, training and certification; pre-deployment selection of an appropriate weapon system; establishment of weapon use parameters; pre-deployment certification of selected weapon systems and training of personnel; application of strategic and military controls over the use of force to produce rules of engagement orders and targeting directives that are acted on through a structured targeting process; and finally after-action evaluation of a weapon system’s performance.⁴⁹

This overview indicates that there are many dimensions to the control of weapon systems beyond direct intervention by a commander or weapon system operator (again, reading ‘control’ broadly as the totality of measures which contribute to determining the actions taken by or with a weapon system during an attack). Where the overarching legal framework governing such a system of control assigns duties to specific individuals (such as ‘those who plan or decide upon attacks’), those individuals must be afforded sufficient means within the system of control to enable them to meet their obligations. In respect of operating weapon systems or overseeing attacks, that ability may be said to rest on two relevant foundations: access to sufficient information about the attack, the nature and behaviour of the weapon and its interaction with targets and the environment; and a sufficient ability to affect the behaviour of the weapon as required by changing circumstances. This is the capacity for human intervention that is required for compliance with IHL.

The need for an ongoing ability to ensure control of weapons during an operation is expressed most clearly in API Article’s 57(1) requirement that ‘[i]n the conduct of military operations, constant care shall be taken to spare the civilian population, civilians and civilian objects,’ but is also expressed or implied by other rules. For example, API Article 57(2)(b) specifies that ‘an attack shall be cancelled or suspended if it becomes apparent’ that proceeding with the attack would result in an indiscriminate or disproportionate outcome.

That, in turn, raises two interrelated questions: what degree of control (in the broad sense) over weapons is sufficient for IHL compliance and what capacity for individuals to intervene (in the narrow sense) in operation of weapon systems is required in order to both ensure that level of control and enable the individuals to meet their obligations. This article does not extend to a general discussion of requisite standards of control over weapons during armed conflict; rather, it assumes that a hypothetical State’s overall system of control ensures behaviour of weapon systems which is deemed to be in compliance with the State’s IHL obligations. The question to consider is: how can one describe the capacity for intervention by responsible individuals in weapon system operation which is required by relevant IHL rules?

Unfortunately, the answer is somewhat unclear. The text of some relevant provisions (such as ‘constant care’) is expressed in terms that suggest a very high standard is required. In practice, though, at least a limited degree of inability to intervene in operation of a weapon is often tolerated in respect of specific technologies. Not all missiles can be aborted at any moment after they are launched, or actively guided in flight; land and sea mines are routinely left emplaced for some considerable time without direct supervision; and so on. The ICRC Commentary on API Article 57(2)(b)⁵⁰ is rather ambiguous about the degree of ability to intervene in an ongoing attack which that Article might require:

It is principally by visual means…that an attacker will find out that an intended objective is not a military objective…However, with the increased range of weapons, particularly in military operations on land, it may happen that the attacker has no direct view of the objective, either because it is very far away, or because the attack takes place at night. In this case, even greater caution is required.⁵¹

The long history of reliance by armed forces on weapons such as missiles, torpedoes and mines the operation of which cannot easily be intervened in after activation indicates that some such inability is not, in itself, a mark of illegality. At the same time, though, regulatory measures such as restrictions on the emplacement of anti-personnel mines⁵² indicate that it is a risk factor which bears on the legal status of a weapon.

Moreover, increasing degrees of autonomous capability in weapon systems further complicate the issue. In current practice, a capacity for intervention by responsible persons is often the only feasible means by which to avoid or minimise harm to civilians or other proscribed outcomes of operating weapons in complex and dynamic battlefield conditions. However, autonomous technologies endow weapon systems with some ability to detect information about their environment and respond to that information in ways determined by control system software which States are obliged to test and review with their IHL obligations in mind. Accordingly, AWS may greatly reduce the need for, and perhaps in some cases the possibility of, direct human intervention in weapon system activities, without necessarily increasing the chance of a proscribed outcome to an operation.⁵³

In so doing, autonomous technologies perhaps expose an assumption deep within IHL that will soon need to be re-examined: the assumption that weapons are ‘dumb’; that their behaviour is fixed and limited to carrying out only those specific actions which have already been fully determined by humans; that active human guidance is required in order to ensure that the behaviour of weapons remains in compliance with the law as situations develop.

That assumption, to whatever extent it exists, is somewhat undermined by weapons that can respond in complex, human-defined ways to events occurring during an operation without necessarily needing a human to directly intervene. Notably also, this development comes at a time when other changes in the ways in which conflicts are fought might necessitate a rethink of the role of the human. For one, the expansion of armed conflict into cyberspace and outer space promises to separate human combatants from the weapons they wield in ways that require the use of artificial intermediaries. Furthermore, steady increases in the tempo of operations and the volumes of information that must be processed in order to support increasingly complex decision-making have increased reliance on artificial systems with advanced decision-making or decision-support capabilities.⁵⁴ These and other developments raise questions about how to formulate appropriate requirements for direct human intervention in operation of weapon systems.

5. An approach to describing the requisite human intervention

If a statement is to be formulated about the minimum acceptable degree of human intervention in autonomous operations, it should be done on a basis which accurately reflects the requirements of the lex lata. In this case, the need is to ensure that persons who are responsible for seeing that weapon use remains within the bounds of the law have the means to do so. That requires that they have a sufficient informational basis on which to found decisions about weapon use and a sufficient capacity to see that the outcomes of those decisions are implemented.⁵⁵

The specific measures required in order to achieve that will, of course, vary considerably depending on the circumstances of the operation in question, but something can be said about the means by which appropriate measures may be arrived at.

It is argued here that the lex lata’s requirement for human involvement with weapon systems, at least insofar as those systems have a degree of autonomous capability, may be usefully characterised as an exercise in risk management,⁵⁶ and a statement of the required type and degree of human intervention in autonomous operations can be formulated, at least in part, on that basis. Where a weapon system is manually operated, the responsibility to ensure it is used in compliance with legal rules is principally a responsibility to operate the weapon in a certain way, or to ensure that subordinate personnel operate the weapon in a certain way. Where a weapon system is operating autonomously, though, its control system plays the role of ‘operator’ to some extent. The task of the responsible person is instead to ensure that the weapon system is operated only in conditions for which it has been designed and certified, consistent with the State’s overall system of control and, as far as is feasible, ensure that the actions the weapon system actually takes during an operation are consistent with the State’s legal obligations. That essentially amounts to monitoring for and responding to developments, whether in the environment or in the weapon system itself, which would unacceptably increase the risk of violations of IHL. In other words, it amounts to managing the risks of operating the AWS: specifically, the risks that the weapon’s control system might cause it to act in a way which would fail to meet the obligations borne by the responsible person.

The ability to effectively manage those risks is premised upon the information made available to the responsible person and the types of actions they are required to take in response, meeting certain criteria.

The information made available to the responsible person must, of course, encompass all matters which must be accounted for by the legal decision that is to be made, in sufficient detail, to the extent that is ‘practicable or practically possible, taking into account all the circumstances at the time of the attack’.⁵⁷ The information must also be practically usable, in terms of its format, presentation, timeliness and any other characteristics which might reasonably be expected to affect the ability of the responsible person to utilise it in making the decision at issue.

The capacity to select the correct response to a given development during an operation requires a sufficient degree of predictability in the conditions that will be encountered as a result of the responsible person’s actions and the behaviour of the weapon system in those conditions:

To comply with IHL rules, notably the prohibition on indiscriminate attacks, the prohibition on indiscriminate weapons, the principle of proportionality and the requirement to take precautions in attack, persons must be capable of limiting the effects of the weapons they use. They can only do this, however, if they can reasonably foresee how a weapon will function in any given circumstances of use and the effects that will result. Predictability is therefore key to ensuring that a weapon’s use complies with IHL.⁵⁸

Predictability requires that the scope of the possible outcomes and the difficulty of relating the responsible person’s actions to those outcomes be kept to manageable levels. Those, in turn, depend on a wide range of situation-specific factors: the complexity of the weapon system and the operating environment, the presence and behaviours of civilians and enemy combatants, the expected length of time until a responsible person will next be able to guide the actions of the weapon system and so on.

The specific arrangements that satisfy those requirements will be highly context specific. However, certain significant risk factors that characterise autonomous machines provide some general guidance on where people who bear legal obligations in relation to weapon use are likely to need the ability to intervene directly in an AWS operation.⁵⁹

A. Runaway failure

When a weapon is operated manually, each act done with the weapon is initiated by a person and following that act the weapon ceases to operate until the person initiates some other act (when a rifle’s trigger is pulled, the rifle fires once or until the trigger is released, then stops). If there is an error or malfunction in use of the weapon, such that a rule of IHL is broken, it is either a human failure or a failure of the weapon which is likely to be immediately known to its human operator (the marksman knows immediately if the shot hits a civilian instead of an enemy combatant). Being aware of the failure, the person operating the weapon can take appropriate action. Generally, that failure will not continue beyond a person becoming aware of it, other than in cases of deliberate violations of the law. On the other hand, if an AWS fails after being activated, in circumstances in which a human cannot quickly intervene, the subsequent behaviour of the AWS will be in the nature of a malfunctioning machine rather than a human who has acted incorrectly. If the failure is of a type that allows the weapon to continue operating, for example an error in the targeting system of an automated gun turret which falsely identifies civilian targets as military targets, the weapon could continue to act in proscribed ways, potentially until human intervention is available. Horowitz and Scharre mention this risk in arguing for ‘controllability’ in AWS operation:

Militaries generally have no interest in developing weapon systems that they cannot control. However, a military’s tolerance for risk could vary considerably…The desire for a battlefield advantage could push militaries to build weapons with high degrees of autonomy that diminish human control…While any weapon has the potential for failure and accidents, [AWS] arguably add a new dimension, since a failure could, in theory, lead to the weapon system selecting and engaging a large number of targets inappropriately. Thus, one potential concern is the development of weapons that are legal when functioning properly, but that are unsafe and have the potential to cause a great deal of harm if they malfunction or face unanticipated situations on the battlefield.⁶⁰

A human operator, being separate from the weapon systems they operate and availed of different senses, behaviours and sources of information, is able to act as a fail-safe device when a weapon system malfunctions. If some part of the role of the human is absorbed into the weapon system, thought must be given to how to retain a fail-safe function in order to contain errors and malfunctions of the weapon system. Of course, most sophisticated AWS are likely to be designed and built with technical safeguards within the weapon system which will, as long as they are operating correctly, enable an AWS to detect either its own malfunctioning or changes in battlefield conditions which are beyond those for which the weapon system has been certified. However, technical safeguards are themselves parts of the weapon system that are subject to failure. They will not necessarily suffice in all circumstances as the assumption is that the weapon system has malfunctioned, which means that any built-in fail-safe devices might also have malfunctioned.

The possibility of this type of failure bears on the most fundamental principles of IHL, which the person responsible for an AWS operation is bound to uphold. When there is a chance that civilian targets will be selected by a weapon system, the principle of distinction is enlivened. When the degree of civilian harm anticipated in an attack may grow unexpectedly, the chance of that impacts a proportionality assessment. When the risk of such unnecessary civilian harm is due to a mode of machine behaviour which only human intervention can reasonably overcome, the ‘constant care’ obligation, at least, arguably requires that intervention is available at all times, or as close as can feasibly be achieved in the circumstances of a particular attack. Further, the ability to intervene should be founded on access to legally sufficient information about the weapon system’s hardware and software components and should grant the ability to detect and either correct or otherwise respond effectively to malfunctions in the weapon system being monitored.

B. Brittleness

Brittleness here refers to the commonly encountered problem of automated systems failing when faced with challenges that go beyond those which their control system was explicitly developed to deal with.⁶¹ Computers are only able to execute the software installed on them; their ability to function adequately in any situation is entirely defined by the software instructions provided to them by their developers. They are, therefore, able to reliably function in the situations which their developers foresaw and programmed them to function in and no others. When faced with situations outside the scope of those which their control software addresses, they are likely to fail; computers cannot exercise common sense in order to deal with unfamiliar situations in the way that humans can.⁶² This is the core of the idea of ‘brittleness’; there is a rigidly defined set of situations in which any software can operate successfully and it is unable to ‘bend’ itself to fit situations not in that set. While programmers naturally attempt to anticipate all situations to which their software will be exposed, the complexity of the weapon systems and the circumstances in which they will operate often limits the success of those attempts. Brittleness may be encountered for various reasons:

the inability of the designer to anticipate and design for all of the scenarios that could arise during the use of the system, a deliberate decision by the designer to use an oversimpliﬁed model of the decision task (due to cost, time or technological limitations), a failure of the designer to correctly anticipate the behavior of the system in certain situations, or a failure to correctly implement the intended design.⁶³

Successful operation of an AWS therefore depends on the combination of its internal (software and other components) and external (environment) states. If either one deviates too far from what has been anticipated, or becomes incompatible with the other, human intervention is likely to be needed. In the event that an AWS encounters circumstances in which its control system, even if operating normally, cannot keep its operation within the bounds of the responsible person’s legal obligations, that person must be in a position to correct the situation in order to avoid or mitigate a breach of the law. Like the risk of runaway failure, the brittleness which may be expected in AWS, thereby reinforces and shapes the need for some capacity for human interaction with AWS during an operation. Unlike the runaway failure problem, though, brittleness points to the need for the information available to a responsible person to include information about the AWS’s operating environment which is sufficient to support a judgement about whether the weapon system is operating in circumstances compatible with those for which it has been tested, reviewed and certified.

C. Software complexity

The final point is a more general one about the challenges presented by the tremendous complexity of expected future (and some existing) weapon systems and the battlefield environments in which they will be required to operate. As noted above, the ability to intervene effectively in the operation of an AWS requires that a responsible person be able to reliably foresee the consequences of their intervention, or of their decision not to intervene. That foreseeability is negatively impacted by the complexity of AWS construction, which may be expected to keep increasing. Modern weapon systems, like other digital systems, comprise large numbers of hardware and software components, many of which are complex in themselves as well as being interconnected with and dependent upon other components in complex ways. That trend of increasing complexity is a natural consequence of increasing the functionality of weapon systems: equipping them to undertake more sophisticated tasks in more difficult environments means enabling more responses to more combinations of command inputs and environmental stimuli in order to perform more, and more complex, operations. Even when the weapon system is behaving as designed, it is therefore potentially difficult for a human observer to reliably foresee whether an intervention or lack thereof is likely to result in behaviour which complies with legal requirements. This is especially so when the consequences of the intervention might depend on incompletely understood environmental factors affecting the operation of the weapon system, or responses by intelligent adversaries. A related problem is that of the difficulty of finding and fixing software errors⁶⁴:

AWS rely on large and extremely complex software applications that are very challenging to write and to maintain. The scale of the challenge presented may be appreciated by considering the rate at which errors typically occur in software. While there is considerable variation, studies have shown the average software industry error rate to be approximately 15 – 50 errors per 1,000 lines of code written.⁶⁵ As a rough indication of the number of errors that may be encountered in practice, one modern weapon system, the F-35 Joint Strike Fighter, uses over 20 million lines of code.⁶⁶ Further, the difficulty of reliably predicting weapon system behaviour is increased exponentially by the combined effects of increasingly complex tasks, difficult and chaotic operating environments and the often unpredictable actions of sophisticated adversaries.⁶⁷

Taken together, the tremendous complexity of autonomous control system software, the ever-present risk of undiscovered bugs in that software and the complex and chaotic battlefield conditions in which control system algorithms must operate point to a need for careful restrictions on the extent to which AWS are allowed to operate in conditions which do not allow timely human intervention.

Decisions about how best to uphold one’s legal obligations while also meeting other professional requirements can be difficult at the best of times. When the decision relates to the operation of a highly complex weapon system in a difficult combat situation, it is very important that the scope of each decision be kept as small as possible so that it is feasible for human decision-makers to be sure they are meeting the stringent standards imposed by IHL. That suggests that armed forces should take a generally cautious approach to permitting AWS to operate in circumstances where human intervention is not possible.

6. Conclusion

IHL does not deal explicitly with the notion of human intervention in operation of weapons. It deals in depth with control over weapons by defining a range of legal obligations the observance of which necessarily requires a high level of control over the application of force, but not specifically with the degree of human intervention that must be employed in applying that control.

This article has attempted to determine the extent of direct human intervention in autonomous operations which is required by IHL today. It has found that the picture is rather unclear, but that some inferences can be drawn from the fact that persons who bear legal responsibilities for the operation of weapon systems must retain the practical ability to meet those obligations, which in turn requires that they be assured a sufficient informational basis about the weapon and its actions and the ability to effectively act on that information. Autonomous control introduces the possibility of applying a high degree of control to weapons without necessarily requiring direct intervention by persons who bear legal responsibility for the use of those weapons. It also introduces new risks of technical failures and of the tragic outcomes that may follow. Those new challenges help to define the capacity for human intervention which must be retained.

Although they are framed here as statements about the requirements of the lex lata, the findings of this article could also be seen as a contribution to an instrument setting out the types and extent of human intervention that are to remain possible in operation of AWS.

In the short term, such a statement, whether in the form of a legal or political instrument, may well be sufficient to address concerns about the rapid development of autonomous capabilities in modern armed forces. In the longer term, perhaps there will be a need to ask deeper questions about the roles of human beings in conflict, as they will have to be asked about the roles of humans in other fields which are being reshaped by automation and machine autonomy. Of course, it is not just a legal question. Ethical, political, security and other concerns abound and will also need to be addressed. For the near future of autonomous weapon development, though, it is hoped that the discussion in this article will help to fill some of the gaps in the international regulatory debate.

Acknowledgements

The research for this article received funding from the Australian Government through the Defence Cooperative Research Centre for Trusted Autonomous Systems. The views and opinions expressed in the article are those of the author, and do not necessarily reflect the views of the Australian Government or any other institution. The author wishes to thank Rain Liivoja, Kate Devitt, Paul Jones and the anonymous reviewers for their helpful comments.

Footnotes

1

Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects (adopted on 10 October 1980, entered into force 2 December 1983) 1342 UNTS 137.

2

The meaning of ‘autonomous’ in the context of weapon systems is discussed in more detail in the next section. It is sufficient to note here that AWS are weapons, often employing advanced computing capabilities, which can select and attack targets on their own to some significant degree, without necessarily needing any direct human involvement. This is broadly consistent with the definition given in: US Department of Defense, ‘Autonomy in Weapon Systems’ (Directive No 3000.09, Office of the Under Secretary of Defense for Policy, 21 November 2012) 13.

3

See generally, P Scharre, Army of None–Autonomous Weapons and the Future of War (Norton 2018).

4

See the wide range of views presented by participants in the United Nations-hosted discussions on regulating AWS: ‘Background on Lethal Autonomous Weapons Systems in the CCW’ (United Nations, Geneva, 2021) <https://www.un.org/disarmament/the-convention-on-certain-conventional-weapons/background-on-laws-in-the-ccw/> accessed 4 May 2022. See also the activities of the Campaign to Stop Killer Robots, a coalition of non-governmental organisations which has been active since 2012 and has participated extensively in multilateral discussions about regulation of AWS <https://www.stopkillerrobots.org> accessed 4 May 2022.

5

Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems, ‘Commonalities in National Commentaries on Guiding Principles’ (September 2020) para 21(c) <https://documents.unoda.org/wp-content/uploads/2020/09/Commonalities-paper-on-operationalization-of-11-Guiding-Principles.pdf> accessed 4 May 2022.

6

Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems, ‘Report of the 2019 Session of the Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems’ (25 September 2019) CCW/GGE.1/2019/3 Annex IV (c).

7

See, eg, T Chengeta, ‘Measuring Autonomous Weapon Systems against International Humanitarian Law Rules’ (2016) 5 Journal of Law & Cyber Warfare 78.

8

See, eg, ‘New Weapons, Proven Precedent: Elements of and Models for a Treaty on Killer Robots’ (Human Rights Watch, 20 October 2020) <https://www.hrw.org/report/2020/10/20/new-weapons-proven-precedent/elements-and-models-treaty-killer-robots> accessed 4 May 2022 ; T Chengeta, ‘Accountability Gap: Autonomous Weapon Systems and Modes of Responsibility in International Law’ (2016) 45 Denver Journal of International Law & Policy 1.

9

See, eg, B Laird, ‘The Risks of Autonomous Weapons Systems for Crisis Stability and Conflict Escalation in Future U.S.-Russia Confrontations’ (The Rand Blog, 3 June 2020) <https://www.rand.org/blog/2020/06/the-risks-of-autonomous-weapons-systems-for-crisis.html> accessed 4 May 2022.

10

Reference is made herein to the recent report: V Boulanin and others, ‘Limits on Autonomy in Weapon Systems: Identifying Practical Elements of Human Control’ (June 2020) Stockholm International Peace Research Institute and The International Committee of the Red Cross. The foundations of the argument presented here are broadly consistent with the findings of that report and the argument itself may be viewed as a partial response to the report’s third recommendation on page 38: ‘States should clarify where rules under [IHL] already set constraints on the development and use of [AWS], and where new rules, standards and best practice guidance may be needed.’

11

The ‘loop’ here refers to the ongoing process of issuing a command to a weapon, receiving information about the effect(s) of that command, formulating a new command based on that new information, issuing the new command and so on: see, eg, G de Boisboissel, ‘Uses of Lethal Autonomous Weapon Systems’ (IEEE International Conference on Military Technologies, Brno May 2015).

12

Protocol Additional to the Geneva Conventions of 12 August 1949, and Relating to the Protection of Victims of International Armed Conflicts (Protocol I) (adopted 8 June 1977, entered into force 7 December 1978) 1125 UNTS 3 (API).

13

For a different approach to this question, but not altogether different conclusion, see ET Jensen, ‘The (Erroneous) Requirement for Human Judgment (and Error) in the Law of Armed Conflict’ (2020) 96 International Law Studies 26.

14

Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems, ‘Provisional Agenda for 2021 Sessions’ (14 July 2021) CCW/GGE.1/2021/1 [5](a).

15

See, eg, Campaign to Stop Killer Robots, ‘Recommendations on the Normative and Operational Framework for Autonomous Weapon Systems’ (Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems, Geneva, December 2021) <https://documents.unoda.org/wp-content/uploads/2021/06/Campaign-to-Stop-Killer-Robots.pdf> accessed 4 May 2022 3B.

16

See, eg, S-S Hua, ‘Machine Learning Weapons and International Humanitarian Law: Rethinking Meaningful Human Control’ (2019) 51 Georgetown Journal of International Law 117; Chengeta (n 8); Human Rights Watch, Losing Humanity: The Case Against Killer Robots (Human Rights Watch, 2012).

17

See, eg, R Crootof, ‘The Killer Robots Are Here: Legal and Policy Implications’ (2015) 36 Cardozo Law Review 1837; C Jenks, ‘False Rubicons, Moral Panic, & Conceptual Cul-De-Sacs: Critiquing & Reframing the Call to Ban Lethal Autonomous Weapons’ (2016) 44(1) Pepperdine Law Review 1; MN Schmitt, ‘Autonomous Weapon Systems and International Humanitarian Law: A Reply to the Critics’ (2013) Harvard National Security Journal Features 4.

18

For a deeper discussion of this point see: T McFarland, Autonomous Weapon Systems and the Law of Armed Conflict: Compatibility with International Humanitarian Law (Cambridge 2020) ch 3.

19

For an overview of the various aspects of control attributed to ‘meaningful human control’, see M Ekelhof, ‘Moving beyond Semantics on Autonomous Weapons: Meaningful Human Control in Operation’ (2019) 10(3) Global Policy 343.

20

Republic of Costa Rica, the Republic of Panama, the Republic of Peru, the Republic of the Philippines, the Republic of Sierra Leone and the Eastern Republic of Uruguay, ‘Joint Working Paper’ (Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems, Geneva, December 2021) <https://documents.unoda.org/wp-content/uploads/2021/06/Costa-Rica-Panama-Peru-the-Philippines-Sierra-Leone-and-Uruguay.pdf> accessed 4 May 2022; ‘Commentary of the Republic of the Philippines on the Normative and Operational Framework in Emerging Technologies in the Area of Lethal Autonomous Weapon Systems’ (Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems, Geneva, December 2021) <https://documents.unoda.org/wp-content/uploads/2021/06/Philippines-.pdf> accessed 4 May 2022.

21

Campaign to Stop Killer Robots (n 15).

22

See, eg, the summary in Section 4 of this article of Australia’s contribution to the 2019 GGE meeting for a description of one such system.

23

Possibly, the clearest example of this disagreement is the divide between those States and organisations which favour promulgation of new, legally binding rules to govern the uptake of AWS and those which assert that existing law is adequate. Based on statements made at the 2021 GGE meetings on regulation of AWS, the Philippines, the Non-Aligned Movement and the ICRC, among others, appear to belong to the former group while Russia, the UK and the USA, also among others, belong to the latter. See the statements and working papers submitted by participants at: ‘Convention on Certain Conventional Weapons–Group of Governmental Experts on Lethal Autonomous Weapons Systems’ (UNODA Meetings Place) <https://meetings.unoda.org/section/ccw-gge-2021_documents_14090_documents_14570/> accessed 4 May 2022.

24

Convention (VIII) relative to the Laying of Automatic Submarine Contact Mines (adopted 18 October 1907, entered into force on 26 January 1910) [1910] ATS 8 arts 1, 3.

25

Convention on the Prohibition of the Use, Stockpiling, Production and Transfer of Anti-Personnel Mines and on Their Destruction (adopted 18 September 1997, entered into force on 1 March 1999) 2056 UNTS 211 art 2.

26

Protocol on Prohibitions or Restrictions on the Use of Mines, Booby-Traps and Other Devices as amended on 3 May 1996 (Protocol II, as amended on 3 May 1996) annexed to the Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons which may be deemed to be Excessively Injurious or to have Indiscriminate Effects (adopted 3 May 1996, entered into force 3 December 1998) 2048 UNTS 93 art 2.

27

Guiding Principles, principle (a).

28

API art 51(4)(b) and (c).

29

This claim is based on the observation that many unclassified development programmes and patents relating to highly autonomous weapon systems indicate that those systems will rely heavily on AI technologies in order to undertake the complex tasks for which they are being designed.

30

For a general discussion of this problem, see J-C Pomerol, ‘Artificial Intelligence and Human Decision Making’ (1997) 99(1) European Journal of Operational Research 3.

31

AL Schuller, ‘Artificial Intelligence Effecting Human Decisions to Kill: The Challenge of Linking Numerically Quantifiable Goals to IHL Compliance’ (2019) 15 I/S: A Journal of Law and Policy for the Information Society 110.

32

See API arts 51(5)(b), 57(2)(a)(iii).

33

Schuller (n 31) 109.

34

Human Rights Watch, Losing Humanity: The Case Against Killer Robots (n 16) 31–32.

35

API art 36.

36

API art 57(2)(a)(ii).

37

This claim is subject to concerns about risk management discussed in Section 5 of this article.

38

See, eg, R Geiss, ‘The International-Law Dimension of Autonomous Weapons Systems’ (Study, Friedrich-Ebert-Stiftung, International Policy Analysis, October 2015) 17; A Pop, ‘Autonomous Weapon Systems: A Threat to Human Dignity?’ (ICRC Humanitarian Law and Policy, 10 April 2018) <https://blogs.icrc.org/law-and-policy/2018/04/10/autonomous-weapon-systems-a-threat-to-human-dignity/> accessed 4 May 2022.

39

See, eg, R Sparrow, ‘EIA Interview with Robert Sparrow on Autonomous Weapon Systems and Respect in Warfare’ (Ethics & International Affairs, August 2016) <https://www.ethicsandinternationalaffairs.org/2016/eia-interview-robert-sparrow-autonomous-weapon-systems-respect-warfare/> accessed 4 May 2022.

40

This obligation is formalized and expanded upon in Common Article 1 to the four 1949 Geneva Conventions: ‘The High Contracting Parties undertake to respect and to ensure respect for the present Convention in all circumstances.’

41

Vienna Convention on the Law of Treaties (adopted on 23 May 1969, entered into force 27 January 1980) 1155 UNTS 331.

42

See, eg, the many references to commanders’ responsibility for taking precautions in attacks in the military manuals cited in J-M Henckaerts and L Doswald-Beck (eds), Customary International Humanitarian Law (Cambridge University Press 2005) vol 2, ch 5.

43

Although views differ on which ranks may qualify. For example, Switzerland entered a reservation to art 57 of API stating that ‘[t]he provisions of paragraph 2 of this article create obligations only for commanders from the battalion or group level up and at higher levels.’ (n 12) 431.

44

I Henderson, The Contemporary Law of Targeting (Martinus Nijhoff 2009) 160.

45

M Sassòli, ‘State Responsibility for Violations of International Humanitarian Law’ (2002) 84 International Review of the Red Cross 405.

46

J-M Henckaerts and L Doswald-Beck (eds), Customary International Humanitarian Law (Cambridge University Press 2005) vol 1, 530.

47

B Zimmermann, ‘Protocol I – Article 1 - General Principles and Scope of Application’ in Y Sandoz, C Swinarski and B Zimmermann (eds), Commentary on the Additional Protocols of 8 June 1977 to the Geneva Conventions of 12 August 1949 (Martinus Nijhoff 1987) 35 [38]–[40].

48

Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems, ‘Australia’s System of Control and Applications for Autonomous Weapon Systems’ (26 March 2019) CCW/GGE.1/2019/WP.2/Rev.1 [3].

49

ibid [7]–[40].

50

The text of that Article being: ‘With respect to attacks, the following precautions shall be taken:…an attack shall be cancelled or suspended if it becomes apparent that the objective is not a military one or is subject to special protection or that the attack may be expected to cause incidental loss of civilian life, injury to civilians, damage to civilian objects, or a combination thereof, which would be excessive in relation to the concrete and direct military advantage anticipated.’

51

C Pilloud and J Pictet, ‘Protocol I–Article 57–Precautions in Attack’ in Y Sandoz, C Swinarski and B Zimmermann (eds), Commentary on the Additional Protocols of 8 June 1977 to the Geneva Conventions of 12 August 1949 (Martinus Nijhoff 1987) 686 [2221].

52

Such as those contained in CCW Protocol II (and, indeed, the outright prohibitions contained in the Ottawa Convention, to the extent that the harmful effects which gave rise to those prohibitions stem from an inability to intervene in the operation of an anti-personnel mine throughout its life).

53

Or, indeed, reducing the chance of such an outcome: ET Jensen and A Hickey, ‘Leveraging Emerging Technology for LOAC Compliance’ in RTP Alcala and ET Jensen (eds), The Impact of Emerging Technologies on the Law of Armed Conflict (Oxford University Press 2019) 45 ; P Margulies, ‘The Other Side of Autonomous Weapons’ in RTP Alcala and ET Jensen (eds), The Impact of Emerging Technologies on the Law of Armed Conflict (Oxford University Press 2019) 147.

54

Accordingly, military decision support is an active field of research: see, eg, K van den Bosch and A Bronkhorst, ‘Human-AI Cooperation to Benefit Military Decision Making’ (NATO 2018) <https://www.karelvandenbosch.nl/documents/2018_Bosch_etal_NATO-IST160_Human-AI_Cooperation_in_Military_Decision_Making.pdf> accessed 4 May 2022.

55

See, eg, Boulanin and others (n 10) 21–22.

56

R Liivoja, M Naagel and A Väljataga, ‘Autonomous Cyber Capabilities under International Law’ (2019) NATO Cooperative Cyber Defence Centre of Excellence Working Paper <https://ccdcoe.org/library/publications/autonomous-cyber-capabilities-under-international-law/> accessed 4 May 2022.

57

Pilloud and Pictet (n 51) 681 [2198].

58

Boulanin and others (n 10) 7.

59

Parts of the discussion of risk factors in the following sub-sections draw on equivalent points made in T McFarland, ‘The Status of Autonomous Weapon Systems under International Humanitarian Law’ (PhD thesis, The University of Melbourne 2017).

60

MC Horowitz and P Scharre, ‘Meaningful Human Control in Weapon Systems: A Primer’ (Center for a New American Security March 2015) 8.

61

For a brief outline, see CQ Choi, ‘7 Revealing Ways AIs Fail: Neural Networks can be Disastrously Brittle, Forgetful, and Surprisingly Bad at Math’ (2021) (October) IEEE Spectrum 42.

62

P Scharre, ‘Autonomous Weapons and Operational Risk’ (Center for a New American Security, Ethical Autonomy Project, February 2016) 6 <https://www.cnas.org/publications/reports/autonomous-weapons-and-operational-risk> accessed 4 May 2022.

63

PJ Smith, CE McCoy and C Layton, ‘Brittleness in the Design of Cooperative Problem-Solving Systems: The Effects on User Performance’ (1997) 27(3) IEEE Transactions on Systems, Man and Cybernetics 360.

64

For a general discussion of risk and AWS reliability, see R Cordeschi, ‘Automatic Decision-Making and Reliability in Robotic Systems: Some Implications in the Case of Robot Weapons’ (2013) 28 AI & Society 431.

65

Scharre (n 62) 13.

66

Scharre (n 62) 13, fn 13.

67

McFarland (n 59) 101.

© Oxford University Press 2022.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.

© Oxford University Press 2022.

Introduction