Lethal autonomous weapon systems (LAWS) are defined as robotic weapons that have the ability to sense and act unilaterally depending on how they are programmed. Such human-out-of-the-loop weapon platforms will be capable of selecting targets and delivering lethality without any human interaction.¹1. See The Economist, ‘Man and Machine’, 25 January 2017.View all notes Both semi-autonomous and other precursor systems are already in service, buoyed by several drivers to a move from merely automatic weapons to fully autonomous ones, able to engage a target based solely on algorithm-based decision-making. LAWS require a material step change in both hardware and software but, once deployed, will mark a significant shift in how humans wage war (notwithstanding the technical, legal, moral and ethical challenges that their introduction entails). This article considers behavioural and leadership challenges as well as policy implications that arise from the deployment of such weapons, and how unsupervised engagements might degrade the commander’s craft.²2. Patrick William Walker, ‘War Without Oversight; Challenges to the Deployment of Autonomous Weapons’, PhD thesis, University of Buckingham, July 2019, p. 5, <https://killerrobots.info/wp-content/uploads/2019/08/war-without-oversight-august-2019-phd-distribution-version.pdf>, accessed 12 May 2020.View all notes

In what this article terms the ‘Delivery Cohort’, levels of command and other relevant parties that together comprise LAWS leadership are amalgamated into a single entity. This is a broad grouping intended to reflect what will, after all, be a distributed deployment responsibility. The Cohort encompasses politicians, generals, local commanders and soldiers on the ground. It also includes maintenance and service personnel, those in manufacturing and procurement, in design and programming, those heading commercial entities, the regulators and lawyers, the press and those from the third sector. Leadership is also considered from the perspective of parties that are local to the use of such weapons and, in order to spotlight the more important themes, the article’s focus is on the deployment of broad-use, wide-capability assets.

The Drivocopter at the 2020 Defense Advanced Research Projects Agency Subterranean Urban Challenge. The challenge aims to improve the decision-making autonomy of robots. Courtesy of NASA/JPL-Caltech

Display full size

Several credible drivers already exist to encourage commanders to embrace weapon autonomy, including political expediency in removing soldiers from frontline harm, the promise of precise lethality and advantages such as increasing operational speed, remote delivery and force multiplication.³3. Paul Scharre, ‘The Coming Swarm: Robotics on the Battlefield’, Real Clear Defense, 19 October 2014. See also Khasha Ghaffarzadeh, ‘New Robotics and Drones 2018-2038: Technologies, Forecasts, Players’, IDTechEX, 2018, <https://www.idtechex.com/research/reports/new-robotics-and-drones-2018-2038-technologies-forecasts-players-000584.asp>, accessed 20 January 2019.View all notes Fielding unmanned and loitering weapon systems in theatres such as Nagorno-Karabakh has recently demonstrated the disruptive effect such munitions can have on enemy forces.⁴4. See John Amble, ‘MWI Podcast: The Conflict in Nagorno-Karabakh Is Giving Us a Glimpse into the Future of War’, 14 October 2020, Modern War Institute, <https://mwi.usma.edu/mwi-podcast-the-conflict-in-nagorno-karabakh-is-giving-us-a-glimpse-into-the-future-of-war/>, accessed 25 February 2021.View all notes Indeed, unsupervised and remote weapon systems are increasingly becoming the technical backbone to states’ anti-access/area denial (A2/AD) operations, as well as the means to enable local activities such as jamming of the electromagnetic spectrum.⁵5. The Economist, ‘Stay Well Back’, 21 January 2018.View all notes Together, these deployment characteristics have encouraged what may be termed a ‘revolution in expectation’ whereby disproportionate and perhaps unfounded assurance is attached to such new methods. This article therefore aims to ensure that leaders are aware of such arguments. This is particularly important given that unsupervised weapons cannot themselves make legal determinations, meaning it remains the responsibility of the LAWS Delivery Cohort to ensure the legal use of such assets.

What Constitutes Military Leadership in LAWS Deployment?

What, then, constitutes leadership in this context? One workable proxy is provided by what would become known as the 11 principles of leadership, first developed in 1948,⁶6. Department of the Army, Training Circular No. 6, 19 July 1948.View all notes and subsequently published in the US Army’s Field Manual on leadership.⁷7. US Army, ‘11 Timeless Principles of Leadership’, US Army Field Manual, 1951.View all notes Sixty years later, its principles are still used across the US armed forces. Its canon is surprisingly monotonal, universally behavioural and human, and circles back again and again almost exclusively to context. As a corpus, it seeks to define the changeable art of motivating a group of individuals to act towards achieving a common goal. The leadership paradigm that it outlines (and against which LAWS deployment must be judged) is therefore an abstract. Leadership is unstructured. It depends on context and defies prescriptive description.

The principles’ repeated notion that leaders should ‘seek [to] take responsibility for their own actions’ evidences this irreducibility. Leaders, for instance, ‘must lead by example’ if they are to expect courage, responsibility, initiative, competence, commitment and integrity from their subordinates.⁸8. Ibid.View all notes If the tenets of leadership cannot be replicated in code-ready packets, the Cohort must factor how weapon autonomy changes existing norms and the policy implications of any such transition. How a state then manages its forces’ manning becomes a case in point: are computer skills suddenly more applicable than an ability to run up a hill? Indeed, it may be that LAWS must outperform human traits such as bravery, initiative and example if autonomous weapons are to be embedded in the Cohort’s battlecraft. A secondary point then becomes the amount of dilution in legal compliance that might be tolerated by states deploying LAWS. An example is the use of LAWS for access denial. An area might be signposted as out of bounds with warnings that any incursion will be met by lethal and autonomous force without human oversight, constituting an excuse for the autonomous weapon to engage without recourse to the laws of armed combat (LOAC) and Article 57 on the protection of civilians. Each of these issues requires delicate and human leadership. A commander’s decision-making, after all, is knowing if to decide and then when and what to decide, based on a clear articulation of the commander’s battlefield understanding.⁹9. Headquarters, Department of the Army, ‘Staff Organization and Operations’, FM 101-5, May 1997, Chapter 5, p. 5-1. See also Pablo Kalmanovitz, ‘Judgement, Liability and the Risks of Riskless Warfare’, in Nehal Bhuta et al. (eds), Autonomous Weapons Systems: Law, Ethics, Policy (Cambridge: Cambridge University Press, 2016), pp. 161–62.View all notes The complexity of this task in the currently manned environment is already evidenced by the required tools for the process, including standard operating procedures, exhaustive field manuals and granular rules of engagement. Unsupervised combat will require precisely similar processes.

LAWS Behavioural Issues and the Role of the Combatant

Fielding independent weapons therefore challenges traditional behavioural notions of what it is to be a combatant. As noted by Christian Enemark, the longbow was outrageous when it first emerged because of the new degree of force that it brought to the battlefield, but also because the lowly peasant could now defeat the aristocrat.¹⁰10. Christian Enemark, Armed Drones and the Ethics of War: Military Virtue in a Post-Heroic Age (Abingdon: Routledge, 2014), p. 99.View all notes In this vein, the drone operator already kills without any material personal risk. For leaders, autonomous weapons promise an even higher degree of user insulation, prompting a question for the Delivery Cohort of where LAWS accountability rests. Enemark terms the notion ‘post heroic warfare’, questioning the traditional construct of states prepared to wage war on the basis of substantial casualties. With the shift to unsupervised engagement, the paradigm of noble death disappears, while a new norm arises based on the practice of remote warfare, with few combatants and fewer casualties. While the technology is not sufficiently mature for all of these characteristics to be immediately deliverable to the Cohort, there has been enough development in currently available systems to influence current norms of warfare and the policies that underpin them.

The deployment of LAWS also has an impact on how leaders might lead. For those in battle, several attributes have long characterised combat soldiers across cultures and time: endurance; courage; strength; skill; and honour. Riskless war dilutes such requisites of valour – there is no longer a need for the leader to be admired, respected, cheered, even mourned. Removing human oversight must therefore blunt traditional notions of duty and self-sacrifice, the covenants binding soldiers together and to the society that they serve, as well as the ageless notion of subjugating self-preservation in the present to make life better in the future. Riskless warfare – which will progress from the unmanned drone to the unsupervised weapon – dilutes the notion of a soldier’s risk-based contract to kill.

LAWS Deployment Models

Incremental replacement of a weapon’s supervised componentry by autonomous agents requires that the local commander understand the boundaries to his or her control and where jurisdiction has actually been ceded to a machine agent. This includes the deployment of a defensive unmanned sentry gun or automatic turrets installed on an otherwise regular military vehicle or sophisticated offensive weaponry such as loitering artillery and other independent A2/AD assets.¹¹11. Kelsey Piper, ‘Death by Algorithm: The Age of Killer Robots Is Closer Than You Think’, Vox, 21 June 2019.View all notes Another example might include machine–human teaming where the experience of human soldiers is then leveraged by companion autonomous technologies. Such ‘hybrid autonomy’ requires hand-off and management of command between human and machine, and presents a whole new set of leadership and policy issues.¹²12. Human Rights Watch and International Human Rights Clinic, ‘Fully Autonomous Weapons: Questions and Answers’, October 2013, p. 2, <https://www.hrw.org/sites/default/files/related_material/10.2013%20Killer%20Robots%20QA.pdf>, accessed 25 February 2021.View all notes Research, after all, points to erratic performance when humans are required to intervene in moments of high stress or in situations of limited information or situational awareness.¹³13. Mark A Staal, ‘Stress, Cognition and Human Performance: A Literature Review and Conceptual Framework’, NASA/TM—2004–212824, NASA STI Programme, August 2004. See also R D Campbell and M Bagshaw, Human Performance and Limitations in Aviation, Revised Edition (Oxford: Blackwell, 2008); Gyanesh Kumar Tiwari, ‘Stress and Human Performance’, Indo-Indian Journal of Social Science Research (Vol. 7, No. 1, 2011), pp. 40–49.View all notes Moreover, the human has hitherto been operator, moral agent and failsafe across an engagement’s whole decision tree. For instance, it is currently the human operator who manages task complexity, its cognitive workload and its number, type and duration.¹⁴14. Paul Scharre, ‘Autonomous Weapons and Operational Risk’, Center for a New American Security, February 2016.View all notes The command issue here is how this equation will now be managed given the remote yet very dynamic state of any such hybrid deployment model. Tasking in manned operations is already muddied by matters of scale, margins for error, task space and colleague assets. As weapon autonomy is introduced, slack time across all battle processes is crucially reduced, scope for rule-bending and initiative is removed and, in the case of weapons based on machine learning (ML), the leader’s ability to predict and influence outcomes is materially lessened.

The Cohort must also navigate the constraining factors arising from international humanitarian law (IHL). It has long been established that all persons who plan, decide on and carry out attacks must comply with rules on distinction, proportionality and precautions-in-attack. Such assessments, however, involve evaluative and contextual judgement for which the local commander remains responsible and accountable. Sufficient human control must therefore be retained to enable situation-specific judgement to comply with those rules.¹⁵15. Maya Brehm, Defending the Boundary: Constraints and Requirements on the Use of Autonomous Weapon Systems Under International Humanitarian and Human Rights Law, Academy Briefing No. 9 (Geneva: Geneva Academy of International Humanitarian Law and Human Rights, 2017).View all notes Commanders must be confident that their unsupervised assets direct attacks only at military objectives which contribute to military action. Similarly, incidental civilian damage should be proportional to the advantage anticipated. IHL also requires that the commander has specific knowledge of the context and circumstances prevailing at the time of an attack prior to authorising that attack. This includes each target’s specifics within the context of such weapons’ overall effects. The commander’s resulting requirement for minimum levels of predictability highlights the importance of appropriate testing and evaluation as well as LAWS transparent validation and verification, especially in instances of adversarial interventions and data poisoning. In this case, enhanced camouflage or an unexpected palette may confound the ‘dark box with a barrel’ search criterion determining target selection for an autonomous anti-tank weapon.

Technical Challenges

An autonomous weapon must work the first time, every time. It must do this to be reliable, fulfil the expectations of its Delivery Cohort, adhere to legal frameworks and meet ethical standards.¹⁶16. Meghan Han, ‘Lethal Autonomous Weapons & Info-Wars: A Scientist’s Warning’, Medium, 6 July 2017.View all notes To justify its deployment, the Cohort must therefore be confident that the LAWS will perform prescribed tasks and undertake them better (whether that be cheaper, quicker, more consistently or more surgically) than available manned alternatives. Commanders, after all, have deployment and allocation choices. It is for this reason that much of LAWS’ efficacy boils down to each weapon’s technical basis and its fitness for purpose.

What, then, are the leadership ramifications from LAWS’ technical spine? ML’s intended role is to enable the weapon to undertake onboard deduction and independent prediction whereby whatever the weapon experiences should inform its future expectations. The intention is that datasets of dark coloured boxes with barrels that comprise the weapon’s training will enable it to recognise adversarial tanks and select targets. In the case of current supervised learning techniques, the objective is to process the parameters of what is a known case whose relationships can then be carried over to the present case. The architectural basis of LAWS is that artificial neural networks learn by example. This, however, requires that several capabilities work seamlessly, including pattern recognition, label matching, data classification and all of the associated training routines that must comprise the conversion of sensed inputs into weapon actions. In essence, should a battlefield signal (data polled by the LAWS’ several sensors) exceed a threshold (as defined in the Cohort’s configuration phase), then it is propagated forward to other neurons to instigate a deliberate action. This is a fundamentally fragile set of arrangements, and failure in any piece is likely to distort weapon outputs.

The deployment of LAWS will require the application and management of prioritising weights to that data in order to encourage the weapon’s intended behaviour. Without such incentives, its neural network will be unable to react to newly sensed inputs drawn from the weapon’s immediate environment. For this to work, however, received data must match that with which the weapon has previously been trained or that comprises its initial representation (‘prior knowledge’). Notwithstanding wide variations in likely deployment characteristics, the network’s prior training must also fit the state space that comprises the weapon’s disposition at any point in time. Issues arising from this characteristic are usefully captured by the acronym CACE (change anything and you change everything). Research demonstrates that the introduction of new parameters or slightly heterogenous data to that data under which the weapon has been trained will confound LAWS’ ML processes for the tasks envisaged, particularly in the changing nature of a battlespace with its prevalence of hidden, partially observable or camouflaged states.¹⁷17. Plamen Angelov and Alessandro Sperduti, ‘Challenges in Deep Learning’, in ESANN 2016 Proceedings, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (Bruges: ESANN, 2016), pp. 489–91. See also Soroush Nasiriany et al., ‘A Comprehensive Guide to Machine Learning’, University of California at Berkeley, 18 November 2019, pp. 82–88, <http://snasiriany.me/files/ml-book.pdf>, accessed 19 June 2020.View all notes

LAWS’ processing will be governed by an error function, and the goal of training is to minimise this. Commanders should note that several challenges arise from this systemic position of ML within LAWS application. The number of datapoints comprising an engagement sequence will, for instance, be intractably large. Such a model also relegates weapon learning to the role of an approximator and, as such, is likely inappropriate for compliance with LOAC where commanders require precise responses to specific battlefield happenings. Research also suggests that ML performance plateaus in line with the frequency that the autonomous agent polls its sensors, making an ever-smaller change to the model’s weights with every new iteration.¹⁸18. Xavier Girot and Yoshua Bengio, ‘Understanding the Difficulty of Training Deep Feedforward Neural Networks’, in Yee Whye Teh and Mike Titterington (eds), ‘Volume 9: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, 2010’, p. 249. See also Charles Perrow, Normal Accidents: Living With High-Risk Technologies (Princeton, NJ: Princeton University Press, 1999), p. 354.View all notes There is no current solution to this fall-off. Adding new training parameters to broaden the LAWS’ data capture instead results in extra layers of non-linearity, further complicating weapon use and challenging subsequent optimisation of its learning routines. A marginally different setup, discrepancies between training dataset and sensed information or data poisoning arising from adversarial intervention may all lead to very substantial variation in that weapon’s decision-making. Moreover, ML will likely ignore sensed features which comprise only a small number of examples in a training set but, in the passage of a lethal engagement, ones that possibly account for features of critical battlefield importance.

It is also intrinsically challenging for the LAWS’ sensors, the likely sole sources of input for its decision processes, to gather consistent information. Smoke, reflectance, image echo and issues of data intensity and saturation all comprise possibly intractable difficulties. Similarly, class boundaries separating different data examples resist definition where that data is partial, noisy or indistinct, requiring human intervention if the weapon is to designate data strings for further statistical analysis. Unless the unsupervised weapon can reliably parse sensed data in a manner that allows, for example, a tank to be identified as an appropriate target, the weapon is systemically unable to complete the analysis and decision process that must precede engagement. Data mismatch against its training set, anything that is statistically out of the ordinary, will confound unsupervised processes, whether this is the result of simple feint, enemy surprise or inadequate data separation. Leaders should therefore be focused on weapon sensitivity (its detection rate) and specificity (here, its false alarm rate), and understand that there is a non-linear tradeoff between the two.

LAWS are therefore reliant on human configuration so that they can be ‘trained’ rather than simply programmed. Leo Tolstoy notes in War and Peace that the real general ‘is never dealing with the beginning of any event’.¹⁹19. Leo Tolstoy, War and Peace, Volume 1, translated by Louise Maude and Aylmer Maude (London: Collector’s Library, 2004), p. 20. Emphasis in original.View all notes Instead, the commander is perpetually situated in the middle of a series of events, each a link in what becomes a chain of causation.²⁰20. Joshua Rothman, ‘The Art of Decision-Making’, New Yorker, 14 January 2019.View all notes The analogy is useful as it reinforces the enduring infeasibility of trying to hard-code rules of engagement. Matching weapon instructions to the complexities of battle requires programmers to engineer unrealistic start and end points in order to code appropriate commands. But the inescapable practicality for local commanders is that each coding choice must invariably be constrained by earlier choices. The policy implication for LAWS deployment is that unless weapon tasking is highly restricted, critical pathways will remain undiscovered, ignored or misunderstood. Absent human oversight, these weapons’ decision processes instead become an overly simple value-maximising exercise that is unacceptably fragile in situations of uncertainty, when priorities change or when instructions require reconfiguration.²¹21. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014), p. 17.View all notes

Cognitive Considerations to LAWS Deployment

A further leadership quandary is that the unsupervised weapon cannot be sure it has found its best actions until it has tried all possible actions in all possible states.²²22. Michael C Horowitz, ‘The Promise and Peril of Military Applications of Artificial Intelligence’, Bulletin of the Atomic Scientists, 23 April 2018.View all notes The requirement that this posits is infeasibly large and commanders risk fielding machines that freeze or behave irrationally. Current ML models are empirically too iterative to manage the parameter-rich procedure that is, for instance, target selection, and they are even more so without any definable end-state. At the very least, this points to command and policy requirements that such weapons be deployed sparingly and meticulously, at least in the medium term, so that the Cohort can first understand and then manage such complexity. Moreover, error in the weapon’s sensing of its current state will presumably carry forward in its future learning and battlefield actions. Similarly, much of the weapon’s learning may be invalid if its environment or combat task changes – a tradeoff between a weapon that is ‘constantly learning’ as opposed to one that is using what is already known to work at the cost of missing out on further improvement.²³23. Jonathan D Cohen, Samuel M McClure and Angela J Yu, ‘Should I Stay or Should I Go? How the Human Brain Manages the Trade-Off Between Exploitation and Exploration’, Philosophical Transactions of the Royal Society B (Vol. 362, No. 1481, May 2007), pp. 933–42.View all notes Indeed, the Cohort must accept that the challenge of incorporating ‘unlearning’ into machine routines remains untested and points to a systemic inefficiency in the preparation of that weapon’s input data. As much as two-thirds of data analysis is currently taken up with data preparation, an inappropriate luxury on a quick-changing battlefield if the weapon is to be valuable yet still compliant.²⁴24. Adrienne LaFrance, ‘Machine Unlearning’, The Atlantic, 18 March 2016.View all notes

What, therefore, should the local commander demand from the cognitive functions of a weapon without meaningful human control? The intended model is that the autonomous weapon must deliver on many processes, including knowledge and memory management, attention and evaluation, and problem-solving and decision-making. Without just one of these elements, is the weapon sufficiently robust to replace the human soldier? The difficulty is that cognition, even in its human condition, is challenging to define. It can be conscious or unconscious, concrete or abstract.²⁵25. See Thomas G Dietterich and Ryszard S Michalski, ‘A Comparative Review of Selected Methods for Learning from Examples’, in Ryszard S Michalski, Jaime G Carbonell and Tom M Mitchell (eds), Machine Learning: An Artificial Intelligence Approach (Wellsboro, PA: Tioga Publishing, 1983), pp. 41, 43–45.View all notes It can also be intuitive (the ‘knowledge’ of a task) or conceptual (the ‘model’ of a task). The deployment construct for LAWS is that cognitive processes must use existing knowledge to generate new knowledge. The challenge for the Cohort, however, is that weapon datasets must be incrementally built as not all data is available to the weapon at the same time. Information invariably arrives in an ad hoc fashion, and any wide-capability LAWS model cannot assume a frictionless inflow of the relevant data necessary to create these datasets. The deployment implication here is that LAWS’ processes will first require robust means to manage data denial, notwithstanding that little transferable research is currently evident on these central competencies.²⁶26. Sue Unerman, ‘Don’t be an Armchair General’, Campaign, <https://www.campaignlive.co.uk/article/dont-armchair-general/1661842>, accessed 8 April 2021; Kamakshi Lakshminarayan et al., ‘Imputation of Missing Data Using Machine Learning Techniques’, in Evangelos Simoudis, Jiawei Han and Usama Fayyad (eds), KDD’96: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (Palo Alto, CA: AAAI Press, 1996); Tobias Cagala, ‘Improving Data Quality and Closing Data Gaps with Machine Learning’, 5 May 2017, <https://www.bis.org/ifc/events/ifc_nbb_workshop/ifc_nbb_workshop_2d3.pdf>, accessed 8 April 2021.View all notes

Leaders should also recognise that the simplification methods which characterise human decision-making are not a suitable starting point for the programming of independent weapons. An analysis of these traits is useful given that leaders must consider LAWS’ relative performance to existing weapon effects and methods before authorising deployment. As currently envisaged, LAWS’ operations attempt to mirror human experience through training on very large datasets. When faced with new circumstances, a soldier on the ground naturally compares those circumstances to similar situations previously encountered and now lodged in his or her memory. Resulting decisions are rarely the product of thoughtful deliberation, especially in a time-constrained environment. Humans instead rely on shortcutting heuristics that may not simply transfer from manned to unmanned functions. As uncertainty arises, whether from enemy action or feint, the independent weapon must similarly sift for the widest relevant intelligence. It is enduringly challenging for the Cohort to replicate human imagination whereby soldiers can backfill for information gaps in order to arrive at what becomes a subjective premonition to compensate for incomplete data.

In considering deployment, the Cohort must also factor for other heuristics.²⁷27. Amos Tversky and Daniel Kahneman, ‘Judgement Under Uncertainty: Heuristics and Biases’, Science (Vol. 185, No. 4157, September 1974).View all notes Leaders should aim to avoid confirmation bias whereby autonomous agents erroneously validate datalinks for further processing or, more likely, apportion inappropriate weight to fitting sensed information with the weapon’s original dataset. Does the weapon give more or less weight to a natural pothole or an artificial obstacle in dictating journey planning? Indeed, it is systemically difficult for LAWS to classify an event or object into a particular category. Its processes will instead review and dissect it according to an inappropriately small number of characteristics, which empirically represent much larger groupings of pre-existing occurrences with the weapon actioning sequences according to that narrow class of experiences. Termed ‘representativeness’, the point for the Cohort is that the characteristic hides other potential biases including insensitivity to prior probability of outcomes and sample size, base-rate neglect and misconceptions of chance.²⁸28. Blair S Williams, ‘Heuristics and Biases in Military Decision Making’, Military Review (September/October 2010), pp. 46–48.View all notes

Coding Constraints to LAWS Deployment

The decision to deploy LAWS is also complicated by weapons’ increasing number of self-governing sub-parts, each of which is capable of independent agency.²⁹29. Christopher Coker, ‘Still “the Human Thing”? Technology, Human Agency and the Future of War’, International Relations (Vol. 32, No. 1, March 2018), pp. 23–38.View all notes Such composite agency is difficult precisely because action selection now depends not only on the impulses of its constituent sub-agents, but also on how these are organised and coded. It is coding, after all, that must express all the intentions and constraints of the Delivery Cohort. Only through coding can an independent weapon’s goals and action selection be effected. The challenge here is that those frequent (possibly continuous) conviction weightings required to prioritise the weapon’s sensed data have unintended ramifications, including data smoothing whereby data (such as information on an engagement sequence, target signature, classification, location, threat and sensitivity to battlefield clutter) becomes inappropriately scaled to the mean to the extent of formlessness. Sudden exogenous events that characterise battlecraft are now prone to be ironed out of the unsupervised weapon’s calculations.

The coding practice for LAWS will be that the weapon continually computes new probabilities for its immediate world. For the Cohort and commanders deploying such platforms, it is important to note that this accelerates data obsolescence causing LAWS to be systemically unstable. An example here is, once again, seen in LAWS’ movement and the fundamental requirement that the weapon’s relevant navigable space must first be identified, processed and made ‘map-ready’ for each representation.³⁰30. For a useful discussion on decision waterfalls for autonomous vehicle motion, see Brian Paden et al., ‘A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles’, arXiv preprint, arXiv:1604.07446v1, 25 April 2016, <https://arxiv.org/pdf/1604.07446.pdf>, accessed 20 March 2020.View all notes The resulting dataset must be searched in real time to evaluate available paths and, following this, the best path after which the weapon’s goals, values and action selection must all be revised to account for that newly selected path. This is a brittle (and enormous) foundation on which to build predictable weaponry.

Coding issues have other tricky command ramifications. Mapping the Cohort’s intention (if, indeed, this is possible to a standard that meets states’ obligations under LOAC) requires that sensed data be linked to a particular representation on one of the LAWS’ learning planes. The intractable difficulty is that each such linkage must be coded so that one associatively connected entity can evoke another. In this way, the meaning of the commander’s instruction is not restricted to just one association. The Cohort must also recognise that the ability of LAWS to capture abstracts will be similarly difficult. The information contained within a command must be coupled to previously given information as well as to information that is to follow. Commanders must note that nested structures and conditionals (which regularly characterise complex instructions) create similar syntactic issues. While it may be human practice to understand what has been directed without having to figure out the precise meaning of the words, this does not translate across in machine coding. Finally, to this point, command and analysis both use several categories of facts within their syntax. Instances here include indexical facts, normative facts, strong convictions, observations and hints, clarifications, reinforcements and basic ontological factual statements. For the commander, all of these sub-types inform human decision-making but must now be precisely recognised only in LAWS’ code. Such categorisations are volatile, changing unpredictably according to new intelligence, feedback and local input from weapon sensors.

The potentially deadly consequences of LAWS programming necessitate reliable on-platform arbitration, but any remote means of determining decisions is challenging. The first issue for the Cohort relates to the degree of step change (the increment of learning termed ‘anchoring’) that each follow-on process exerts on the weapon’s immediately prior set of beliefs. To what extent should the LAWS’ current set of representations be updated to account for newly sensed data? Returning to the example of the simple pothole, to what degree does the autonomous weapon amend future action sequences in light of sensing a pothole immediately to its front so early in its deployment? Should it expect a pattern to emerge and then action similar responses? A second challenge is that weapon actions must comprise the appropriate reaction to every relevant sensed stimulus. LAWS cannot offer intermittent or erratic performance where only specific sensed inputs lead to weapon outputs.

LAWS deployment is also compromised by data ambiguity,³¹31. Maksim Lapin, ‘Image Classification with Limited Training Data and Class Ambiguity’, dissertation, Saarland University, June 2017, p. 2, <http://scidok.sulb.uni-saarland.de/volltexte/2017/6909/pdf/lapin17phd.pdf>, accessed 23 March 2020.View all notes and the complexities of context. Action is not just about the dispatch and enactment of a message. There is typically a gulf between the spoken or written word and the intention behind that message. Here, the commander must factor that lexical ambiguities arise from omitted or imprecise script. Semantic ambiguities in orders and tasks concern interpretative uncertainty while pragmatic ambiguity disadvantages communicating parties with mildly different contextual bases. These challenges raise both coding and ethical issues for the commander and Cohort. For instance, are ‘occasional mistakes’ acceptable on the basis that unsupervised violence is as susceptible to ethical deficiency as human soldiers? And is a lower legal bar appropriate for machines with less lethality? The verso here is that there may be occasions when the hiatus offered by compliant LAWS (that need neither self-protection nor to rush action sequences) may prevent ethical transgressions that have characterised knee-jerk human behaviour in the chaos of battle. This ignores, however, difficulties arising from the simple firing sequence for such weapon instructions. Each routine may result in quite different outputs being triggered depending on the order in which command instructions are processed by the unsupervised weapon.

Uncertainty, moreover, will be everywhere: the weapon may travel erratically, enemy forces will dissemble and battlefield obstacles move unpredictably. It is here that leaders leverage human experience, judgement and intuition. Algorithms are systemically unable to match the human agent’s ability to make difficult decisions in a fast, frugal and explainable manner. At such higher levels of expertise, battlefield commanders do not even recognise that they are making decisions; rather, they are interacting with a changing environment and responding to patterns that they have long recognised in order to factor both situational awareness and an appreciation of context into intended outcomes.

Action Selection

Commanders need to understand what drives LAWS’ decision-making. At each step, the independent weapon is selecting an action with the highest expected utility. However, this utility model poses various challenges for the Cohort and has important policy implications, not least due to its high margin for error with the possibility of unacceptable high-regret outcomes. The utility function must also factor for what constitutes appropriate use of force, the involvement of colleague assets, consideration of next steps, a data audit ahead of an action sequence and post-event communication of each engagement. Commanders do this innately.

Similarly intractable is the Cohort’s requirement to set the objectives that will govern weapon priorities and subsequent action selection.³²32. Walker, ‘War Without Oversight’, pp. 236–46.View all notes Here, leaders need to distinguish between values and goals when deploying unsupervised assets capable of lethal engagement. Goals, for instance, will prompt the independent weapon to develop plans of action while values will enable it to assess the comparative merits of such plans. The policy implication here is that if this process is stunted or inappropriately undertaken, the weapon will either be illegal or useless. Goals concern what must be undertaken at once, what should be undertaken next, the resumption of a task that was previously discontinued and, more complex for the weapon, what actions should subsequently take place in order to capitalise on battlefield opportunities. Errors here may also have quite unforeseen battlefield consequences that include ‘infrastructure profusion’ where an independent weapon might unexpectedly allocate disproportionately large parts of its reachable resources into the service of some inappropriately set goal.

The pivot here for commanders is that the difference between a weapon’s current and desired state is its observed error, with the object of the LAWS’ action selection being to minimise that error.³³33. Douglas Allchin, ‘Error Types’, Perspectives on Science (Vol. 9, No. 1, March 2001), pp. 38–59.View all notes Two issues arise for leaders. First, the pace of this error correction is not obvious. It depends, for instance, on how often the error is computed by the weapon and how much correction is then made on each feedback loop. Feedback loops, moreover, are significantly less effective at modifying a weapon’s higher-level action selection such as prioritisation, coordination and collaboration. Second, LAWS must also be appropriately front-facing and determine its system state ahead of time. For this reason, independent weapon systems (and their deploying commanders) must be able to support parallelism – the complex ability to monitor and execute multiple actions at once – with the danger being that sequential processing would otherwise risk missing events that might be critical to a compliant and winning operation. Similarly, goals and behaviour models must also support a workable process of ‘data forgetting’ in order to mitigate information overflow or the retention of sub-optimal (or wrong) data sequences. For the Cohort, it is key that an unsupervised weapon should not, after all, generally forget acquired skills (the notion of ‘catastrophic forgetting’), a recognised limitation of neural network models.³⁴34. See James Kirkpatrick et al., ‘Overcoming Catastrophic Forgetting in Neural Networks’, PNAS (Vol. 114, No. 13, March 2017), pp. 3521–26; German I Parisi et al., ‘Continual Lifelong Learning in Neural Networks: A Review’, Neural Networks (Vol. 113, May 2019), pp. 54–71.View all notes

The ‘attention’ of LAWS is also an enduring leadership challenge.³⁵35. Helgi Páll Helgason et al., ‘Towards a General Attention Mechanism for Embedded Intelligent Systems’, International Journal of Computer Science and Artificial Intelligence (Vol. 4, No. 1, March 2014), pp. 1–7.View all notes The human brain appears free to choose what it looks at, listens to and thinks about. Human commanders can focus their attention as they please. This information access, however, must be managed in LAWS in order to avoid memory overflow and the notion of ‘contradictory neural cacophony’ whereby the machine is again either paralysed or manoeuvred into inappropriate action. The difficulty here is that the LAWS must therefore actively select both the source and quantity of its information, what to process and store, and what peripheral information to attenuate into which decision processes – the so-called ‘cocktail party effect’. Research confirms that this is difficult.³⁶36. Adelbert W Bronkhorst, ‘The Cocktail Party Phenomenon: A Review of Research on Speech Intelligibility in Multi-Talker Conditions’, Acta Acustica (Vol. 86, January 2000), pp. 117–28; and Asher Cohen, ‘Selective Attention’, Encyclopaedia of Cognitive Science (Holbroken, NJ: John Wiley and Sons, 2006).View all notes Similar to cognition, attention is divided into the voluntary and involuntary. The issue for the Cohort relates again to arbitration and an appropriate model to determine that one sensor input is preferred over others, the problem of settling input intensities. After all, two variables that may be useless by themselves can be useful together. Similarly, a single variable that is useless by itself can be instrumental with others. Nor can it be that the loudest signal is automatically the one on which the weapon should focus. Data sensitivity and habituation remain material challenges to LAWS deployment.

Technical Debt

Thus far, this article has evidenced the significant ‘technical debt’ involved in removing human supervision from lethal engagement. Indeed, that notion provides a useful metaphor for the Cohort and deployment policies.³⁷37. Philippe Kruchten, Robert L Nord and Ipek Ozkaya, ‘Technical Debt: From Metaphor to Theory and Practice’, IEEE Software (Vol. 29, November/December 2012), pp. 18–19.View all notes Technical debt arises from inappropriate architecture, shortcuts resulting from commercial pressures, poor testing protocols and poor holistic understanding of LAWS. It also arises from inadequate project ownership, poor technical leadership and the consequences of pervasive changes in weapon specification. Scale compounds technical debt through an exponentially growing number of interactions and interdependencies between developers’ code and those charged with deploying these weapons. Such debt compounds as projects evolve, including management of the weapon’s configuration, integration, verification and validation.

LAWS’ correction routines will be similarly challenging. Remediation requires that the correcting party factor for both the specific modification and the distribution of system errors. The complication for commanders is that patches empirically give rise to deadlock whereby the local optimum for a learning system quickly becomes iterative so that neither the component nor its attached routine can then be improved. This is also not helped by the weapon being a bundling of incongruent proprietary routines held together by ‘glue’ or ‘spaghetti code’.³⁸38. See Pranay, ‘Machine Learning and Technical Debt with D. Sculley’, Software Engineering Daily, 17 November 2015, <https://softwareengineeringdaily.com/2015/11/17/machine-learning-and-technical-debt-with-d-sculley/>, accessed 17 May 2020.View all notes Here, ‘glue’ relates to the quantity of supporting code that must be incorporated to allow data transfer within these unsupervised routines. The phenomenon increases fragility, not least because glue code anchors the weapon to deployment idiosyncrasies and the initially programmed states of each autonomous component and so discourages experimentation, the intended advantage of ML. Researchers note that ML systems end up being 5% executive code and 95% glue code.³⁹39. D Sculley et al., ‘Hidden Technical Debt in Machine Learning Systems’, in C Cortes et al. (eds), NIPS’15: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2 (Cambridge, MA: MIT Press, 2015).View all notes Configuration is similarly problematic for the wider Cohort. If a weapon updates on new data, then its old, manually set thresholds may be invalid, all the more so given the weapons’ likely different learning states. Thus, the Cohort must ensure that configuration be undertaken in real time and validated across the whole of the weapon class to ensure the weapon remains both compliant and valuable. The leadership challenge, moreover, is what metrics should be monitored given that a purpose of its ML is, naturally, that the weapon adapts over time.

Finally, legal, social and political constraints will necessitate that configuration operational limits be set conservatively which, should an action boundary unexpectedly trigger and the weapon close down, might compromise its usefulness. A further issue here is that autonomous sub-systems then become ‘undeclared consumers’ – consuming the output of a particular prediction routine as input to another constituent of that sequence – thereby creating unintended feedback loops between the weapon’s algorithm and external world. Such loops are akin to filter bubbles in social networks whereby noise suppression mechanisms inadvertently suppress nonconforming data. The phenomenon eventually makes it problematic for the commander to execute any changes to the weapon’s firmware.

Empirical Operations

Specific operational difficulties will complicate the tasking of LAWS. Target profiles – a central cog in LAWS deployment where lethal engagement may be authorised if sensed characteristics match onboard target signatures – give rise to several challenges, particularly around the initial construction of those embedded target profiles.⁴⁰40. Article 36, ‘Targeting Profiles’, August 2019, <https://article36.org/wp-content/uploads/2019/08/Target-profiles.pdf>, accessed 17 May 2020.View all notes Such profiles must remain immutable and not update under the same learning intended for the balance of the weapon’s systems.

It is, however, the practical essentials arising from LAWS deployment which present fundamental leadership tests, including the logistics of in-field replenishment (and the advance planning of this function), their repair and servicing, polling data and LAWS performance, the dynamic audit of these systems’ actions and the frictionless liaison between machine and command structure. Commanders must understand the weapons’ competences, notwithstanding that each deployed system will, by definition, be more than fractionally different from its colleague system, depending on the learning path individual LAWS or components encounter since deployment. Other practical matters include the failure modes for such weaponry (issues of outright veto, ‘fail safe’, ‘fail dangerously’ and ‘fail deadly’) as well as the integration of otherwise independent assets into the commander’s wider portfolio of battleplan assets.⁴¹41. Scharre, ‘Autonomous Weapons and Operational Risk’, p. 42.View all notes A reliable ‘abort’ mechanism is particularly problematic given the ramifications of a cyber attack, the Cohort’s overarching requirement for a resilient and predictable asset against legal and ethical repercussions in instances of the weapon’s catastrophic failure.

Conclusion

The fundamental leadership issue for all levels of the Delivery Cohort is whether it can ever be appropriate to cede the decision to kill to an algorithm, and whether the local commander can reasonably allocate tasks within a battleplan to weapons that cannot be overseen. The Cohort must therefore decide whether disruptive hardware will outperform the training and ingenuity of the human soldier, especially given that sophisticated electronics might actually be confounded by quite low-tech solutions.⁴²42. This potential implication stems from the theory of nullification, which forecasts short-lived advantages from new military technology.View all notes A second ramification is that commanders must retain a fundamental scepticism around LAWS technical feasibility, all the more so given the many alternative levers available to leaders to achieve their objectives. ◼

How Erosion of Human Supervision Over Lethal Engagement Will Impact How Commanders Exercise Leadership

Published online:

Introduction