FES NE W YORK ANALYSIS PEACE AND SECURITY Laws for LAWS Towards a treaty to regulate lethal autonomous weapons Elizabeth Minor February 2023 • A broad range of countries agree on a two-pronged approach towards the regulation of Lethal Autonomous Weapons Systems(LAWS): prohibiting autonomous weapons systems that do not allow for human control, while devising positive obligations for the human control of those systems that are not prohibited. • In 2023, states should build on this common ground and move towards clear international legal standards for those states that are willing to participate. • Committed states should ­promote policy convergence on key elements of regulation; examine a prohibition on autonomous weapons that target people; and work towards initiating negot­iations through the UN General Assembly or a standalone process.

Laws for LAWS Towards a treaty to regulate lethal autonomous weapons • The automation of decision-making has impacts and implications in many areas of society. Increasing autonomy in weapons systems represents the most deadly iteration and moving forward towards legal regulation is an urgent task. • A broad range of countries – including many involved in these technological and military developments – have acknowledged that regulation of lethal autonomous weapons should be based on a two-pronged approach: prohibiting autonomous weapons systems that do not allow for sufficient human control, while devising positive obligations for the human control of those systems that are not prohibited. • In 2023, states should take steps to build on this common ground towards establishing clear international legal standards for those states that are willing to participate. Such standards can, in turn, exert a wider influence. Committed states should promote policy convergence on the key elements of regulation; examine a prohibition on autonomous weapons systems that target people; and work towards initiating negotiations through the UN General Assembly or a standalone process. For further information on this topic: https://ny.fes.de/topics/sustaining-peace

PEACE AND SECURITY Laws for LAWS Towards a treaty to regulate lethal autonomous weapons

CONTENTS Contents 1 INTRODUCTION 2 2 INCREASING AUTONOMY IN WEAPONS SYSTEMS TODAY 2 3 POLICY CONVERGENCE 4 3.1  Prohibition+ positive obligations 4 3.2  Human control and compliance 5 4 POLITICAL DIFFERENCES – AND CCW DEADLOCK 5 5 OPPORTUNITIES FROM COMMON GROUND 6 6 THE NEED FOR SWIFT NORM-SETTING 7 7 MOVING FORWARD 7 1

FRIEDRICH-EBERT-STIFTUNG – Laws for LAWS 1 INTRODUCTION States have been considering how to respond to the ethical, legal and peace and security challenges posed by increasing autonomy in weapons systems since 2013, first at the Human Rights Council, and then since 2014 at the Convention on Conventional Weapons(CCW). The CCW’s discussion has focused on»emerging technologies in the area of Lethal Autonomous Weapons Systems(LAWS)«. 1 As these deliberations proceed, several countries are investing heavily in the research, development and deployment of more advanced technologies and systems that could be used to automatically apply force to targets – including people – based on the processing of sensor data. 2 Developments in such systems and their usage risk eroding human control in the use of force and automating the taking of human life. The absence of clear, international legal norms and standards only heightens these risks. As of early 2023, a broad range of countries – including many involved in these technological and military developments – have acknowledged that ensuring human control over weapons systems facilitates compliance with existing law. Moreover, these countries generally agree that the regulation of LAWS should be based on a two-pronged approach: prohibiting autonomous weapons systems that do not allow for sufficient human control, while devising positive obligations for the human control of those systems that are not prohibited. This broad policy convergence has been a significant and welcome development during the past two years of discussions in the CCW. Notwithstanding this broad policy convergence, political positions continue to differ among its adherents. Is an international legally binding instrument needed that could develop and codify standards on this basis? Or, alternatively, would national standards or the international discussion of good practices be an adequate response? Although progress in developing common understandings has been made at the CCW, its recent meetings have shown that meaningful outcomes – whether legally or political binding – cannot currently be secured in this consensus-based forum. This paper argues that strong international norm-setting in this area nevertheless remains an urgent, necessary, and achievable task for states. In 2023, states can build on the common ground that has been achieved in policy discussions, towards initiating a process to set international legal standards for those who are willing to participate. Even if not acceded to by all actors, these standards can, in turn, exert a wider influence. 2 INCREASING AUTONOMY IN WEAPONS SYSTEMS TODAY The International Committee of the Red Cross(ICRC) describes autonomous weapons systems as those that»select and apply force to targets without human intervention«. The ICRC explains that therefore: After initial activation or launch by a person, an autonomous weapon system self-initiates or triggers a strike in response to information from the environment received through sensors and on the basis of a generalised»target profile«. This means that the user does not choose, or even know, the specific target(s) and the precise timing and/or location of the resulting application(s) of force. 3 Ethical, legal and peace concerns arise from states’ development and use of these systems because of the uncertainty they entail as to when, where and to what they will apply force. These characteristics raise challenges for ensuring adequate human control over the use of force, which has legal and humanitarian implications, as well as implications for international stability. They also raise more fundamental challenges around dignity and further dehumanisation in the use of force. For example, if such systems are used against people, this entails individuals being sensed, processed and targeted as patterns of data and objects by machines – which is qualitatively different to other weapons systems. 4 1 On the latest mandate for the CCW’s discussion, see Meeting of the High Contracting Parties to the Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects.(2022)»Final report« UN Document CCW/MSP/2022/7; available at: https://unoda-documents-library.s3.amazonaws.com/ Convention_on_Certain_Conventional_Weapons_-_Meeting_of_ High_Contracting_Parties_(2022)/CCW-MSP-2022-7-Advance_ version.pdf(last accessed on 28.01.2023). 2 For some of the latest news and developments, see, for example, the Automated Decision Research project; available at: https://automatedresearch.org(last accessed on 28.01.2023). 3 International Committee of the Red Cross.(2021)»ICRC position on autonomous weapon systems«; available at: https://www.icrc.org/ en/document/icrc-position-autonomous-weapon-systems. The concept of»target profiles« is elaborated in Richard Moyes.(2019), »Target profiles,« Article 36 https://article36.org/wp-content/ uploads/2020/12/Target-profiles.pdf(both last accessed on 28.01.2023). 4 For a discussion of the issues around targeting people, see Maya Brehm.(2019),»Targeting people,« Article 36; available at: https:// article36.org/wp-content/uploads/2019/11/targeting-people.pdf (last accessed on 28.01.2023). 2

Increasing autonomy in weapons systems today The challenges of ensuring adequate human control with autonomous weapons systems increase with the lag time between human decision-making and the system’s application of force, and with the size and complexity of the area over which it operates. They also grow with the scope and complexity of the‘target profile’, because the operator may not adequately understand what the effects of the system will be on the environment and the objects it contains. States have been using sensor-based weapons systems for some time. These systems use sensors to gather information, calculate whether this matches a pre-determined‘target profile’, which may for example be a heat shape, radar signature, or object of a certain weight, and apply force if so. 5 Some such systems have raised significant problems to which the international community has responded. For example, anti-personnel landmines, prohibited by the Anti-Personnel Mine Ban Convention, 6 have an indefinite duration of operation and wide target profile, which contributes to their causing indiscriminate harm to civilians long after their emplacement during armed conflicts. Other systems, such as missile defence systems, have challenged militaries to ensure adequate human control and decision-making when using high-speed automated weapons. These systems come with dangers for civilians when the latter fall within their target profiles. Militaries seek to reduce the dangers of their deployment by limiting the time and area of operation. 7 Currently, several states are investing in researching, developing and acquiring weapons systems that integrate more autonomy into various aspects of their functioning, including the selection and engagement of target objects and the construction of target profiles. Such developments include incorporating advanced computational techniques under the broad umbrella of‘artificial intelligence’. 8 These advances have the potential to extend the duration and geographical scope over which militaries operate weapons systems that apply force automatically, as well as the complexity of the target profiles used in these systems. 9 These developments present risks to ensuring meaningful human control over the use of force, and, ultimately, risk increasing automation in the killing of people. 5 Also discussed in Moyes(2019) above note 3. 6 The Convention on the Prohibition of the Use, Stockpiling, Production and Transfer of Anti-Personnel Mines and on Their Destruction(also known as Anti-Personnel Mine Ban Convention, Ottawa Convention, or Mine Ban Treaty) was agreed in 1997. See the convention’s official website: https://www.apminebanconvention.org/en/(last accessed on 28.01.2023). 7 See also Ingvild Bode and Tom Watts.(2021)»Worried about the autonomous weapons of the future? Look at what’s already gone wrong,« Bulletin of the Atomic Scientists; available at: https:// thebulletin.org/2021/04/worried-about-the-autonomous-weaponsof-the-future-look-at-whats-already-gone-wrong/(last accessed on 28.01.2023). 8 For recent overviews of developments, see, for example, Automated Decision Research.»Reports and briefings,« https://automatedresearch.org/reports-briefings/ and recent publications by UNIDIR »Current Research Artificial Intelligence and the Weaponization of Increasingly Autonomous Technologies«; available at: https://unidir. org/projects/artificial-intelligence-and-weaponization-increasinglyautonomous-technologies?page=0%2C0%2C0(both last accessed on 28.01.2023). As has been noted elsewhere, the character or direction of current developments mean that»rather than a watershed moment there are likely to be continued steps towards more autonomy and a reduction of the role of the human user(s) in the decision-making process.« 10 There are currently a range of systems that are being developed or have recently come into use that could raise concerns around human control if used in certain modes of operation. Especially problematic are features that lead to digital dehumanisation, a process whereby humans are reduced to data, which is then used to make decisions or take actions that negatively affect their lives, or both. 11 One example is that of loitering munitions systems, aerial systems released over a target area for a period of time, which can automatically apply force to detected target objects. Such systems are being developed or used by the US, Israel, Turkey and Russia, amongst others, and their use has been reported on in various current conflicts. For instance, a UN Panel of Experts reported the use of the Kargu-2 loitering munition in Libya in 2021. 12 The system has automatic target recognition capabilities, which the manufacturer is reportedly extending to include facial recognition. 13 The use of such a feature would raise concerns about the automatic targeting of people as objects sensed by autonomous weapons systems, which the ICRC and others have highlighted as ethically unacceptable. 14 The use of loitering munitions in Ukraine during 2022 has also been widely reported on. 15 The KUB loitering munition used by Russia, 16 for example, reportedly has»AI visual identification, which can perform‘real-time recognition and classification of detected objects’.« 17 These features potentially raise concerns around increasingly complex target profiles: 9 Daan Kayser.(2021)»Increasing autonomy in weapons systems: 10 examples that can inform thinking,« Automated Decision Research; available at: https://automatedresearch.org/news/report/increasingautonomy-in-weapons-systems-10-examples-that-can-informthinking/(last accessed on 28.01.2023). 10 Ibid. 11 For an overview of this concept, see Catherin Connelly.(2022) »Autonomous weapons and digital dehumanisation,« Automated Decision Research; available at: https://automatedresearch.org/news/ report/autonomous-weapons-and-digital-dehumanisation-a-shortexplainer-paper/(last accessed on 28.01.2023). 12 See Letter dated 8 March 2021 from the Panel of Experts on Libya established pursuant to resolution 1973(2011) addressed to the President of the Security Council March 2021; available at: https://documents-dds-ny.un.org/doc/UNDOC/GEN/N21/037/72/PDF/ N2103772.pdf?OpenElement(last accessed on 28.01.2023). 13 Kayser(2021) above note 9 14 See ICRC(2021)»ICRC position and background paper«; available at: https://www.icrc.org/en/download/file/166330/icrc_position_on_ aws_and_background_paper.pdf; Brehm(2019) above note 4; Stop Killer Robots,»Our policy position«; available at: https://www. stopkillerrobots.org/our-policies/(both last accessed on 28.01.2023). 15 Automated Decision Research.(2022)»Weapons systems with autonomous functions used in Ukraine«; available at: https://automatedresearch.org/news/weapons-systems-with-auto nomous-functions-used-in-ukraine/(last accessed on 28.01.2023). 16 Wired.(2022)»Russia’s Killer Drone in Ukraine Raises Fears About AI in Warfare«; available at: https://www.wired.com/story/ai-dronesrussia-ukraine/(last accessed on 28.01.2023). 17 Kayser(2021) above note 9 3

FRIEDRICH-EBERT-STIFTUNG – Laws for LAWS the risk grows that operators do not adequately understand or predict what the effects of a system will be in the environment of use. This erodes the level of meaningful human control over the weapons system. These recent developments make it all the more urgent that states should agree on the regulation of LAWS. The imperatives to do so range from the adequate protection of civilians in armed conflict to managing the risky dynamics of international competition in weaponry between highly militarised states. 3 POLICY CONVERGENCE A significant development over the past two years has been the emergence of common understandings amongst a range of countries on some key policy aspects for the regulation of autonomous weapons systems. The level of agreement amongst states should not be overstated: there are continuing differences on finer technical points, but also on political questions, such as the need to develop new international law. Nevertheless, there is now a broad convergence amongst a wide range of countries on two key policy points: first, the need to combine prohibition with positive obligations; and second, the linkage of human control and legal compliance. A group of European states including France and Germany has also proposed a»two-tiered approach« of recognising the prohibition of systems not compliant with International Humanitarian Law(IHL) while regulating others to ensure legal compliance. 21 China suggested distinguishing between prohibiting and regulating different autonomous weapons. 22 The Non-Aligned Movement also called for a legally binding instrument containing prohibitions and regulations. 23 A paper submitted by the US and others drew attention to both a consensus that weapons not usable in accordance with IHL would be prohibited and identified»measures to mitigate the risk of unintended engagements« and»good practices related to human-machine interaction«. These measures and good practices are similar in content to the regulations or positive obligations suggested by other countries to maintain meaningful human control over weapons systems. They include, for instance, limiting the types of targets systems can engage, the»duration, geographical scope, and scale of the operation« of a weapons system, and taking steps to ensure that operators and commanders understand how systems function. 24 The Chair of the Group of Governmental Experts(GGE), the CCW’s current forum for discussing autonomous weapons systems, sought to reflect this convergence and the common 3.1 Prohibition+ positive obligations Firstly, a broad convergence has developed regarding the structure required for the regulation of autonomous weapons systems: countries have proposed that this should recognise both the types of systems that are or should be prohibited, as well as positive obligations or regulation to ensure adequate human control over other autonomous weapons systems. The working papers submitted to the CCW in 2022 give an indication of the positions of the states most active in the discussion. In these, a structure of prohibition and regulation was put forward in a paper on elements for a legally binding instrument proposed by Chile and Mexico, 18 as well as in various papers submitted by a group of Latin American, Asian and African states proposing that a new legally binding instrument should be adopted. 19 Many of these countries also belong to a regionally diverse group of 23 states recommending that the outcomes of the CCW’s work in 2022 should include a commitment to»work collaboratively to prohibit autonomous weapons systems that are not sufficiently predictable or controllable to meet legal requirements«. Commitments should also be made to »identify and agree on limits and other regulations to uphold the rules of international humanitarian law for other types of autonomous weapons systems« including limits on the type of target and»duration, geographical scope and scale of use«. 20 18 Chile and Mexico.(2022)»Elements for a legally binding instrument to address the challenges posed by autonomy in weapon systems«; available at: https://documents.unoda.org/wp-content/uploads/ 2022/08/WP-Chile-and-Mexico-.pdf(last accessed on 28.01.2023). 19 The range of states endorsing different papers varied slightly; but see, for example, Argentina, Costa Rica, Guatemala, Kazakhstan, Nigeria, Panama, the Philippines, Sierra Leone, State of Palestine and Uruguay. (2022)»Proposal: Roadmap Towards New Protocol on Autonomous Weapons Systems«; available at: https://view.officeapps.live.com/op/ view.aspx?src=https%3A%2F%2Fdocuments.unoda.org%2Fwpcontent%2Fuploads%2F2022%2F05%2F20220311-G10-proposallegally-binding-instrument.docx&wdOrigin=BROWSELINK(last accessed on 28.01.2023). 20 Argentina, Austria, Belgium, Chile, Costa Rica, Ecuador, Guatemala, Ireland, Kazakhstan, Liechtenstein, Luxembourg, Malta, Mexico, New Zealand, Nigeria, Panama, Peru, the Philippines, Sierra Leone, Sri Lanka, State of Palestine, Switzerland and Uruguay(2022)»Working Paper submitted to the 2022 Chair of the GGE on emerging technologies in the area of lethal autonomous weapons systems(LAWS)«; available at: https://documents.unoda.org/wp-content/ uploads/2022/05/2022-GGE-LAWS-joint-submission-workingpaper-G-23.pdf(last accessed on 28.01.2023). 21 Finland, France, Germany, the Netherlands, Norway, Spain and Sweden.(2022)»Working paper submitted to the 2022 Chair of the Group of Governmental Experts(GGE) on emerging technologies in the area of lethal autonomous weapons systems(LAWS)«; available at: https://documents.unoda.org/wp-content/uploads/2022/07/ WP-LAWS_DE-ES-FI-FR-NL-NO-SE.pdf(last accessed on 28.01.2023). 22 People‹s Republic of China.(2022)»Working paper on Lethal Autonomous Weapons Systems(English version)«; available at: https://documents.unoda.org/wp-content/uploads/2022/07/WorkingPaper-of-the-Peoples-Republic-of-China-on-Lethal-AutonomousWeapons-Systems(English).pdf(last accessed on 28.01.2023). 23 Bolivarian Republic of Venezuela on behalf of the Non-Aligned Movement(NAM) and other States Parties to the CCW.(2022) »Working paper«; available at: https://documents.unoda.org/ wp-content/uploads/2022/08/WP-NAM.pdf(last accessed on 28.01.2023). 24 Australia, Canada, Japan, the Republic of Korea, the United Kingdom and the United States.(2022)»Principles and Good Practices on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems«; available at: https://view.officeapps.live.com/op/view. aspx?src=https%3A%2F%2Fdocuments.unoda.org%2Fwp-content %2Fuploads%2F2022%2F05%2F20220307-US-UK-RoK-JAP-CANAUS-Final-proposal-laws-principles-and-good-practices.docx&wd Origin=BROWSELINK(last accessed on 28.01.2023). 4

Political differences – and CCW deadlock elements highlighted by a broad range of states in the first draft of his report of the Group’s work(which was not, ultimately, adopted). 25 This broad convergence on»an approach based on the prohibition of autonomous weapon systems that cannot be used in compliance with IHL, and the regulation of other types of autonomous weapon systems« was also, significantly, highlighted in a joint statement led by Austria delivered to the UN General Assembly’s First Committee in 2022. This statement had the support of states from many of the regional and political groupings whose interventions at the CCW are described above. The list of endorsing states included NATO members and countries involved in developing autonomy in weapons systems, as well as some of the states most strongly opposed to increasing such autonomy. 26 3.2 Human control and compliance The second significant policy point on which broad convergence has developed in recent years at the CCW is the linking of legal compliance – particularly regarding IHL – to human control, decision-making and accountability over weapons systems. This linkage has been highlighted by a similarly broad range of countries. In 2022, for example, the working paper submitted by the above-mentioned group of 23 states suggested that the CCW should recognise that these factors are necessary»in order to ensure compliance with International Law«. 27 The paper by the US and others suggests that»human-machine interaction … should ensure that the potential use … is in compliance with applicable international law«. 28 A paper submitted by the UK also suggests that the process to agree guidelines and good practices it proposes»would need to address the level of human involvement[that] is necessary to achieve the IHL ends«. 29 Several countries agreed on»the necessity for human beings to exert appropriate control, judgement and involvement in relation to the use of weapons systems in order to ensure any use is in compliance with International Law«. Highlighting that sufficient human control is a means to achieve compliance was also noted in the Austrian-led statement to the UN General Assembly First Committee that was supported by a diverse grouping of 70 states. 30 4 POLITICAL DIFFERENCES – AND CCW DEADLOCK There are both policy and political differences amongst the states that recognise these key policy points. For example, states are at variance when it comes to identifying which systems and uses are already‘de facto’ prohibited, or which should be prohibited. These countries also differ over what political response is required to develop and enact the principles and structures of regulation that they have highlighted. The Automated Decision Research project’s»State positions« monitor numbers, at the time of writing, 86 countries that have spoken in favour of developing a legally binding instrument to regulate autonomous weapon systems. 31 Some states within the broad policy convergence described above, such as the US, UK and others that endorsed the US-led working paper to the CCW in 2022, have explicitly rejected negotiating a legal instrument, proposing instead that agreement on principles and good practices should be developed within the CCW. 32 The working paper by France, Germany and others suggests that standards should be operationalised by states at the national level. 33 Some states in this grouping have explicitly rejected international legal regulation(for example, France 34 ), some have a national mandate to pursue it(for example, the Netherlands 35 ), and some have not declared their position. Notably, some European countries appear to have higher-level domestic political support or mandates that could be used to advance legal regulation, but which are not matched or pursued in their diplomatic positioning in international fora such as the CCW. For example, the Foreign Minister of the previous German government called for states parties at the UN General Assembly in 2018 to support Germany’s »initiative to ban fully autonomous weapons – before it is too late!« 36 In 2019 the MFA stated that Germany’s goal was a worldwide ban on fully autonomous lethal weapons 25 Available at: https://reachingcriticalwill.org/images/documents/ Disarmament-fora/ccw/2022/gge/documents/chair-draft-report1.pdf (last accessed on 28.01.2023). 26 For the statement and full list of endorsing countries, see Austria et al.(2022)»Joint Statement on Lethal Autonomous Weapons Systems First Committee, 77th United Nations General Assembly Thematic Debate – Conventional Weapons«; available at: https://reachingcritical will.org/images/documents/Disarmament-fora/1com/1com22/ statements/21Oct_LAWS.pdf(last accessed on 28.01.2023). 27 Argentina et al.(2022) above note 20 28 Australia et al.(2022) above note 24 29 United Kingdom.(2022).»Proposal for a GGE document on the application of International Humanitarian Law to Emerging Techno­ logies in the Area of Lethal Autonomous Weapons Systems«; ­available at: https://view.officeapps.live.com/op/view.aspx?src= https%3A%2F%2Fdocuments.unoda.org%2Fwp-content%2F uploads%2F2022%2F05%2F03032022-UK-Proposal-for-Mar-2022LAWS-GGE.docx&wdOrigin=BROWSELINK(last accessed on 28.01.2023). 30 Austria et al.(2022) above note 26 31 Automated Decision Research,»State positions«; available at: https://automatedresearch.org/state-positions/(last accessed on 28.01.2023). 32 Australia et al.(2022) above note 24 33 Finland et al.(2022) above note 21 34 Automated Decision Research,»State positions«; available at: https://automatedresearch.org/state-positions/(last accessed on 28.01.2023). 35 Netherlands Ministry of Foreign Affairs(2022)»Letter to parliament with Cabinet response to AIVCAVV advice on autonomous weapons systems«; available at: https://www.government.nl/ministries/ ministry-of-foreign-affairs/documents/publications/2022/10/10/ letter-to-parliament-autonomous-weapon-systems(last accessed on 28.01.2023). 36 Heiko Maas.(2018)»Speech by the Federal Minister of Foreign Affairs«; available at: https://reachingcriticalwill.org/images/docu ments/Disarmament-fora/unga/2018/28September_Germany.pdf (last accessed on 28.01.2023). 5

FRIEDRICH-EBERT-STIFTUNG – Laws for LAWS systems 37 – but Germany has not adopted a clear position in the CCW in favour of international legal regulation, and has concentrated its efforts on political standards, such as‘good practice’ approaches. Similarly, the Norwegian government’s latest political platform commits it to»take necessary steps to regulate the development of autonomous weapons systems« 38 – but Norway’s negotiators in the CCW have not used this political agreement as a basis to support and pursue a legal instrument. Generally, within this broad policy convergence, countries more involved in the development and acquisition of weapons systems incorporating features of autonomy favour politically binding or‘good practice’ approaches, while others favour stronger, legal regulation. At the same time, there are a number of states – including, for example, India, Israel and Russia – that sit more outside the development of common understandings described here, and reject the development of legal or any other specific responses beyond further discussion in the CCW. Automated Decision Research has counted 11 states that have spoken against an international legal response – as well as 36 that have not adopted a clear position. 39 The latter include several more militarised countries, including NATO members, many of which endorsed the Austrian-led General Assembly statement noted above. Even major differences between states’ positions do not have to be an absolute barrier to useful progress in addressing contentious issues. In recent years, however, the CCW has proved to be an ineffective forum for securing meaningful outcomes and responses on the autonomous weapons issue and on a range of other weapons issues that raise humanitarian concerns. 40 Russia, in particular, has taken on the role of preventing the adoption of reporting or outcome language on areas of broad agreement that do not reflect its national position. In doing so, it has mobilised an interpretation of ‘achieving consensus outcomes’ frequently employed in the CCW that is wielded as an effective veto, and favours the positions of countries that have come to be considered the most significant to the work of the CCW(the so-called major military powers). 41 37 See Federal Foreign Office.(2019)»Foreign Minister Maas on agreement of guiding principles relating to the use of fully autonomous weapons systems«; available at: https://www.auswaertiges-amt.de/ en/newsroom/news/maas-autonomous-weapons-systems/2277194 (last accessed on 28.01.2023). 38 Ministry of Foreign Affairs(2021)»The Government’s political platform: A government for ordinary people«; available at: https://www.regjeringen.no/en/dokumenter/government_platform/ id2877512/(last accessed on 28.01.2023). 39 Automated Decision Research,»State positions«; available at: https://automatedresearch.org/state-positions/(last accessed on 28.01.2023). 40 For reporting on the proceedings and outcomes of the latest Meeting of High Contracting Parties, and the Review Conference of 2021, see for example Reaching Critical Will,»Convention on Certain Conventional Weapons(CCW)«; available at: https://reachingcriticalwill.org/ disarmament-fora/ccw(last accessed on 28.01.2023). The relevance and legitimacy of the CCW as a forum for dealing with weapons, means and methods of warfare that pose humanitarian problems have been severely challenged by the behaviour of states seeking to force their positions to be adopted as‘consensus’. Using the CCW’s own tools, such as interpretations of its rules of procedure, these tactics have undermined the convention’s work, while other states seem to be unable to effectively challenge these tactics. 42 As of early 2023, the landscape of states’ positions on what should be the legal or political response to the problem of increasing autonomy in weapons systems, and the way in which‘consensus’ is currently being interpreted at the CCW, means that no meaningful political or legal agreed outcome can currently be achieved there. This will remain the case for the foreseeable future. 5 OPPORTUNITIES FROM COMMON GROUND Notwithstanding this, the policy convergence that has developed within international discussions at the CCW and elsewhere provides opportunities for making progress towards an effective international response to the issues raised by increasing autonomy in weapons systems. It is clear that any effort to set international legal standards – or even to develop politically binding guidelines or consensus on good practice – will not enjoy the agreement or participation of at least some of the countries most involved in the development of autonomous weapons systems and military practice around them. Any such efforts, then, will involve a smaller number of countries seeking to establish and develop principles and rules. These, in turn, would be designed to shape global expectations around how technologies should be used, and so aim to influence the behaviour of the international community more generally. This is not an unusual dynamic in international law and policymaking around weapons technologies. The common ground that has been developed in international discussions forms a strong basis for such a norm-setting effort. The way forward could begin with those states that are committed to developing new international law in this area. They could set out from the key central points of common ground they have developed, as the basis for setting clear rules and principles on the issues that have emerged in the policy discussion as the most important. Negotiations on such an instrument could have relatively wide participation, and potentially generate broader effects – based as it would be on existing common ground. 41 For further discussion see Human Rights Watch.(2022)»An Agenda for Action Alternative Processes for Negotiating a Killer Robots Treaty«; available at: https://www.hrw.org/report/2022/11/10/ agenda-action/alternative-processes-negotiating-killer-robots-treaty (last accessed on 28.01.2023). 42 For further detail, see reporting by Reaching Critical Will, above note 40. 6

Moving forward As any treaty negotiation would have to be initiated outside the CCW, some states driving increasing autonomy in weapons systems may be more likely than others to participate. Evolving domestic mandates, and a proposed basis for negotiations that is sufficiently close to existing national policies, may encourage some to participate – even if their present position is opposed to such a currently hypothetical process. Some states with sophisticated weaponry, such as NATO member the Netherlands, already have mandates to pursue international efforts for the prohibition and regulation of autonomous weapons. However, the Dutch government has emphasised that such efforts will be ineffective without »major military powers«. 43 Others have kept a strong focus on restricting discussions to the CCW – however, there is currently no alternative process taking place. A legal instrument negotiated on the basis of broad convergence also has the potential to influence states within this convergence that nevertheless do not join the negotiations. Any international legal rules agreed upon would provide a context for discussions on good practice and national standards on the same subject matter in different fora and groupings. That could provide a reference point that other states might use in making different types of commitments. It has been noted by some states opposed to developing new law at this time that first creating politically binding standards, such as declarations or guidelines, would not preclude a legally binding instrument later 44 – the reverse is also true. More generally, states are very likely to continue their discussions within the CCW on LAWS for the foreseeable future, alongside any other developments that may occur. Any legal standards set outside the CCW framework would be relevant to, and could not be ignored by, states’ deliberations there. Though some states are likely to emphasise that they are not bound by standards they have not signed up to, their practice and positions will be increasingly scrutinised in relation to them. 6 THE NEED FOR SWIFT NORM-SETTING In a context of evolving technology and military practice in relation to increasing autonomy in weapons systems – where there are challenges to identifying when activities cross clear thresholds of concern – it is crucial for the international community to achieve greater clarity and certainty by establishing strong rules and standards. The strongest tool with which to establish such standards is international law. A process initiated by states that share a clear common purpose and goal of concluding a legal instrument 45 would greatly sharpen the discussion of the rules that are needed, and through its negotiation and conclusion would influence the broader conversation on autonomy in weapons systems by providing a strong point of reference built on existing convergences. Based on current positions, any negotiation process that might realistically take place in the coming years would be likely to involve a regionally diverse range of states, and some, but certainly not all, potential users and producers of weapons systems with autonomous features. In the absence of such a process, restricted as it may be, customary norm development will still occur. But it will be left largely to a small minority of states that are at the forefront of scientific developments and weapons deployment. The longer this is the case, the greater the risk that unacceptable technologies and practices will be developed, normalised, and fuel instability between states. 7 MOVING FORWARD The CCW’s discussions on LAWS have produced useful policy discussions in recent years. Because of how it currently functions as a forum, these cannot be taken forward into meaningful legal or political outcomes there. States were not even able to acknowledge aspects of the policy convergence outlined in this paper – such as a»two-tier« approach to regulation – in the latest report on their work at the 2022 Meeting of High Contracting Parties. Many countries, including Germany and Switzerland, observed this with regret. 46 This should not be accepted as an adequate mode of operation for an international forum dealing with matters of the conduct of war – and so of life and death. Elsewhere in 2022, over 80 states, including many of those most visible in the CCW’s discussions on LAWS, from NATO members to states from all regions concerned with peace and security, came together to agree new political standards to better protect civilians from the use of explosive weapons in populated areas. 47 Such ambitious and innovative approaches to shape global norms effectively can and should be used to approach the issue of increasing autonomy in weapons systems. To move forward, states that wish to address the challenges posed by increasing autonomy in weapons systems should: – Work to increase policy convergence on the key elements of regulation. A broad range of countries agree on the need for human control, which was shown by the Austrian-led statement of 70 states to the UN General Assembly’s First Committee on autonomous 43 Netherlands Ministry of Foreign Affairs(2022) above note 35 44 This point was raised at the July session of the CCW’s GGE by the US, for example. Reaching Critical Will.(2022).»CCW Report, Vol 10 No 10«; available at: https://reachingcriticalwill.org/images/documents/ Disarmament-fora/ccw/2022/gge/reports/CCWR10.10.pdf(last accessed on 28.01.2023). 45 One of the characteristics for a successful process identified in Human Rights Watch(2022) above note 41 46 See Reaching Critical Will.(2022)»CCW Report Vol 10 No 11«; available at: https://www.reachingcriticalwill.org/images/documents/ Disarmament-fora/ccw/2022/hcp-meeting/reports/CCWR10.11.pdf (last accessed on 28.01.2023). 47 See International Network on Explosive Weapons.(2022)»Dublin Conference to Adopt the Political Declaration on Explosive Weapons«; available at: https://www.inew.org/dublin-conference-to-adoptthe-political-declaration-on-explosive-weapons/(last accessed on 28.01.2023). 7

FRIEDRICH-EBERT-STIFTUNG – Laws for LAWS weapons systems. 48 States should build on this, and develop the key points of agreement on the elements of regulation that are necessary to ensure human control. These would in all likelihood focus on the adequate understanding of systems and their effects in the context of use, and limiting the location and duration of operation of the systems. These discussions should be pursued in all relevant international fora, including the CCW, Human Rights Council and First Committee, in addition to regionally and nationally convened meetings, such as the regional conference on autonomous weapons being organised by Costa Rica in February 2023. – Examine the specific humanitarian and ethical challenges posed by autonomous weapons systems that target people, and the need to prohibit such systems. The ICRC, Stop Killer Robots and others have drawn attention to specific problems relating to dignity, dehumanisation, civilian protection and discrimination posed by autonomous weapons systems used to target people, and have called for a clear prohibition of these systems alongside other prohibitions and regulations. 49 Despite many states observing that life and death decisions should not be left up to machines, more indepth consideration is currently lacking – though such a prohibition would be a clear and achievable component of a broader structure. 50 The automation of decision-making has impacts and implications in many areas of society, particularly for human rights and digital dehumanisation, of which increasing autonomy in weapons systems represents the most deadly iteration. Away from the CCW, the potential risks to human rights of emerging military technologies using algorithms and machine learning were also recognised by states at the Human Rights Council in 2022. With concern remaining high amongst civil society, international organisations and the UN Secretary-General, 52 the tech and robotics sectors 53 and a large number of states, moving forward towards legal regulation is an urgent task. With convergence developing amongst states on many principles and points of policy, those states that are willing to participate must now take the next step towards developing these elements into international law that could win broad support and exert widespread influence. – Take steps towards initiating negotiations on an international legal instrument to prohibit and regulate autonomous weapons systems. In taking steps towards initiating negotiations either through the UN General Assembly or a standalone process, committed states should focus on their goal of an international legal instrument and pursue a process that will achieve that, rather than being wedded to a particular forum. They should find the most effective process that brings together states united by the common purpose of effectively regulating autonomy in weapons systems, cannot be blocked by‘consensus’, has clear deadlines for its conclusion, and is inclusive of all relevant stakeholders. 51 48 Austria et al.(2022) above note 26 49 See for example ICRC, Stop Killer Robots, above note 14 and Elizabeth Minor and Richard Moyes.(2020)»Key resource: Regulating autonomy in weapons systems,« Article 36; available at: https:// article36.org/updates/treaty-structure-leaflet/(last accessed on 28.01.2023). 50 See Brehm(2019) above note 4 51 For an elaboration on these recommendations on the characteristics of an effective process, see Human Rights Watch(2022) above note 41 52 Automated Decision Research.(2022)»UN Secretary-General reiterates call for action on autonomous weapons,« https://automatedresearch.org/news/un-secretary-general-reiteratescall-for-action-on-autonomous-weapons/ 53 See for example Stop Killer Robots(2022)»Robotics industry takes a stand«; available at: https://www.stopkillerrobots.org/news/roboticsindustry-takes-a-stand/(last accessed on 28.01.2023). 8

IMPRINT ABOUT THE AUTHOR IMPRINT Elizabeth Minor is an Advisor at Article 36, a UK-based specialist non-profit organisation focused on reducing harm from weapons. Her work currently focuses on the themes of autonomous weapons and addressing the ongoing harm caused by past nuclear weapons activities. She sits on the international steering groups of Stop Killer Robots and the International Campaign to Abolish Nuclear Weapons for Article 36, as well as the steering group of the UK campaign to Stop Killer Robots. Publisher: Friedrich-Ebert-Stiftung e.V.| Godesberger Allee 149| 53175 Bonn| Germany Email: info@fes.de Register no.: VR2392 Bonn Register of Associations Bonn Local Court Issuing Department: Division for International Cooperation/ Global and European Policy Responsibility for content: Dr. Michael Bröning| Executive Director| Friedrich-Ebert-Stiftung| New York Office| 747 Third Avenue, Suite 34D| New York, NY 10017| USA Phone+1-212-687-0208 https://ny.fes.de Contact/Order: Christiane.Heun@fes.de Layout: Ludger Stallmeister, Wuppertal NEW YORK OFFICE The office of Friedrich-Ebert-Stiftung(FES) in New York serves as the liaison for FES offices worldwide with the United Nations(UN) in New York and the international financial institutions(International Monetary Fund and World Bank) in Washington, D.C. The office addresses peace, justice and economic issues, working closely with academia, civil society, multilateral institutions and their Member State governments to convene multi-stakeholder debates. The formats of our work include international conferences, expert workshops and high-level meetings involving government representatives, as well as published policy briefs and analytical studies. Our overarching mission is to bring a special focus to the perspectives of trade unions, women, and developing and emerging-market countries in an effort to forge consensus toward multilateral solutions to international and global challenges. The views expressed in this publication are not necessarily those of the Friedrich-Ebert-Stiftung. Commercial use of media published by the Friedrich-Ebert-Stiftung(FES) is not permitted without the written consent of the FES. Publications by the Friedrich-Ebert-Stiftung may not be used for electioneering purposes. ISBN: 978-3-98628-249-3 © 2023 www.fes.de/bibliothek/fes-publikationen