AI for All 10 Social Conundrums for India August 2019

IV Preface “After all, we make ourselves according to the ideas we have of our possibilities.” V.S. Naipaul There is no doubt that the technological advancement has become the game changer of our times. From the Industry 4.0 discourse launched in Germany in 2011 to the scientific advisory report presented to the former US president Barrack Obama on big data and privacy concerns in 2014, to India’s NITI Aayog Artificial Intelligence for All strategy of 2018. A lot of debates have culminated in the questions about the Future of Work in the context of the International Labour Organisation’s Centenary in 2019. Triggered by the disruptive forces of technology based start-ups and new business models, a new race for innovations and war for talents has arisen and with it, a new form of global and fierce competition. Technology has become the holy grail of progress though it did not take long to realise that there is a social dimension attached to it. The platform economy has had severe effects on the bargaining power of suppliers and workers. Data analytics opened a whole array of ethical questions regarding personal tracking and privacy. Further, technological upgrades create productivity gains by efficiency which in turn requires reduced human labour. This poses a particular threat to emerging economies, like India, which need to create new jobs on massive scale for its young and growing population. The utopia around Artificial Intelligence in the times of jobless growth presents a whole new set of challenges. Is the Indian economy ready to ride the AI wave? Who will benefit from AI: investors, big tech, users, or society as a whole? What is and can be India’s role in this global race for innovation? Is tech gender neutral? What about privacy and user protection? How to ensure decent work and social protection in this new age tech revolution? But mostly, how can we turn Artificial Intelligence for All into a reality? To foster this debate, the FES India Office has teamed up with several experts and organisations across the country to explore ground realities with the objective to understand how technology is already unfolding in selected sectors, draft scenarios of what might happen and to ensure proper safeguards are put in place at the right time. Artificial Intelligence like any other technology is neither good nor bad. It is what we make out of it- the rules and regulations – which define the outcome of the game. Just like other countries, in India too, a mass scale application of AI is far from being established. It is still in a nascent phase and can be moulded into a success story. A success story in India AND Indian success story for all. Patrick Ruether and Mandvi Kulshreshtha August 2019 Friedrich-Ebert-Stiftung, New Delhi

V Note of thanks Friedrich-Ebert-Stiftung(FES) India Office is thankful to its partner Tandem Research for preparing this research paper. Tandem Research is an interdisciplinary research collective that generates policy insights at the interface of technology, society, and sustainability. FES India and Tandem Research are co-organising six policy labs to unpack AI applications in key sectors like healthcare, education, agriculture and urbanity. We are grateful to our colleagues at Tandem Research for preparing the research, drafting this paper and refining the manuscript to reflect our joint vision. We have to express our appreciation to all the experts and resources persons who participated in these labs, for their constructive contribution and valuable time during the course of this research. Tandem Research’s Technology Foresight Group(TFG) brings together multiple stakeholders to collectively and iteratively diagnose issues and challenges pertinent to technology and society futures in India. The present paper was developed at the AI Lab held in July 2018 and first published in December 2018(https://tandemresearch. org/assets/AI_For_All_Working_Paper_Tandem-Research2018-WEB-compressed.pdf). The brief is based on discussions of the TFG but should not be seen as a consensus document— participant views differ and this document need not reflect the views of all participants. Member s of the T echnology F oresight G roup Anita Gurumurthy, founding member and executive director, IT for Change Anja Kovacs, directs the Internet Democracy Project Arnab Kumar, leads NITI Aayog’s initiatives in Emerging Technologies(Artificial Intelligence, FinTech, BlockChain) Baneen Karachiwala, Public Health Researcher Chinmayi Arun, founder, Centre for Communication Governance Daniel Buhr, Head, Steinbeis Transfer Center for Social and Technological Innovation Deepak Khemani, professor, Department of Computer Science and Engineering, Indian Institute of Technology, Madras Mahesh Venkateswaran, Chief Growth Officer, Knack Malavika Jayaram, inaugural Executive Director, Digital Asia Hub Naveen Thayyil, faculty, Department of Humanities and Social Sciences, Indian Institute of Technology, Delhi Osama Manzar, global leader on the mission of eradicating information poverty from India and global south using digital tools Satish Sangameswaran, Principal Program Manager, Microsoft Research India Sean Blagsvedt, founder and former CEO, Babajob. com, Principal at Jaaga.in Sumandro Chattapadhyaya, Research Director, Centre for Internet and Society(CIS) Usha Ramanathan works on the jurisprudence of law, poverty and rights Patrick Ruether, Country Representative& Head of Office, FES India Sehaj Malik, Adviser, Socio-Economic Transformation, FES India Urvashi Aneja, Founding Director, Tandem Research Vikrom Mathur, Founding Director, Tandem Research

Contents List of abbreviations................................................................................................................................... VIII I. Artificial Intelligence for all..................................................................................................................... 1 II. 10 Social conundrums for india.............................................................................................................. 3 1. Reconciling multiple, competing social narratives............................................................................. 3 2. Concentration& collusion of power................................................………………………………….... 4 3. Work, mobility and digital labour................................................... ……………………………….…… 5 4. Social/ digital identity..................................................................................................................... 5 5. Biased robots................................................................................................................................. 6 6. Concentration of knowledge(and power)........................................................................................ 7 7. Privacy frameworks: unfit for purpose?............................................................................................ 7 8. Reporting citizens/ surveillance state............................................................................................... 8 9. What/ how to govern.................................................................................................................. 9 10. Reconfiguring human agency....................................................................................................... 10 III. Conclusion: Building a capacity for reflexiveness................................................................................ 11

VIII List of abbreviations AI Artificial Intelligence IoLT BW Business World IoT IIT Indian Institute of Technology TFG IIIT Indian Institute of Information Technology TRAI Internet of Living Things Internet of Things Technology Foresight Group The Telecom Regulatory Authority of India

Artifical Intelligence for All 1 Artificial Intelligence for all Artificial Intelligence(AI) has been a subject of much speculation and debate since the 1950s, when Alan Turing asked if machines could think. 1 However, the development and deployment of AI has proliferated only recently, enabled by access to vast amounts of data, a massive increase in computational power, and better algorithms. From automating warfare to composing art, AI has the potential to radically transform society. The nearly boundless promise of efficiency and productivity gains, along with new forms of value creation, has focused the attention of technology AI, thus, needs to companies and policy makers be conceived of as on AI development. Yet, AI a‘socio-technical is not just a new frontier for system’, technical innovation and technology; feasibility and social its social dimensions and viability are deeply implications are even more interlinked. complex. The development and deployment of AI is likely to be a thoroughly social affair, shaped not only by technological possibilities but also an interplay of power, interests, values, and user behaviours. As a system that does not function autonomously but is the outcome of the activities of human actors, and which encompasses the production, diffusion, and use of technology. 2 We need to think both technically and socially. The term‘artificial intelligence’ does not lend itself to a simple, straightforward definition, at least partly because of the growing hype around it and the resultant tendency to describe various data-driven applications or algorithmic decision-making processes as AI. AI generally refers to the use of digital technologies to create systems that are capable of performing tasks thought to require intelligence. Machine learning is a technique or subsystem of AI, whereby digital systems can improve their performance on a given task over time through experience. 3 ‘General artificial intelligence’ still remains a thing of the distant future.‘Narrow AI’, involving sophisticated pattern recognition across multiple data points to generate probabilistic models and correlations, is already ubiquitous across multiple spheres of life— from algorithms that filter out spam to those that increase the accuracy of detecting cancerous growths. The National Institution for Transforming India- NITI Aayog’s discussion paper on India’s national AI strategy, titled‘Artificial Intelligence for All’, seeks to position India as a global AI leader by promoting AI solutions for healthcare, education, agriculture, mobility and smart cities. 4 While the paper hits many of the right notes in suggesting the importance of societal objectives in steering technological trajectories, it arguably overplays the potential of AI and underplays the challenges and risks entailed. This is particularly concerning in light of the suggestions for creating a data marketplace and positioning India as a‘garage’ for testing AI solutions applicable to the developing world. AI strategy and policy ultimately presents a‘wicked problem’ for public policy. Wicked problems are those that have multiple interacting systems— social, ecological, and economic— a number of social and institutional uncertainties, and imperfect knowledge. Possible solutions to existing problems create a new set of additional challenges and the choice between available alternatives are often largely about competing values. BW Head describes wicked problems as representing a confluence of three factors: complexity of subsystems and interdependencies; uncertainty regarding risks and consequences of interventions; and a divergence or fragmentation in values, viewpoints and strategic intentions. 5 ----------------------------------------------------------------------------------------------------------------1 Turing, Alan M.(1950). Computing machinery and intelligence._Mind_ 59(October):433-60. 2 Geels, F. W.(2004). From sectoral systems of innovation to socio-technical systems. Research Policy, 33(6-7), 897–920. 3 Miles Brundage et al.‘ The Malicious Use of Artificial Intelligence: Forecasting, Prevention and Mitigation’, Future of Humanity Institute, University of Oxford, and Partners. February 2018. 4 Niti Aayog(2018) National Strategy for Artificial Intelligence- Artificial Intelligence for All. Discussion Paper 5 Head, B. W.(2008).“Wicked problems in public policy.” Public Policy 3(2): 101. See also: Camillus, J. C.(2008).“Strategy as a wicked problem.” Harvard business review 86(5): 98.

2 Artifical Intelligence for All From development and deployment to policy and strategy, conversations around AI reflect these complexities. The AI universe represents a complex set of interlocking sub-systems and issue areas, from the ownership and governance of data to the materiality of AI systems. At the heart of many debates in many, or each, of these subsystems are asymmetries in power and information; the coalition of particular interests and their ability to rally action and opinion; and value judgements about what constitutes a good and fair society and respective entitlements within it. Risks are emerging and consequences are unknown or not fully understood. Yet, the dangers of technological and policy lock-in are real, as are the risks of exploitation and misuse. Addressing wicked problems requires engaging multiple stakeholders in iterative and adaptive strategies; enabling collaborative sense-making, experimentation, and learning; and building capacities for reflexiveness and foresight. This paper takes the first step in the direction of developing‘a capacity for reflexiveness’ by engaging a range of actors through a Technology Foresight Group (TFG) in an in-depth diagnosis of the social conundrums pertaining to AI trajectories in India. This brief presents 10 social conundrums for AI trajectories in India, arising from the inherent wickedness of AI futures. The largescale deployment of AI technologies is still at an early stage in India, and impact is hard to identify and assess. Yet, the range of potential social conundrums need to be identified early and contextualised to the Indian context to be able to generate anticipatory knowledge about plausible and preferable future policy pathways.

10 social conundrums for India 3 II. 10 social conundrums for India 1. Reconciling multiple, competing social narratives Multiple and often competing narratives on AI are emerging as new knowledge is refracted through the mindsets and social frames AI has long oscillated of various social groups. between a narrative of In science fiction and popprogress and culture, public discourse on AI of moral panic. has long oscillated between a narrative of progress and of moral panic, a utopian vision of AI saving the world, or a dystopian vision in which it outsmarts and takes over human civilisation. Much of the contemporary discourse focuses on the impact of AI on jobs and the future of work. 6 While policy conversations are focused on identifying relevant coping strategies, particularly through re-skilling, in other corners, a narrative of freedom and liberation is being articulated- while jobs will be lost, this will create more time for creativity and leisure- humans may finally have the freedom they desire! 7 Yet, this remains a distant dream for many millions across the world, particularly in India, for whom work is necessary to survive; while proposals for a universal basic income and other redistributive mechanisms have been proposed, these are deeply contested, both in terms of principle as well as the practice and its likely impact. The potential for AI in warfare is setting in motion an‘AI arms race’ among global powers. A‘Winner Takes All’ frame is increasingly visible in the national strategies of a number of states. China and the US are in the lead, having invested billions of dollars towards AI research. The Russian president Vladimir Putin recently claimed, for example, that‘Artificial intelligence is the future… Whoever becomes the leader in this sphere will become the leader of the world.’ 8 With large technology companies at the forefront of much AI innovation, creating an enabling environment for the private sector is increasingly seen as a key strategy for winning this race. For the business community, appropriating AI solutions, even if only in name, is increasingly seen as critical to maintaining market competitiveness. In the past few years, the framing of‘AI for Good’ has captured the imaginations of policy makers and technology companies alike. AI is being framed as a silver bullet that can address persistent socio-economic challenges, for the benefit of society at large. 9 This imaginary is already propelling significant investments in health, education, agriculture and urban city management systems. A sense of AI solutionism seems to be driving much research and innovation— with an attempt at match-making between AI-based interventions and social challenges Lawyers, academics and activists alike, continue to raise concerns about the threats to civil liberties and the scope for discrimination, misuse, and new unknown risks. They point out that narratives are rarely neutral— they obfuscate certain beliefs and interests, while promoting others. Opposing or questioning AI is seen as stifling innovation, and now, with an‘AI for Good’ framing, as standing in the way of development. For governments and industry, the narrative of‘AI for Good’ helps invisibilise many of the dangerous effects of AI— from surveillance and warfare to new and unknown risks. Opposing or questioning AI is seen as stifling innovation, and now, with an‘AI for ----------------------------------------------------------------------------------------------------------------6 See for example: World Bank(2019) The Changing Nature of Work. World Development Report 7 See for example: Peter Frase(2016), Four Futures: Life after Capitalism, Verso Books. 8 Putin says the nation that leads in AI‘will be the ruler of the world....” 4 Sep. 2017,https://www.theverge.com/2017/9/4/16251226/russiaai-putin-rule-the-world. Accessed 30 Oct. 2018. 9 “Could AI Solve the World’s Biggest Problems?- MIT Technology Review.” 12 Jan. 2016, https://www.technologyreview.com/s/545416/couldai-solve-the-worlds-biggest-problems/. Accessed 30 Oct. 2018.

4 10 social conundrums for India Good’ framing, as standing in the way of development. Many are concerned about the sense of inevitability that characterises contemporary AI discourses— AI is akin to a discourse of modernity, of which we are all part, whether we like it or not. 2. Concentration& collusion of power The materiality of the AI universe is often overlooked— whether in terms of the menial labour 10 required to train AI systems or the natural resources required to build specific technological devices. 11 Yet, these are a clear and stark illustration of the A few‘superstar’ vast asymmetries in power global technology and wealth that underlie the companies have access AI universe. A recent study to a majority of global estimates that it will take digital data, creating 700,000 years for a child new data oligopolies working in a cobalt mine in that are being used to Bolivia 12 to earn what Jeff reshape behaviours and Bezos, founder and CEO of preferences, disrupting Amazon, earned in a single the workings of day in the first quarter of governments, markets, 2018 13 . A few‘superstar’ and communities, to global technology companies benefit only a few. have access to a majority of global digital data, creating new data oligopolies that are being used to reshape behaviours and preferences, disrupting the workings of governments, markets, and communities, to benefit only a few. 14 Whether it’s Facebook targeting depressed teens or Cambridge Analytica manipulating elections, these examples show how the interests of those deploying advanced data systems can overshadow public interest, acting in ways that are contrary to individual autonomy and collective welfare, often working in ways that are invisible and unquantifiable. 15 Further, as AI converges with the Internet of Things and the Internet of Living Things, the power of companies is likely to increase at an unprecedented rate, often without informed consent and adequate data security mechanisms. Recent policy announcements in India suggest some attempt to reign in global superstar technology companies through provisions for data localisation; a common argument of government officials — both in rhetoric and in law— is that localisation will help Indian law enforcement access data. 16 Yet, this could create new domestic oligopolies— crowding out many smaller players in the Indian market and privileging powerful incumbent players. 17 Many in civil society also note the growing nexus between state power and corporate interests, particularly technology companies. They warn that as government bodies are still grappling with understanding AI technologies and the way they work, this has given greater space for technology companies to influence public policy decision-making. They worry that, in this paradigm, citizens will have nothing to offer but their data— citizens will be reduced to a form of digital labour. ----------------------------------------------------------------------------------------------------------------10 Lilly Irani(2013),” The Cultural Work of Microwork”, New Media& Society, 0:(0 1-21 11 see for example: Anders SG Andrae(2017), Total Consumer Power Consumption Forecast, Presentation at Nordic Digital Business Summit, Helsinki, Finland, October 5, 2017 12 cobalt critical 13 Kate Crawford and Vladan Joler,“Anatomy of an AI System: The Amazon Echo As An Anatomical Map of Human Labor, Data and Planetary Resources,” AI Now Institute and Share Lab,(September 7, 2018) https://anatomyof.ai 14 Stucke, Maurice E.,"Here Are All the Reasons It's a Bad Idea to Let a Few Tech Companies...." 27 Mar. 2018, https://hbr. org/2018/03/here-are-all-the-reasons-its-a-bad-idea-to-let-a-few-tech-companies- monopolize-our-data. Accessed 30 Oct. 2018. 15 Foer, Franklin. World Without Mind: The Existential Threat of Big Tech., 2018. Print. 16 Chapter VIII. The Personal Data Protection Bill, 2018. MeitY, 2018. http://meity.gov.in/writereaddata/files/Personal_Data_Protection_Bill,2018. pdf. Accessed 30 Oct. 2018 17 Mihir Sharma,‘How Data Localisation limits possibilities for India’s startups, consumer’, Business Standard, 14 September 2018

10 social conundrums for India 5 3. Work, mobility and digital labour The deployment of machine learning technologies will reduce the need for low-medium skill labour, and increase demand for higher-order skills. This could pose multiple and contradictory challenges for India. On one hand, high skill labour constitutes a very small portion of the population— most workers are low to medium-skilled andthus are at higher risk of technological unemployment or displacement. Many of these low-medium skilled jobs, like in call centres or retail outlets, are what many youth working in informal employment within the unorganised sector aspire towards. An important opportunity for upward social-economic mobility may thus shrink as businesses adjust to new technological possibilities. 18 Yet, a new class of low-skill jobs A new class of low skill are also being created to fuel jobs are also being an AI world— in particular, data created to fuel an annotators that can label and AI world. These jobs sort data sets needed to train AI are often underpaid, systems. This‘invisible work’ or characterized by ‘artificial artificial intelligence’ 19 poor employment that is the foundation of highconditions, and entirely tech AI applications is often disconnected or underpaid, characterised by removed from the high poor employment conditions, technology they are and entirely disconnected or helping produce. removed from the artefacts it creates. 20 India could go from being the back-end for global business processing to being the back-end data annotator of the world. Already, Indian workers are one of the largest contributors to online micro-work platforms, such as Amazon Mechanical Turk. 21 AI is also drastically changing ways of work, and hiring and firing practices. A recent study showed for example, that AI technologies are likely to be used by a majority of American firms in the next decade, as a way of monitoring and improving worker productivity. 22 Workplace surveillance could be a serious concern in India, particularly where job competition is high; labour rights are poorly understood; and conversations about data privacy are at a nascent stage. Further, the use of AI for hiring new candidates and measuring their performance could improve firm-level productivity, but constrain opportunities for upward socio-economic mobility and challenge civil liberties. 4. Social/ digital identity The deployment of AI technologies risk entrenching old inequities, while creating new ones. In the Indian context, inequities exist across multiple dimensions beyond economic The deployment wealth— identity and caste are of AI technologies equally, if not more important risk entrenching old markers. A digital identity inequities, while risks limiting people’s capacity creating new ones. to move across these identity markers— for marginalised and oppressed social groups, identity fluidity can be critical for both physical safety and social mobility. 23 Further, individuals have more than one identity, and intersecting identities imply that an individual can simultaneously belong to more than one group. ----------------------------------------------------------------------------------------------------------------18 Tandem Research. 2018. Emerging Technologies& The Future of Work in India Goa: Tandem Research 19 Mary L. Grat& Siddharth Suri,“The Humans working behind the AI curtain”, Harvard Business Review, 9 January 2017 20 Hope Reese and Nick Heath,“Inside Amazon’s clickworker platform: How half a million people are being pennies to train AI’. Tech Republic 21 Neha Gupta, David Martin, Benjamin V. Hanrahan, and Jackie O’Neil,“Turk-Life in India”, Group’14, November 9-12, 2014. 22 Romy Ribitzky,“Active Monitoring of Employees Rises to 78%”, ABC News, 18 April 2018 23 Judith A. Howard,“Social Psychology of Identities”, Annual Review of Sociology, 26:2000

6 10 social conundrums for India AI is being prescribed as a tool to enable the delivery of more efficient and scalable government welfare services, replacing existing human intermediaries with automated systems. This requires the creation of an authenticable digital identity that can be recognised across multiple interconnected networks; but this also presents a reductionist approach to identity, reducing subjectives to measurable and distinct categories. Further, removing human intermediaries can result in the loss of localised awareness and sensitivity to problems and issues, dismantling existing systems of kinship and patronage. Admittedly, these systems are often exploitative, and create their own sets of winners and losers; yet, this is precisely what renders AI as a wicked problem, whereby a new set of challenges are created in the process of addressing older ones. Further, there are instances in which digital identity is taking precedence over a physical identity, as seen in cases of the denial of welfare services to individuals without an authenticated Aadhar number. 24 5. Biased robots Non-representative or biased data can further entrench existing inequities, as AI systems reproduce the representation gaps and biases of the data sets on which they are trained. 25 Data can be seen through multiple frames: the frame of the uncounted(those who don’t exist because they are not included in any sort of database), unaccounted(the portrayals of people with less inclusion into the digital world and therefore not entirely represented, maybe due to economic reasons) and discounted(they exist and are in the system but are not of interest to the people who would serve them such as governments or companies because they do not have enough money). data sets. This type of data can easily privilege socioeconomically advantaged populations, those with greater Existing data sets in access to connected devices India, whether for and online services. In India, labour markets or less than 30 per cent of India’s health records, are internet users are women and often fragmented, only 14 per cent of women outdated, or in rural Indian own a mobile unrepresentative. phone. 26 Existing data sets in India, whether for labour markets or health records, are often fragmented, outdated, or unrepresentative. 27 The impact of such data bias can be seriously damaging in India, particularly at a time of growing social fragmentation. It can contribute to the entrenchment of social bias and discriminatory practices, while rendering them invisible and pervasive through the AI systems. For example, historically certain communities were forced towards thievery due to caste discrimination and were labeled as born criminals. Alienation and stereotyping of these communities due the historical association still continues today by the police and media. 29 The AI trained on this historic data is likely to view people of these communities as thieves even if none of them currently continue to thieve. According to a 2014 report, Muslims, Dalits, and tribals make up 53 per cent of all prisoners in India; National Crime Record Bureau data from 2016 shows in some states, the percentage of Muslims in the incarcerated population was almost thrice the percentage of Muslims in the overall population. 30 If AI applications for law and order and the delivery of social justice and welfare systems are built on this data, it is not unlikely that they will be prejudiced against these groups. Data is expensive and hard to come by at scale; AI training relies on available data sets, rather than complete ----------------------------------------------------------------------------------------------------------------24 Nikhil Dey and Aruna Roy,“Excluded by Aadhaar,” The Indian Express, June 5, 2017 25 EUBANKS, VIRGINIA. Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. S.l.: PICADOR, 2019. Print 26 LIRNEasia.(2018). AfterAccess: ICT access and use in Asia and the Global South(Version 1). Colombo: LIRNEasia 27 Samarth Bansal,“From missing data to unreliable numbers, India’s statistical ecosystem needs an overhaul’, Hindustan Times, 21 September 2017. 28 Samarth Bansal,“From missing data to unreliable numbers, India’s statistical ecosystem needs an overhaul’, Hindustan Times, 21 September 2017. 29 “How Denotified Tribes In India Face Discrimination...- Youth Ki Awaaz.” 2 Aug. 2016, https://www.youthkiawaaz.com/2016/08/denotifiedtribes-discrimination-and-violence/. Accessed 30 Oct. 2018. Crime In India 2016, NCRB- National Crime Records Bureau.” 10 Oct. 2017, 30 Crime In India 2016, NCRB- National Crime Records Bureau." 10 Oct. 2017, http://ncrb.gov.in/StatPublications/CII/CII2016/pdfs/ NEWPDFs/Crime%20in%20India%20-%202016%20Complete%20PDF%20291117.pdf. Accessed 30 Oct. 2018.

10 social conundrums for India 7 6. Concentration of knowledge(and power) The increasing ubiquity of AI systems controlled by a few can also subjugate certain forms of knowledge, while creating new dependencies on technological applications. Companies and governments deploy AI systems as superior sources of credible information and insights, advocating unquestioned adherence to AI recommendations. This can end up displacing existing and hybrid knowledge systems, who are now confronted with AI outputs as objective truths. This risks creating a situation in which entire sectors and underlying knowledge systems are reconfigured in their entirety. For instance, the merger of Bayer and Monsanto can be seen as an attempt to fuse complementary data sets for soils and seeds— they can then flood the market with subsidised seeds and recommend best practices to farmers through AI analysis of the soil and seed data, arguably pushing a dependency model with the company’s best interest at heart. It is believed this alliance can corner as much as 61% of the global seed and pesticide markets, having serious implications on the local autonomy of the farmer. 31 7. Privacy frameworks: unfit for purpose? From recent controversies around the use and misuse of Aadhar data, to targeted social media messaging to influence electoral outcomes, data privacy is already an urgent concern in India. AI systems depend on gathering the maximum amount of available data and drawing correlations across a disparate and often unknowable set of data points. In this sense, AI technologies are fundamentally at odds with current privacy frameworks based on the idea of consent and data minimisation. 32 The Sri Krishna report, which articulates a draft data protection framework for India, hinges on the idea of Meaningful consent consent. 33 Meaningful consent is hard enough to is hard enough to ascertain ascertain in India, in India, where low levels of where low levels education and awareness of education and hinder capacities to exercise awareness hinder informed choice. Moreover, capacities to exercise with AI systems, data is used, informed choice. shared, and made sense of in ways that cannot even be imagined. How can consent be given in such a context? In the case of misuse or manipulation, can consent be taken back? Is this even technically possible— when individual data is being fed into complex and deeply layered algorithms, where the ways in which data is correlated remains unknown and unpredictable? Further, even the anonymisation of personal data may not be adequate— by triangulating between multiple data points, re-identification of individuals can be possible. With new IoT home appliances that record energy usage, or the use of AI for urban planning through the deployment of IoT devices, patterns about an individual lifestyle or movements can be ascertained. 34 Individuals may also choose not to share certain data about themselves, but data shared by a larger group or collective can still affect the particular individual’s agency. In other words, even digitally excluded or disconnected ----------------------------------------------------------------------------------------------------------------31 Bayer-Monsanto Analysis: EU Approval Is About Competition....& 22 Mar. 2018, https://www.bloomberg.com/view/articles/2018-03-23/ bayer-monsanto-analysis-eu-approval-is-about-competition. Accessed 30 Oct. 2018. 32 How Companies Learn Your Secrets- The New York Times.” 16 Feb. 2012, https://www.nytimes.com/2012/02/19/magazine/shoppinghabits.html. Accessed 30 Oct. 2018. 33 Chapter III. The Personal Data Protection Bill, 2018. MeitY, 2018. http://meity.gov.in/writereaddata/files/Personal_Data_Protection_Bill,2018. pdf. Accessed 30 Oct. 2018. 34 “Smart Cities May Be The Death of Privacy As We Know It – Futurism.” 7 Nov. 2017, https://futurism.com/privacy-smart-cities. Accessed 30 Oct. 2018.

8 10 social conundrums for India will be impacted by the deployment of AI systems in spheres of social life. Paradoxically, India is the one of the most connected countries in the world in terms of the number of people online, but perhaps one of the Paradoxically, India is least connected in terms of the one of the most percentages. The poorest of connected countries in poor constitute a large part of the world in terms of the unconnected population the number of people among which women are online, but perhaps often the least connected— one of the least 88% of rural Indian women connected in terms of are not digitally-connected. 35 percentages. A large A large proportion of India’s proportion of India’s population can thus be thought population can thus be of as digitally-excluded but thought of as data-included. AI systems that digitally-excluded but are deployed, especially by data-included. the government, impact the entire population immaterial of whether they are a part of the decision-making process or even if they are digitally-included. Framing data privacy policies hinges on the understanding of how personal data is viewed, as a right or property. If treated as a right, regulation needs to specify what data can be collected and traded and what data cannot. While treating it as property enables commodification of data, the collecting and trading of which is then under the control of the individual. The former holds the government responsible for protecting privacy while the latter pushes the onus on the individual, but, the individual can profit from their data. The private sector obviously favours the latter. Another option is licensing or conditional ownership, where the individual claims stake in the product or purpose for which the individual’s data is used. 8. Reporting citizens/ surveillance state A number of states in Asia, including India, are investing in mass surveillance systems— from facial recognition technologies to social media analysis cells. Government agencies are already using automated tools to allocate resources and monitor people. This raises significant concerns about civil rights and liberties. 36 Contemporary AI systems intensify practices of surveillance systems, which require the collection of massive amounts of data. Marginalised communities and populations already subjected to disproportionate government scrutiny will bear the brunt of these new surveillance technologies. 37 Some voices within civil society suggest that India risks resembling characteristics of a surveillance state; the government is a major stakeholder in the current process of data collection, and individuals are expected to be reporting citizens with little option but to comply. The state now also has an interest in data that would not otherwise be relevant, but which will now be collected since AI can process this information. With the Collection of Statistics Act, 2008, 38 refusing to part with information or providing inaccurate data to the state a punishable crime. The draft Data Protection Bill, submitted by Srikrishna Committee, also gives the state the power to access and process personal information for reasons of national security. The bills further proposes data localisation which requires companies to store all personal data in India, which the government can then access citing national security as a reason, furthering concerns among civil society about the surveillance capacities of the state. 39 ----------------------------------------------------------------------------------------------------------------35 Osama Manzar,“Rural India: Living Under Digital Exclusion”, NDTV, 6 January 2017 36 The Big Eye: The tech is all ready for mass surveillance in India....” 13 Aug. 2018, https://factordaily.com/face-recognition-mass-surveillancein-india/. Accessed 30 Oct. 2018. 37 Eubanks, Virginia. Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. S.l.: PICADOR, 2019. Print. 38 The Collection of Statistics Act, 2008. https://indiacode.nic.in/bitstream/123456789/2081/1/A2009-07.pdf. Accessed 30 Oct. 2018. 39 Chapter VIII. The Personal Data Protection Bill, 2018. MeitY, 2018. http://meity.gov.in/writereaddata/files/Personal_Data_Protection_Bill,2018.pdf. Accessed 30 Oct. 2018.

10 social conundrums for India 9 9. What/ how to govern As AI systems are deployed across a number of sociopolitical domains, the transparency and accountability of these systems is becoming an urgent concern. On the one hand, the outcomes of deep learning are unpredictable and unknowable at the outset. This renders ineffective many existing frameworks for accountability. How can the unpredictable and invisible be governed? Data used for AI-based public service delivery has been asked to be auditable, to check for biases, and provide iteratively improved solutions. Some suggest that peer review processes need to be put It is essential to in place, whereby AI algorithms ensure the output are reviewed by neutral peers. 40 and working of the Another essential check AI systems is fair, suggested is output analysis to unbiased, inclusive and ensure the output and working non-exploitative. of the AI systems is fair, unbiased, inclusive and nonexploitative. Independent state regulatory bodies might be instated to verify, test and approve of AI algorithms before being deployed in the market— similar to the FDA’s role in the pharmaceutical industry. Requiring an explanation or interpretation of AI and machine learning systems might also serve as an accountability measure. 41 There is the critical question of who should be held accountable— the developer, the designer, the deployer, or governments? Among technology companies at the forefront of AI development, the conversation about accountability tends to be framed in the language of ethics. Yet, ethics as a code is not legally enforceable and is a fuzzy point of discussion that can be agreed or disagreed with; it is a convenient way for technology companies to claim they are self-regulating. Ethics, however, cannot be substituted for legal responses to harm. While AI development is still at a nascent phase, so are understandings of the risks and unknowns around AI. We need to ask the purpose of development and deploying certain technologies, else we risk embarking on technological trajectories that will soon While AI development outpace society’s capacity for is still at a nascent control. This question of social phase we need to and technological choice must ask the purpose of figure in conversations around development and accountability. 42 Accordingly, deploying certain new interdisciplinary technologies, else knowledge clusters comprised we risk embarking of political scientists, on technological technologists, sociologists, trajectories that will anthropologists, and lawyers, soon outpace society’s among others, are a need of capacity for control. the hour. Community-based visioning and public engagement around AI should be a normative process and there is a need to push the state towards providing platforms for constructive public engagement. The Telecom Regulatory Authority of India’s(TRAI) platform for citizens to voice opinions over net neutrality is a good antecedent example. Expert activism through evidence gathering and demonstration of alternative trajectories is required, along with the traditional policy consultation methods, particularly to counter the current grip of technocrats on the decision-making process. Public engagement initiatives and public art can also help increase awareness by unveiling the cloak of invisibility that usually envelops issues around AI. ----------------------------------------------------------------------------------------------------------------40 Cognitive Bias in Machine Learning – The Data Lab – Medium.” 17 Aug. 2018, https://medium.com/ibm-watson-data-lab/cognitive-bias-inmachine-learning-d287838eeb4b; see also: Mittelstadt, Brent.“Automation, Algorithms, and Politics| Auditing for Transparency in Content Personalization Systems.” International Journal of Communication[Online], 10(2016): 12. Web. 30 Oct. 2018 41 Knight, Will.“There’S A Big Problem With AI: Even Its Creators Can’T Explain How It Works”. MIT Technology Review, 2018, https://www. technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/. Accessed 12 Nov 2018. 42 Wendall Wallach, A Dangerous Master: How to Technology from slipping beyond our control, Basic Books, 2015.

10 10 social conundrums for India 10. Reconfiguring human agency The focus in many conversations about AI is on the potential range of solutions and innovations it can deliver. AI has become an industry in itself to promote, rather than one among many tools that can be used to serve socially-identified goals. But, AI technologies also risk transforming what it means to be human, shaping not just human behaviour, but also desires and preferences, and now with the emergence of biotechnology, the fundamental biological building blocks of humans. Technological advances in genomics and synthetic biology are increasingly converging with automation, artificial intelligence, and cloud computing. What if humans were no longer required to perform the analysis, writing, and editing of DNA? 43 The merging of information technology with biotechnology will hit at the core of what it means to be human, to have the capacity for free-will and independent decision-making. 44 yet again, the focus with‘augmentation’ is on the capacities of AI rather than human needs or societal priorities. Manual scavenging, for example, continues as a regular practice in India, claiming over 200 lives in the past year alone; yet, investments towards automating such dangerous, demeaning, and dirty tasks, are few and far between. With growing evidence of the harmful impacts of AI, technologists and others have advanced the idea of ‘parenting AI’ i.e. more time and investments are needed to teach the AI, to eliminate bias and other negative consequences. 46 But, who should take on the burden of this parenting; when has it been trained adequately; and who will suffer in the meantime? Already vulnerable and marginalised communities are likely to bear the burden and dangers of unparented AI, reducing the incentives for timely and rigorous parenting. This suggests that we need to The notion of look beyond the applications of ‘augmentation’ is AI to the kind of relationship we increasingly being have with AI. This may enable enrolled to address putting humans at the centre concerns about AI of the conversation, their desire taking over human and needs, rather than just jobs, or humanity the technological possibilities more generally; yet and limitations of AI. Data, again, the focus with for example, is often treated ‘augmentation’ is on as a disembodied subject, the capacities of AI dissociated from individuals. 45 rather than human Yet, data is ultimately about needs or societal the lives of real people, their priorities. needs, preferences, and beliefs. The notion of‘augmentation’ is increasingly being enrolled to address concerns about AI taking over human jobs, or humanity more generally; ----------------------------------------------------------------------------------------------------------------43 Garrett Dunlap and Eleonore Pauwels,“The Intelligent and Connected Bio-Labs of the Future: Promise and Peril in the Fourth Industrial Revolution”, Wilson Briefs, September 2017. 44 Yuval Noah Harari, 21 Lessons for the 21st century, Random House 2018 45 see for example: Martin Dodge& Rob Kitchin,“ The Ethics of Forgetting in an Age of Pervasive Computing”, CASA Working Paper Series, No. 92, 2005. 46 See for example: Lila Tretikov,“ We should not raise AIs like parents, not programmers- or they’ll turn into terrible toddlers”, Quartz, 5 January 2018

Conclusion: Building a capacity of reflexiveness 11 III. Conclusion: Building a capacity of reflexiveness Responsible research and innovation: The policy discourse on‘responsible research and innovation’ has emerged over the last decade in Europe and elsewhere as policy makers and policy analysts grapple with the special challenge of regulating technologies characterised by both technical and social uncertainty, and complexity and ambiguity; these technologies Technology assessment require fundamentally different in‘real-time’ is policy-making processes imagined in contrast and approaches. 47 These to the linear model conversations have their of innovation policy; roots in earlier discussions and an awareness about the ethical and social and consideration for implications of areas such as stakeholder values, nanotechnology and genomics, aspirations and risk and call for the need to perceptions rather than address the social and ethical management of just dimensions of technology ‘technical’ risks. and innovation early on. The framework of anticipatory governance 48 emphasises the need for deliberation on the social conundrums of technology at an early stage of the policy conversation. Various forms of public engagement and open opportunities for deliberation can help build a‘capacity for reflexiveness’ in science and technology institutions and decision-making processes. 49 A framework for reflexivity for AI development in India would need to have five key elements: 1. Thinking both technically and socially: The dominant narrative around AI in India frames the development and use of AI for social good, or otherwise, as a technical problem— to be addressed through the creation of better innovation and startup ecosystems and investments in technical manpower. A range of complex social concerns— access, equity, privacy and power— are acknowledged, for example in the governments AI strategy, but not addressed and often brushed under the carpet. Policy makers and policy analysts will need to pay attention to both the social and technological issues around AI and support better socio-technical integration. 2. Anticipation and futuring: Extant policy approaches are limited in managing emerging and socially-challenging technologies such as AI because of the imperative to demonstrate short term policy impact and success. AI is not likely to be a silver bullet, which could be deployed instrumentally to solve problems: long-term strategies will be needed to navigate the complex social and technical challenges to applying AI to address persistently insoluble challenges like health and education. Reflexivity about social conundrums around AI will be needed to identify the diverse plausible trajectories in order to formulate policies to navigate towards desirable socio-technical futures. 3. Knowledge systems: While the temples of technological research in India— the IITs and IIITs— are gearing up to develop AI applications, social science research and knowledge systems around AI are less developed and attract limited funding support from either the public or private sector. More investment and capacity is needed for research on complex social and ethical issues around AI development, to a point where they can be meaningfully assessed before wide-scale deployment. ----------------------------------------------------------------------------------------------------------------47 R.Owen, P. Macnaghthen and J. Stilgoe,“ Responsible Research and Innovation- from Science in Society to Science for Society”, Science and Public Policy, 39(2012) 48 Risto Karinen and David H Guston,“Towards Anticipatory Governance: The Experience with Nanotechnology”, in M. Kaiser et al., Governing Future Technologies: Nanotechnology and the Rise of an Assessment Regime, Springer 2010 49 Guston, D. H.,& Sarewitz, D.(2002). Real-time technology assessment. Technology in Society, 24(1-2), 93-109

1 2 Conclusion: Building a capacity of reflexiveness 4. Policy portfolio and experiments: Responsible research and innovation on AI will require strategies and policies that cut across traditional decision-making silos in India. AI is being primed for use in agriculture, education and health– but any applications will need to be built upon an understanding of both the needs of the AI‘users’ in these sectors and the institutional and policy context of the challenges that the sectors face. The NITI Aayog has proposed‘proof of concept’ pilots for AI applications across sectors, but untested technical fixes will not address the policy failures that plague these sectors. 5. Deliberative decision making: Globally, many models of technology assessment based on public engagement, participation and deliberation— for example, through consensus conferences— are being tested. 50 There also is growing evidence that in the case of ethically problematic and socially complex technologies, early engagement with the public, on the wider dimensions of the technology, leads to more socially acceptable design and development. Widely-debated norms and general governance principles around development and deployment of emerging technologies are likely to be more widely-accepted. In a democratic political context, like India, deliberative process can lead to more dynamic technology strategies and can help avoid static regulatory and legal approaches to technology control. The complex social dimensions around the research, development and deployment of AI need to understood, rather than brushed aside. Innovation and regulatory frameworks will need to co-evolve. This should be designed to happen in such a way that there is an opportunity to progressively strengthen mutual expectations and collaboration, and buy-in of all the parties involved: government, industry, academia, civil society organisations and wider publics. A continuous interaction between research, experimental action, regulation and assessment of this kind, within a framework to which public deliberation will make a central contribution, will support AI trajectories that align with societal goals. ----------------------------------------------------------------------------------------------------------------50 Simon Joss and John Durant eds. Public Participation in Science: The Role of Consensus Conferences in Europe, Science Museum, 1995.

About the authors Urvashi Aneja is the Founding Director of Tandem Research. She works on the governance and sociology of emerging technology; southern partnerships for humanitarian and development assistance; and the power and politics of global civil society. Vikrom Mathur is the Founding Director of Tandem Research. Vikrom is an anthropologist of science and technology. His diverse research interests include the governance of emerging technologies, social and cultural dimensions of technological transitions. Abishek Reddy K is a Research Fellow at Tandem Research where he leads the AI& Society initiative. His current work is focused on the technological trajectories and social effects of Artificial Intelligence. The views expressed in this publication are not necessarily those of the Friedrich-Ebert-Stiftung. Imprint © 2019 Friedrich-Ebert-Stiftung India Office K-70-B, Hauz Khas Enclave| New Delhi-110016 India Responsible: Patrick Ruether| Resident Representative Mandvi Kulshreshtha| Program Adviser T:+ 91 11 26561361-64 www.fes-india.org FriedrichEbertStiftungIndia To order publication: info@fes-india.org Commercial use of all media published by the Friedrich-Ebert-Stiftung(FES) is not permitted without the written consent of the FES. https://www.fes-asia.org/ Friedrich-Ebert-Stiftung(FES) is the oldest political foundation in Germany. The foundation is named after Friedrich Ebert, the first democratically elected president of Germany. FES is committed to the advancement of both socio-political and economic development in the spirit of social democracy, through civic education, research, and international cooperation. This publication is part of a series on Artificial Intelligence in India, a collection of papers developed under the Socio-Economic Transformation program coordinated by the FES India office.