A N A LYS I S Payal Arora and Weijie Huang January 2025 Gender Data: What is it and why is it important for the future of AI Systems? Competence Centre on the Future of Work

Imprint Published by Friedrich-Ebert-Stiftung e.V. Godesberger Allee 149 53175 Bonn, Germany info@fes.de Issuing Department Competence Centre on the Future of Work Cours Saint Michel 30a, 1040 Brussels, Belgium For more information about the Competence Centre on the Future of Work, please consult: https://www.futureofwork.fes.de Responsibility for Content and Editing Dr. Inga Sabanova inga.sabanova@fes.de Design/Layout pertext| corporate publishing www.pertext.de The views expressed in this publication are not necessarily those of the Friedrich-Ebert-Stiftung(FES). Commercial use of media published by the FES is not permitted without the written consent of the FES. Publications by the FES may not be used for electioneering purposes. January 2025 © Friedrich-Ebert-Stiftung e.V. ISBN 978-3-98628-687-3 Further publications of the Friedrich-Ebert-Stiftung can be found here: ↗ www.fes.de/publikationen

Payal Arora and Weijie Huang January 2025 Gender Data: What is it and why is it important for the future of AI Systems?

Content Summary.......................................................  3 Background.....................................................  3 What is gender data?.............................................  4 AI and gender discrimination.......................................  4 How can gender data enable AI systems?.............................  5 Examples in healthcare, employment and education ...................  5 Overcoming the barriers to building gender data to train AI systems.......  6 Barriers to gender data ...........................................  6 Breaking down barriers..........................................  6 Policy recommendations ............................................  7 Conclusion and call for action .......................................  7 References.......................................................  8 2 Friedrich-Ebert-Stiftung e.V.

Summary Background Gender bias in AI systems poses significant challenges to achieving equality. AI, dependent on data for training and decision-making, often perpetuates gendered inequalities due to biased datasets that reinforce stereotypes and exclude diverse experiences. This impacts women’s access to healthcare, employment and education, compounding structural inequities. Addressing these biases requires integrating gender-sensitive data throughout AI development to ensure fairness and equity. Barriers include technical opacity, binary gender norms and patriarchal structures, particularly in the Global South. Solutions involve algorithmic transparency, inclusive data frameworks, cross-sectoral collaboration and feminist ethics. Embedding gender data in AI is essential to empower marginalised groups and promote social justice. Gender equality is achieved when women and men enjoy the same rights and opportunities across all sectors of society, including economic participation and decision-making, and when the different behaviours, aspirations and needs of women and men are equally valued and favoured. However, the existing artificial intelligence (AI) development process from design to deployment perpetuates gender bias, which can significantly disadvantage women’s lives in access to knowledge, resources and opportunities. AI depends on data to learn patterns, make predictions and improve its accuracy, but the quality and quantity of data directly influence its performance and outcomes. When an AI algorithm is trained on data that embodies traditional gender roles and stereotypes, its results may be biased and perpetuate those same norms and assumptions. This can result in unfair practices and decisions that adversely affect women and marginalised communities. Tackling gender bias in AI and data requires proactive efforts to ensure diverse representation in data collection, algorithm design and decision-making processes. Generating high-quality statistics relies on eliminating gender bias at all stages of the production process. Summary 3

AI and gender discrimination What is gender data? “Artificial intelligence”, or the simulation of human intelligence in machines designed to perform tasks that typically require human cognition, such as learning, reasoning and problem-solving, has permeated our everyday lives. AI is increasingly applied to decision-making across areas such as hiring, healthcare, education, security and resource allocation. Yet beneath its promise of efficiency and innovation in the industry, these systems often perpetuate gender discrimination. The rise of these biases is due to AI models being trained and optimised by existing large datasets which are full of missing and incomplete data on women and misrepresentations that often fail to consider their needs. Recent studies across sectors are revealing the negative impacts of gender data bias on AI deployment and use, including in healthcare(Baum­ gartner et al., 2024), education(Arora, 2024) and hiring/ employment(Capasso et al., 2025). As gender biases accumulate in AI decision-making processes, AI systems can reinforce structural inequalities. This can transcend individual women’s lives and significantly affect the larger political and economic structure of society. This phenomenon comes from historical biases – such as the underrepresentation of women in leadership and their invisibility in positions of power – and has been embedded in societal power relations, preventing equal opportunities for women and other marginalised groups in resource allocation, economic participation and political representation. While women have been historically discriminated against and disadvantaged in many fields, gender bias in AI applications can amplify these disparities and have more severe consequences. From digital healthcare misdiagnosing women, to recruitment tools favouring male candidates and financial algorithms preventing female entrepreneurs from accessing investment, this gendered impact can be compounding. Hence, it is crucial to address these gender biases when building AI-enabled systems, because it ensures that models are trained on diverse, representative data that capture gender-specific needs, behaviours and disparities. “Gender data” is information that provides insights into the differences in experiences, needs and outcomes among various genders, essential for understanding and addressing gender disparities when designing our tools, services, platforms and institutions(data.org, 2024). The“male norm” in datasets is a default across contexts (Perez, 2019), skewing tools, services and policies to wards male-centred experiences, often overlooking the distinct lived realities of other genders. This bias leads to designs that fail to account for gender-specific differences, perpetuating inequities in health, safety, economic access and overall well-being. For example, a Europe-wide FES project that examined the social impact of Covid-19 on gender in 15 countries found that gender care gaps and pay gaps increased as women performed more care work. This led to policy recommendations around increasing the supply of childcare and extending childcare opening times(Wiesner, 2023). According to the UN Women report, we lack data for 80 per cent of the gen der equality indicators across the Sustainable Development Goals(2022). International institutions and stakeholders such as the United Nations, the Clinton Foundation and the Bill and Melinda Gates Foundation have called for closing the “gender data gap” to help achieve gender equality(Colaço& Watson-Grant, 2021). However, it is essential to recognise cultural factors that help identify the gender gaps in our datasets. For instance, there are data deficits on women and girls’ sexual behaviour in socially conservative contexts in the Global South. These gaps demand novel and creative approaches in accessing these groups and getting them to participate in such data building efforts(Arora, 2024). 4 Friedrich-Ebert-Stiftung e.V.

How can gender data enable AI systems? The International Labour Organization(ILO) points out that unbiased data are critical for identifying and reducing gender-based inequalities(Watson& Gardner, 2023). The core of AI systems are algorithms, which AI systems rely on to learn data patterns and generate decisions. However, algorithms are not objective; they rely on training data and inherent biases in the datasets. Therefore, gender data ensure that AI systems can be trained on unbiased datasets. Gender data allow AI systems to identify and address gender issues and consider gender-based disparities, improving the accuracy and fairness of their decision-making in fields such as healthcare, employment, education and beyond. One of the main challenges is that most global data collection focuses on binary cisgender sex-disaggregation classifications(Colaço& Watson-Grant, 2021). When AI systems are trained on datasets that lack gender sensitivity, the algorithms tend to amplify existing social biases present in the datasets. These algorithmic biases include presentation bias, filter bias, selection bias, historical bias, aggregation bias and interaction bias – which further the gender gap by limiting access to resources and opportunities for women and marginalised communities (Gutierrez, 2021). For example, biased algorithms reinforce harmful gender roles by associating women with housework in images, further increasing social stereotypes. Incorporating gender data into AI systems will not only assist in reconstructing societal power relations, but also improve the traditional gender hierarchy and promote fair distribution in society. By integrating gender data, AI systems can better recognise and address gender biases, leading to more equitable outcomes in applications ranging from healthcare diagnostics to employment and social services. Without it, AI systems risk reinforcing existing inequalities and overlooking the unique experiences and challenges faced by different genders in varied contexts. Examples in healthcare, employment and education To illustrate the importance of gender data in AI, we need to examine current practical examples of how gender data AI systems across industries can reduce healthcare, employment and education inequalities. AI systems have shown significant potential in diagnosis and personalised treatment in healthcare. However, studies revealed that many medical AI systems fail to accurately predict the health of people from economically disadvantaged backgrounds due to a lack of intersectional data(Baumgartner et al., 2023). Datasets for medical AI systems favour middle-class and high-income people in developed countries, resulting in a lack of gender and ethnic data in digital health records. The underlying causes of pain in people of colour and myocardial infarctions in women cannot be effectively diagnosed, as AI systems are trained on data centred on white males. To address this problem, it is necessary to integrate gender and ethnic data into medical datasets. This could improve the diagnostic accuracy of AI systems, provide more personalised care and improve treatment for women from marginalised groups. In employment, AI systems have exposed deep-rooted biases in recruitment and gig platforms. For example, female drivers in India who are from lower-income and marginalised communities often face obstacles such as cultural norms, lack of necessary documents and safety issues when using online ride-hailing platforms such as Uber and Ola for flexible income opportunities(Bansal et al., 2023). This results in female drivers being pushed into lower-paid and less secure roles. The problem is that the platform fails to incorporate gender-sensitive data and reinforces systemic exclusion. Incorporating gender data into platform algorithms, designing safety measures and tailoring recruitment strategies will provide women with safer employment opportunities and promote more inclusive economic development. Gender bias in AI within education is a significant concern. Studies revealed that course recommendation algorithms exhibit gender bias towards female students(Chinta et al., 2024). These recommendation systems tend to steer male students towards STEM(science, technology, engineering, mathematics) fields, and female students towards humanities and social sciences, reinforcing social stereotypes. Moreover, the system may lead to fewer advanced courses being recommended for minority students. The consequences of these biases limit the academic and career opportunities of women and minorities in high-paying and traditionally male-dominated fields. To mitigate these biases, it is essential to incorporate gender-disaggregated data and fairness-aware algorithms into AI systems, ensuring more inclusive and equitable educational experiences, regardless of students’ gender and ethical background. In addition, regular review and adjustment mechanisms are key to ensuring educational equity. How can gender data enable AI systems? 5

Overcoming the barriers to building gender data to train AI systems Barriers to gender data While integrating gender data into AI systems is essential for promoting inclusivity, numerous barriers hinder its collection and use in AI development. Historically, data collection processes come in the form of gender binary templates, leaving little space for non-binary and gender-fluid identities, and limiting the diversity of gender data(Colaço& Watson-Grant, 2021). Technical barriers increase these structural problems. Many AI systems function as opaque“black boxes” with a lack of transparency as to what kinds of data go into the training of their algorithms (Baumgartner et al., 2023). This opacity creates signifi cant obstacles in identifying and understanding the gender data gaps, and correcting these biases in practical applications. In addition, gender-disaggregated data between countries, platforms and industries reduce AI models’ accuracy and limit the effectiveness of bias mitigation strategies. Another concern is the societal barriers. Deep-rooted patriarchal cultural norms, gender roles and structural inequalities still exist in many Global South countries. In male-dominated industries, people who work in leadership positions do not prioritise or acknowledge the need for gender data due to a lack of interest or motivation. In the end, women are excluded from male-dominated fields, further perpetuating existing biases. Breaking down barriers Measures can be taken to effectively overcome barriers to collecting and using gender data in AI systems. First, algorithmic transparency is increasingly a prerequisite for AI regulations globally. Regulators are requiring AI systems to be explainable to ensure the decision-making process is accountable. Second, data collection frameworks should include diverse gender identities beyond the binary gender category and ensure that these different experiences are captured. Third, cross-sectoral collaboration, especially between computer scientists and data feminists, should be promoted to ensure that AI systems are designed with a gender perspective. Finally, industries should establish gender-sensitive continuous monitoring and evaluation mechanisms to ensure that AI systems are inclusive in different contexts. 6 Friedrich-Ebert-Stiftung e.V. The Horizon Europe-funded Box 1 FINDHR project provides effective solutions for pre-deployment testing and post-deployment monitoring of gender bias in AI-enabled recruitment systems. For pre-deployment testing, computer scientists in collaboration with humanities scholars created a semi-synthetic dataset based on more than 1,100 donated CVs. This can then be used to test for biases introduced by automated CV ranking or selection systems. The dataset is artificially generated data that mimic data patterns from these gendered datasets, without containing actual personal or sensitive information. The development of the dataset focuses on ensuring that historically marginalised groups are not ignored and introduces as much variability as possible to achieve diversity while protecting privacy. For post-deployment monitoring, FINDHR provides an equality monitoring protocol for third-party monitoring of biases in AI-based recruitment pipelines. The protocol is accompanied by software, documentation and legal analysis. Finally, FINDHR mobilises civic communities invested in promoting and building gender data to embed such practices via masterclasses and feminist participatory action research interventions(Capasso et al., 2025).

Policy recommendations Conclusion and call for action Governments should develop frameworks and commit resources to address the gender data gap to promote social equity. Governments should also require all AI-driven public services to conduct gender equality assessments early on in development to enable the detection and correction of biases in design. Policies should promote gender data collection and require algorithms to be transparent and explainable to detect and correct gender bias. In addition, industry leaders should promote feminist ethics in AI development through public-private partnerships, ensuring the building and securing of gender data and use in ways that empower marginalised gender groups. Moreover, the industry needs to invest in monitoring and correcting algorithmic bias in AI processes, with a special focus on gender bias. Last, the industry should increase leadership diversity and encourage women to participate in AI development. Integrating gender data into AI systems is a technical necessity and an ethical requirement for ensuring fair outcomes in society. As AI increasingly influences societal structures, their development must reflect the diversity of the populations they serve. Without gender data, these systems risk reinforcing the inequalities they should eliminate. Overcoming technical and social barriers to gender data collection requires collaboration. Policymakers, industry leaders and technologists must prioritise transparency and implement ongoing monitoring to mitigate gender bias. Treating inclusivity as an addon is not effective; gender data must be embedded in every stage of AI development. In the technological landscape, AI has the potential to either reinforce discrimination patterns or serve social justice. Given that technologies including AI are in the making, we have the opportunity to intervene strategically. We can train AI with new gender data that are representative of diverse and intersectional genders with their needs, concerns and aspirations. As they say, garbage in is garbage out. As a corollary, we can replace historically gender-biased data with new datasets that can shape our products, services, platforms and policies in ways that empower women and other genders. The tools are already here; it is up to us to ensure they are steered in the right direction. Policy recommendations 7

References Arora, P.(2024). From Pessimism to Promise: Lessons from the Global South on Designing Inclusive Tech. MIT Press. Bansal, P., Arora, P.,& Raman, U.(2023). Feminist design framework for envisioning gender-inclusive ride-hailing sector: Perspectives from India. Baumgartner, R., Arora, P., Bath, C., Burljaev, D., Ciereszko, K., Custers, B., Ding, J., Ernst, W., Fosch-Villaronga, E., Galanos, V., Gremsl, T., Hendl, T., Kropp, C., Lenk, C., Martin, P., Mbelu, S., Bruss, S. M. D. S., Napiwodzka, K., Nowak, E., … Williams, R. (2023). Fair and equitable AI in biomedical research and healthcare: Social science perspectives. Artificial Intelligence in Medicine, 144, 102658. https://doi. org/10.1016/j.artmed.2023.102658 Capasso, M., Arora, P., Sharma, D.& Tacconi, C.(2025). On the right to work in the age of artificial intelegence: Ethical safeguards in algorithmic human resource management. Business and Human Rights Journal(2025): 1–15. Chinta, S. V., Wang, Z., Yin, Z., Hoang, N., Gonzalez, M., Quy, T. L.,& Zhang, W. (2024). FairAIED: Navigating fairness, bias, and ethics in educational AI applications. arXiv preprint arXiv:2407.18745. Colaço, R.,& Watson-Grant, S.(2021). A global call to action for Gender-Inclusive data collection and use. https://doi.org/10.3768/rtipress.2021.pb.0026.2112 data.org(2024) Understand and Build Gender Data. https://data.org/playbooks/­ gender-data-and-climate/c/understand-and-build-gender-data/ Gutierrez, M.(2021). New Feminist Studies in Audiovisual Industries| Algorithmic Gender Bias and Audiovisual Data: A Research Agenda. International Journal of Communication, 15, 23. https://ijoc.org/index.php/ijoc/article/view/14906/3333 Perez, C. C.(2019). Invisible Women: Data bias in a world designed for men. https://www.amazon.com/Invisible-Women-Data-World-Designed/dp/1419729071 UN Women(2021, September 17). Progress on the Sustainable Development Goals: The gender snapshot 2022. https://www.unwomen.org/en/digital-library/publications/­ 2022/09/progress-on-the-sustainable-development-­goals-the-gender-snapshot-2022 Watson, S.,& Gardner, J.(2023, March 20). Breaking the bias for better gender data. ILOSTAT. https://ilostat.ilo.org/blog/breaking-the-bias-for-better-gender-data/ Wiesner, C.(2023). The Gendered Impact of the Covid-19 Pandemic. Friedrich-EbertStiftung. https://nordics.fes.de/e/the-social-impact-of-covid-19.html . 8 Friedrich-Ebert-Stiftung e.V.

About the Authors Payal Arora is a Professor of Inclusive AI Cultures at Utrecht University and co-founder of FemLab, a Femi nist Futures of Work initiative, and Inclusive AI Lab, a Global South-centred debiasing data initiative. She is a digital anthropologist with two decades of user experience in the Global South. She is the author of several award-winning books, including The Next Billion Users (Harvard Press) and From Pessimism to Promise: Lessons from the Global South on Designing Inclusive Tech (MIT Press). Forbes called her the“next billion champion” and the”right kind of person to reform tech”. Weijie Huang is a PhD candidate at the Faculty of Humanities at Utrecht University and Feminist AI cluster lead at the Inclusive AI Lab. Her research focuses on intersecting media, AI and gender studies in China, with a special interest in how AI and digital platforms shape female players’ experiences and belongingness. Weijie holds a master’s degree in Media and Creative Industries from Erasmus University Rotterdam. Prior to her PhD, she worked for a decade as a journalist for the Chinese National Geography and Condé Nast China.

Gender Data: What is it and why is it important for the future of AI systems? Gender bias in AI systems poses significant challenges to achieving equality. AI, dependent on data for training and decision-making, often perpetuates gendered inequalities due to biased datasets that reinforce stereotypes and exclude diverse experiences. This impacts women’s access to healthcare, employment and education, compounding structural inequities. Addressing these biases requires integrating gender-sensitive data throughout AI development to ensure fairness and equity. Barriers include technical opacity, binary gender norms and patriarchal structures, particularly in the Global South. Solutions involve algorithmic transparency, inclusive data frameworks, cross-sectoral collaboration and feminist ethics. Embedding gender data in AI is essential to empower marginalised groups and promote social justice. Further information on the topic can be found here: ↗ fes.de