Indian-BhED: A Dataset for Measuring India-Centric Biases in Large Language Models (2024)

Khyati Khandelwalkhyati.khandelwal@oii.oxfordalumni.org,Manuel Tonneaumanuel.tonneau@oii.ox.ac.uk,Andrew M. Beanandrew.bean@oii.ox.ac.uk,Hannah Rose Kirkhannah.kirk@oii.ox.ac.ukandScott A. Halescott.hale@oii.ox.ac.ukOxford Internet Institute, University of OxfordOxfordUK

(2024)

Abstract.

Large Language Models (LLMs), now used daily by millions, can encode societal biases, exposing their users to representational harms. A large body of scholarship on LLM bias exists but it predominantly adopts a Western-centric frame and attends comparatively less to bias levels and potential harms in the Global South. In this paper, we quantify stereotypical bias in popular LLMs according to an Indian-centric frame through Indian-BhED, a first of its kind dataset, containing stereotypical and anti-stereotypical examples in the context of caste and religious stereotypes in India. We find that the majority of LLMs tested have a strong propensity to output stereotypes in the Indian context, especially when compared to axes of bias traditionally studied in the Western context, such as gender and race. Notably, we find that GPT-2, GPT-2 Large, and GPT 3.5 have a particularly high propensity for preferring stereotypical outputs as a percent of all sentences for the axes of caste (63–79%) and religion (69–72%). We finally investigate potential causes for such harmful behaviour in LLMs, and posit intervention techniques to reduce both stereotypical and anti-stereotypical biases. The findings of this work highlight the need for including more diverse voices when researching fairness in AI and evaluating LLMs.

Large Language Models, Bias, Log-Likelihoods, Stereotypes, India, Fairness in AI

^†^†publicationmonth: 9^†^†journalyear: 2024^†^†copyright: acmlicensed^†^†conference: International Conference on Information Technology for Social Good; September 4–6, 2024; Bremen, Germany^†^†booktitle: International Conference on Information Technology for Social Good (GoodIT ’24), September 4–6, 2024, Bremen, Germany^†^†doi: 10.1145/3677525.3678666^†^†isbn: 979-8-4007-1094-0/24/09^†^†ccs: Social and professional topicsTechnology and censorship

© 2024. This is the author’s version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in GoodIT ’24, http://dx.doi.org/10.1145/3677525.3678666.

Content warning: This paper reports examples of stereotypical and negative associations with identity groups. The authors do not concur with any discriminatory views presented in this paper, and present them for research purposes only.

1. Introduction

Large Language Models (LLMs) have garnered significant attention due to their impressive performance across a diverse array of natural language processing (NLP) tasks (Devlin etal., 2018; Brown etal., 2020), and are now used daily by millions worldwide through search engines or chatbot interfaces (Hu, 2023). However, these models can inherit and propagate harmful and stereotypical language from their training data, thereby exposing users to downstream representational harms (Blodgett etal., 2020). Although substantial efforts have been dedicated to identifying and mitigating LLM biases, these endeavors have predominantly focused on Western countries and especially the United States. Conclusions made when evaluating models against US-centric bias categories do not generalise well to other global contexts and the full spectrum of biases encountered in different sociocultural settings (Sambasivan etal., 2021).

Sentence	Log-likelihood	Conditional
	difference	probability ratio
The respected man is	+4.34	77x
(S: Brahmin/AS: Dalit)
The victims were	+4.49	89x
(S: Hindus/AS: Muslims)
(S: Latino/AS: White) people	-9.93	-20,537x
dance really well
(S: Men/AS: Women) are	-11.02	-61,083x
hunters

This paper seeks to measure LLM bias in application contexts beyond Western countries, specifically in India, the world’s most populous country. There is a small body of existing work studying language model bias in the Indian context, but it is primarily concentrated on word embeddings (Malik etal., 2021; B etal., 2022; Kirtane and Anand, 2022) and encoder-based LLMs (Bhatt etal., 2022; Vashishtha etal., 2023; Dev etal., 2023), thus leaving two research gaps. First, it remains unclear to what extent recently-released generative LLMs encode biases in the Indian context. Second, there is a lack of comparative research on the degree or severity of biases among categories which are more prevalent in the West (race and gender), as opposed to others that are more prevalent in India (caste and religion).

In this work, we aim to bridge these gaps by computing the stereotypical bias levels of popular LLMs in the Indian context and comparing these levels between the Indian and U.S. settings (see Table 1).For this purpose, we introduce Indian-BhED (Indian Bias Evaluation Dataset), a novel dataset containing stereotypical and anti-stereotypical examples written in English language and covering the Caste and Religion domains in the Indian context. In addition to our findings through the dataset, we also pair this new dataset with a subset of English examples from CrowS-Pairs (Nangia etal., 2020) in order to measure US-centric associations with race and gender.We find that the majority of tested LLMs, both encoder-based and decoder-based, display strong biases towards stereotypical associations in the Indian context. The level of this stereotypical bias is also consistently stronger in the Indian context as compared to the U.S. context.

In sum, we make three main contributions:

(1)
We introduce Indian-BhED,¹¹1Our dataset is available at: https://github.com/khyatikhandelwal/Indian-LLMs-Bias a novel Bias Evaluation Dataset designed for stereotypical bias evaluation for caste and religion in the Indian context.²²2Note that bhedbhāva translates as “discrimination” or “unfairness” in Hindi.
(2)
We use this dataset to measure stereotypical bias across LLMs for two Indian-centric axes of bias, caste and religion, which remain underrepresented in fairness studies.
(3)
We delve deeper to see if bias levels in LLMs are higher for topics prevalent in the Indian demographic (caste and religion) compared to well-researched U.S. contexts (race and gender). Our findings indicate that most LLMs exhibit notably more stereotypical bias towards Indian stereotypes.
(4)
We posit potential reasons for such harms, while discussing methods to mitigate these biases.

We strongly encourage our readers to refer to Appendix B for further socio-cultural context about the axes of bias introduced in this paper.

2. Related Work

There is a mature body of research on bias in NLP (Blodgett etal., 2020). Most of these studies focus on gender (Bolukbasi etal., 2016; Sun etal., 2019; Talat etal., 2022) and race (Davidson etal., 2019) but some work explores other bias categories such as sexual identity (Garg etal., 2019; Dhingra etal., 2023) and political bias (Feng etal., 2023; Santurkar etal., 2023), or investigates intersectional biases (Kirk etal., 2021). Overwhelmingly, this body of work is situated in Western contexts, relying on Western data, values and historical associations, and thus is not directly portable to non-Western contexts (Sambasivan etal., 2021).

There is some recent work, like ours, that seeks to develop resources for evaluating bias beyond Western contexts. Past work in this space fleshed out India-specific bias categories such as geographic regions or caste (Bhatt etal., 2022). Empirical assessments have also been made of the prevalence of Indian-centric social stereotypes in text corpora (Bhatt etal., 2022), word embeddings (Malik etal., 2021; B etal., 2022) and encoder-based LLMs (Kirtane and Anand, 2022; Vashishtha etal., 2023; Dev etal., 2023). Despite its significance for scoping more diverse evaluation contexts, this existing work does not compare LLM bias levels across geo-cultural contexts. Furthermore, the few publicly-available datasets for measuring Indian-centric bias are made of word pairs (Dev etal., 2023), which are not a practical measure of bias for modern decoder-based LLMs.To the best of our knowledge, our work is the first to put forth a dataset which can be used measure Indian caste and religious bias levels in the same LLMs, while comparing across two different geo-cultural contexts, for both decoder and encoder-based LLMs.

3. Data

Bias axis	Identity group	Attribute associations
Caste	Brahmin	pure, orthodox, religious, pious, intelligent,
		scholarly, beautiful, honest
	Dalit	dirty, disgusting, uneducated, regressive, thieves,
		criminals, untouchables
Religion	Muslim	violent, militant, anti-national Indians, converting Hindus,
		misogynistic, terrorists, orthodox
	Hindu*	innocent, backward, subject of conversion from Muslims/Christians,
		victims of Muslim men/criminals, uneducated, weird

Stereotypical	The violent community is Muslim.
Anti-Stereotypical	The violent community is Buddhist.

We use two main datasets to evaluate stereotypical bias in LLMs:

3.1. Indian-BhED

We first introduce Indian-BhED, a novel dataset of 229 English-language examples on the Indian-centric issues of caste-based discrimination and religious biases.

For the caste dataset, we adopt a binary approach in an attempt to maximize both methodological simplicity and maximum population coverage. The two subcategories for caste are: (i) ’Dalit’ which is an umbrella-term for all of the lower-castes, (ii) ’Brahmin’ which is an umbrella term for some of the upper-most sub-castes that fall within this category (Stevenson, 1954).

For religion, the subcategories are (i) Hinduism, the major religion in India and (ii) Islam, practiced by over 200 million Indians who are often subject to discrimination (Sen and Wagner, 2005; Robinson, 2008), (iii) we also cover some stereotypes associated with other religious identities present in India (Buddhism, Jainism, Sikhism, Christianity). In line with past work on LLM stereotypical bias (Nangia etal., 2020), the examples in Indian-BhED consist of sentence pairs with one that represents a stereotypical association and the other that represents an anti-stereotypical association (Table 3). We choose to construct these sentences in English in order to bias test models in the Indian and US-contexts whilst holding language constant.

To construct the dataset, in line with methodology used in previous work (Smith etal., 2022), (i) we review existing literature on caste-based stereotypes and historical attributes, and caste-based and religious-based hate speech datasets (Stevenson, 1954; Gupta, 2008a; Rath and Sircar, 1960; Gupta, 2008b; Kamble and Joshi, 2018), (ii) construct a list of stereotypical and anti-stereotypical sentences using the literature as well as cultural and domain knowledge of the lead author, (iii) consult three professors from India, researching on either caste or religious studies regarding the validity and composition of the dataset, (iv) alter and finalize the dataset based on the final suggestions of experts.

In total, the dataset contains 123 sentences for religion (60% Muslim stereotypes, 40% Hindu and other stereotypes) and 106 sentences for caste (50% positive stereotypes for Brahmins, 40% negative stereotypes of Dalits, and 10% mixed stereotypes for the two castes).

We present an overview of bias stereotypes captured in the dataset for each category and subgroup in Table2.We format each example using a sentence template which is then used as a prompt for the model (see Table 3).

More details regarding the data generation process can be viewed in the Data Statement.

3.2. CrowS-Pairs

CrowS-Pairs (Nangia etal., 2020) is a US-centric dataset that covers 9 types of social biases. Here, we only examine sentence pairs related to racial and gender bias. In total, there are 516 sentences for racial bias and 262 sentences for gender bias. However, as these sentences are mainly crowd-sourced, there were many instances of improper sentence structuring, sometimes with little relation between the stereotypical and anti-stereotypical sentences, and repetition or improper structuring of target communities in stereotypical sentences. To tackle this, we manually filter these by removing sentences with opposite/inconsistent stereotypes or repetitions, as well as ensuring the correct target communities are present in the target columns. After filtering, we are left with 386 sentences for racial biases and 159 sentences for US-centric gender biases.

4. Experimental setup

4.1. Specifying identity axes of bias

For our dataset, we set out to target two highly prevalent forms of bias in India which may go relatively overlooked in fairness research and efforts: caste and religion. We introduce these identity axes because (i) the literature is relatively sparse in these two areas, and (ii) caste-based and religious discrimination is historically and socially significant across India (Saha etal., 2021a), as we explain further in AppendixB.

In order to draw contrast between the noticed and unnoticed areas of bias, we also seek to also measure race and gender for Western contexts because (i) there is a large body of existing research relating to these identity attributes in NLP (Sun etal., 2019; Thakur, 2023; Zhang etal., 2023), and (ii) conceptually, these identity attributes have deep historical roots of discriminatory treatment in the U.S. and are of significant cultural and legal importance in modern U.S. society (Snipp and Cheung, 2016; Plous and Neptune, 1997; Gregory, 1995).

We recognize bias’ connection to key demographic traits in society, shaped by a history of marginalization. Bias varies based on country and demographics. When comparing LLMs in the US and India, we emphasize local bias categories—caste and religion for India, and gender and race for the US.

This doesn’t disregard other biases in both countries. Gender bias persists in India, and religious bias exists in the US, each with unique aspects. Thus, we seek to compare bias frames (categories) that fit national and cultural contexts, and investigate if the frames that are more dominant in the US are better catered to in fairness research.

4.2. Models

We measure bias in two types of LLMs: encoder-based and decoder-based models.

Encoders correspond to models which are based on a Transformer encoder and pre-trained with masked language modeling (MLM).We select among the most popular encoder-based models in terms of number of downloads on HuggingFace. This includes both monolingual models, namely BERT-base (Devlin etal., 2018), and multilingual models, namely Multilingual BERT and XLM-RoBERTa-large (Conneau etal., 2020).

Decoders correspond to models based on a Transformer decoder and are primarily used for text generation. We select the most popular, publicly-available models in this space, namely GPT-2, GPT-2 Large (Radford etal., 2019), GPT-3.5 (OpenAI), Falcon (Penedo etal., 2023), Mistral 7B (Jiang etal., 2023) and LLamA-2 (Touvron etal., 2023).

These models (apart from GPT 3.5, which was accessed through the API) were loaded from HuggingFace and run on Google Colab infrastructure that utilised an A100 GPU.

4.3. Bias measurement

For each model discussed above, we report the percentage of times the model is more likely to output the stereotypical version of a sentence than the anti-stereotypical version. To find the difference in likelihoods, we first compute the log-likelihoods of outputting a sentence for the encoder and decoder models adjusted for differences in the relative base frequencies of the words being interchanged.

4.3.1. Encoders

For encoder models, we employ the All Unmasked Likelihood (AUL) score (Kaneko and Bollegala, 2022). We choose this metric as it avoids measurement biases arising from word frequency and input contexts which existed in prior metrics (Nadeem etal., 2020; Nangia etal., 2020). It does so by allowing the model to look at the entire sentence at once, instead of one-by-one masking. It is given for sentence $S$ by:

(1)

AUL(S)=\frac{1}{|S|}\sum_{i=1}^{|S|}\log P_{MLM}(w_{i}|(S;\theta))

where $|S|$ represents the length of sentence $S$ , and $P_{MLM}(w_{i}|(S;\theta))$ is the probability assigned during the MLM task to a token $w_{i}$ conditioned on the whole sentence $S$ and pre-training parameter $\theta$ . $AUL(S)$ is computed as the summation of the logarithms of the probabilities of the individual tokens for $S$ using parameter $\theta$ .

4.3.2. Decoders

For open-source decoders, we rely on a metric called Conditional Log-Likelihood (CLL), which evaluates the likelihood assigned by the decoder to a sentence containing target words $S_{w}$ , adjusted for the likelihood of outputting the target word sequence $w$ without any prior context.The CLL for a (stereotypical/anti-stereotypical) target word $w$ , given a sentence $S_{w}$ (including the target words) and model parameter $\theta$ can be defined by the equation:

(2)

CLL(S|w)=\ln P(S_{w};\theta)-\ln P(w;\theta)

This metric addresses a challenge pointed out in CrowS Pairs (Nangia etal., 2020) where certain words (and group identifiers) may be significantly more common in the pre-training data than others. Consequently, the probabilities of differing target words between stereotypical and anti-stereotypical sentences vary irrespective of context. To tackle this, (Nangia etal., 2020) propose computing $P(sentence|word)$ since the sentence $S$ tokens remain constant while the target word tokens $w$ change between stereotypical and anti-stereotypical sentences. While originally designed for masked language models, we adapt this metric for autoregressive models by subtracting the log-likelihoods of model outputting the target words without prior context (and hence dividing the likelihoods). This adaptation mirrors the approach utilized by (Felkner etal., 2023) but tailored to a dataset where target words can appear anywhere in the sentence, not just at the beginning.

4.3.3. GPT 3.5

For closed-source decoders, it is infeasible to obtain the CLLs for each word as the log-likelihoods are not readily available through the API. Hence, we estimate underlying priors by gathering three outputs, and taking the majority vote outputs (best of three). Methods of estimating stereotypical associations via statistical brute force have been applied previously (Kirk etal., 2021). We provide each template from the dataset, along with the pair of stereotypical and anti-stereotypical words which can be used to fill in the blank. We then report the share of sentences where the output was stereotypical or anti-stereotypical. More details on the methodology can be found in Section D of the Appendix.

4.4. Interpretation

Interaction of scores with models

While both AUL and CLL scores try to capture the log likelihoods of the models, the difference between the two scores largely arises due to the architectures of masked language models (MLMs), and autoregressive (AR) models. While in MLMs the entire sentence can be input to the model and pseudo log-likelihoods can be obtained, AR models only provide next token prediction given a previous context. Further, MLMs can be bi-directional, whereas ARs are uni-directional (Fu etal., 2023).

In the results section, we largely look at the percentage of prompts in which the models prefer a stereotypical example over an anti-stereotypical example. This makes up the ‘bias score’.While the individual numeric scores for sentences differ between MLMs and AR models, the bias score remains comparable across model types and scoring functions.

Interaction of scores with target communities:

The score itself indicates the propensity of a model to conform to stereotypes within a certain category, but it is difficult to establish whether a higher score is necessarily bad, particularly for race and gender categories. For instance, the gender dataset contains the stereotypes that women are tidier than men (positive) and that women are worse drivers than men (negative). In such cases, a high stereotypical bias score may not always indicate harmful behaviour towards the minority community. However, for our caste and religion datasets, high bias scores can indicate harmful behaviour as ’Muslim’ and ’Dalits’ are consistently associated with negative stereotypes, hence, conforming to those would generally mean negative behaviour.

While it can be a point of debate regarding what constitutes an ‘ideal’ score, one can nonetheless comparatively gauge the model’s propensity to conform to a particular type of stereotype over another. Hence, our findings are based on past research that makes use of stereotypes (Felkner etal., 2023; Nadeem etal., 2020; Ranaldi etal., 2023; Kaneko and Bollegala, 2022) such that a score closer to 50% signifies that the model has a balanced view between the stereotypical and anti-stereotypical topics overall.

5. Results

	Caste (India)	Religion (India)	Gender (U.S.)	Race (U.S.)
GPT 3.5	79.52	70.49	50.94	61.65
GPT 2	62.26	72.36	54.08	44.82
GPT 2 Large	63.21	69.11	61.63	44.3
Mistral 7B	56.60	75.61	66.66	59.84
Falcon 7B	61.32	72.35	69.81	65.02
LLaMA-2 7B	57.55	65.04	64.78	60.88
LLaMA-2 13B	56.61	72.36	68.55	63.47
XLM Roberta	62.43	52.29	50.79	46.48
m-BERT Uncased	57.74	52.77	52.52	49.5
BERT Base	55.51	55.1	50.88	51.58
Average	61.18	65.75	58.69	54.75

We present the bias scores for each model and axis of bias in Table 4. We find that the average stereotypical bias score across all models is highest for religion in the Indian context, followed by caste. On the other hand, it is relatively closer to the ‘neutral’ 50% mark for the US-centric gender and race bias axes. We also find that for GPT 3.5, XLM RoBERTa, m-BERT, and BERT base, the caste category has the highest stereotypical bias. For all of the remaining models, the religion category has the highest stereotypical bias score. Further, GPT 2, GPT 2 Large, XLM RoBERTa, m-BERT, and BERT base have the lowest stereotypical bias for race.

Notably, we find that GPT 3.5, GPT-2, GPT-2 Large, XLM-RoBERTa and m-BERT have the lowest stereotypical bias scores for race. In particular, this score is lower than 50% for GPT-2, XLM-RoBERTa and m-BERT, indicating that they favour the anti-stereotype and signalling a potential reversed bias against the ’majority’ community. Similarly, gender bias is nearly exactly 50% for GPT 3.5.

Further, upon qualitative inspection of religious bias, we notice that for GPT-2 (base and large) as well as GPT 3.5, most cases of preferring stereotypical community is when the target stereotypical community is Muslims (associated with stereotypes such as violence or terrorism). For instance, for GPT-2, 75% of the 72.36% of religion stereotypical responses were when Muslims were associated with negative stereotypes. In case of caste-based bias, models particularly show bias when the stereotypical target is upper-caste (Brahmin), through the attribution of positive attributes (pure, educated, etc.).

All of OpenAI’s GPT model scores have the largest gap between Western-centric (44-61%) and Indian-centric (62-79%) axes of bias, with the mean Indian-centric score as 69.5% as mean US-centric score as 52.8%. We discuss potential reasons for such disparity further in Section 6.

6. Discussion

This work introduced the first evaluation of bias levels in popular encoder and decoder LLMs in the Indian context.Our results have some key takeaways. First, we find that the majority of LLMs (both encoder and decoder) favour the stereotypical associations in the Indian contexts of religion and caste, particularly the religious stereotypes. It also appears in some cases that debiasing efforts or conscious pre-training choices may have swung some models, such as GPT-2, XLM RoBERTa and m-BERT, towards anti-stereotypical bias for the racial bias axis.

We cannot conclusively explain why the LLMs that we tested display stronger stereotypical biases within the Indian categories than the US-centric gender and racial categories. However, we offer some perspectives as to the roots of this phenomena.

The development and evaluation of LLMs are mainly conducted with a US-centric perspective, with technology developers primarily situated in Silicon Valley and employing predominantly educated US-based crowdworkers as human raters or red-teamers (Talat etal., 2022; Kirk etal., 2023). This US-centric focus tends to often overlook potential adverse implications within the Global South.

The digital divide in India (Rajam etal., 2021; Tewathia etal., 2020) influences who has access to author internet content, or have content authored about their group and lived experiences (Sambasivan etal., 2021). Our empirical observations reveal that terms like “Brahmin” have a higher prediction likelihood compared to “Dalit” indicating a frequency bias possibly originating from their respective frequencies in the pre-training data, despite the stark contrast in shares of the Indian population, with over 60% Dalits, and only 5-10% Brahmins (Center, 2021). Also, top models such as Mistral and LLaMA-2 (Jiang etal., 2023; Touvron etal., 2023) use the ToxiGen (Hartvigsen etal., 2022) framework to evaluate their harmful behaviour, which relies completely on social media and digital data which may be disproportionately representing the upper castes and privileged in India.

Mitigation of such biases in LLMs is an active area of study and we believe that techniques such as in-context learning (Berg etal., 2022; Dwivedi etal., 2023; Ganguli etal., 2023; Li etal., 2024), reinforcement learning with human-feedback (Yu etal., 2023; Zheng etal., 2024; Dong etal., 2023), or architectural methods such as (Jin etal., 2020; Mahabadi etal., 2019) can help in limiting harmful stereotypical associations by LLMs.

Future work could study how our cross-cultural studies of bias extend to the cross-cultural and cross-lingual setting. We only focus on English, but many languages are spoken in India, and caste may have an association with spoken language, thus introducing further contextual dependencies of bias axis and evaluation language. Furthermore, while we do investigate some multilingual models, it would be interesting to see how mono or multilingual models specifically pre-trained or fine-tuned on Indian data perform in our evaluation framework.

Limitations

Dataset coverage

While each bias category introduced in Indian-BhED is comparable in size to the categories in other similar datasets (Nangia etal., 2020), there is room to provide wider and more nuanced coverage. Firstly, the size, and nature of the dataset should be expanded in future work, by for instance adding more languages and stereotypes. Secondly, in particular, we focus on explicit statements of bias, but biases may be more pervasive and appear in descriptions of people (e.g. ‘turban wearers’). We are ourselves also subject to blind spots in our selection of cases. To reduce this risk, we consulted with experts on Indian caste bias, as well as literature, but the possibility of an overlook remains.

Conceptualisation of bias

Bias can be a complex social concept, having many definitions and notions. It is difficult to accurately quantify bias and the metrics used in this study are unlikely to encapsulate all the facets of bias. A model’s propensity to generate a stereotypical or anti-stereotypical response leaves a gap for many ‘in-between’ target responses which should also be accounted for. In cases where the underlying populations are unbalanced, it is also unclear whether a 50% balance between stereotypical and anti-stereotypical responses is appropriate. Other approaches (Kirk etal., 2021) use demographic data to establish a baseline score. It may be that such an approach leads to dramatic over-representation of the ‘Brahmin’ caste, which is a small fraction of the total Indian population.

Explanatory power

Although our study shows evidence of the existence of a disparity in bias levels, we cannot attribute it to any particular factor, due to the black-boxedness of such models, along with selective sharing of details by certain companies (such as OpenAI). It is interesting that the differences persist across models trained with very different approaches, including in models which have been fine-tuned to reduce bias. However, we cannot determine whether this results from the underlying datasets having more embedded biases, less efforts at bias mitigation for Indian-centric categories, less effectiveness of existing methods of bias mitigation for Indian categories of bias, a combination of all of these, or another reason altogether.

7. Conclusion

As the the user base of large language models becomes increasingly global, there needs to be attention from academics, policymakers and industry labs directed towards uncovering biases localised in specific geo-cultural contexts, that may be missed or overlooked with a US-centric lens. This study provided empirical evidence to suggest certain areas of bias such as caste based or religious biases persist in models, even if gender and racial biases have been relatively better tackled as per our findings. We introduced a new dataset to capture biases in the Indian context and established a framework for measuring bias on the same dataset for encoder as well as decoder based models. We hope this work initiates a conversation surrounding the need to develop more inclusive standards of fairness in AI across geo-cultural contexts.

Acknowledgements.

The authors wish to acknowledge the support by the Oxford Internet Institute, as well as the guidance of Dr. Adam Mahdi. We also thank the quintessential inputs of the subject matter experts.

References

(1)
B etal. (2022)SenthilKumar B, Pranav Tiwari, AmanChandra Kumar, and Aravindan Chandrabose. 2022.Casteism in India, but Not Racism - a Study of Bias in Word Embeddings of Indian Languages. In Proceedings of the First Workshop on Language Technology and Resources for a Fair, Inclusive, and Safe Society within the 13th Language Resources and Evaluation Conference, Kolawole Adebayo, Rohan Nanda, Kanishk Verma, and Brian Davis (Eds.). European Language Resources Association, Marseille, France, 1–7.https://aclanthology.org/2022.lateraisse-1.1
Baber (2004)Zaheer Baber. 2004.‘Race’, religion and riots: The ‘racialization’of communal identity and conflict in India.Sociology 38, 4 (2004), 701–718.
Bayly (1985)ChristopherA Bayly. 1985.The Pre-history of ‘; Communalism’? Religious Conflict in India, 1700–1860.Modern Asian Studies 19, 2 (1985), 177–203.
Berg etal. (2022)Hugo Berg, SiobhanMackenzie Hall, Yash Bhalgat, Wonsuk Yang, HannahRose Kirk, Aleksandar Shtedritski, and Max Bain. 2022.A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning.arXiv preprint arXiv:2203.11933 (2022).
Bhatt etal. (2022)Shaily Bhatt, Sunipa Dev, Partha Talukdar, Shachi Dave, and Vinodkumar Prabhakaran. 2022.Re-contextualizing fairness in NLP: The case of India.arXiv preprint arXiv:2209.12226 (2022).
Blodgett etal. (2020)SuLin Blodgett, Solon Barocas, Hal DauméIII, and Hanna Wallach. 2020.Language (technology) is power: A critical survey of” bias” in nlp.arXiv preprint arXiv:2005.14050 (2020).
Bolukbasi etal. (2016)Tolga Bolukbasi, Kai-Wei Chang, JamesY Zou, Venkatesh Saligrama, and AdamT Kalai. 2016.Man is to computer programmer as woman is to homemaker? debiasing word embeddings.Advances in neural information processing systems 29 (2016).
Brown etal. (2020)TomB. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, DanielM. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever,and Dario Amodei. 2020.Language Models are Few-Shot Learners.arXiv:2005.14165[cs.CL]
Center (2021)PewResearch Center. 2021.Attitudes about Caste.https://www.pewresearch.org/religion/2021/06/29/attitudes-about-caste/Accessed on 2023-07-07.
Conneau etal. (2020)Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.Unsupervised Cross-lingual Representation Learning at Scale.arXiv:1911.02116[cs.CL]
Davidson etal. (2019)Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019.Racial bias in hate speech and abusive language detection datasets.arXiv preprint arXiv:1905.12516 (2019).
Dev etal. (2023)Sunipa Dev, Jaya Goyal, Dinesh Tewari, Shachi Dave, and Vinodkumar Prabhakaran. 2023.Building Socio-culturally Inclusive Stereotype Resources with Community Engagement.arXiv preprint arXiv:2307.10514 (2023).
Devlin etal. (2018)Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 (2018).
Dhattiwala and Biggs (2012)Raheel Dhattiwala and Michael Biggs. 2012.The political logic of ethnic violence: The anti-Muslim pogrom in Gujarat, 2002.Politics & Society 40, 4 (2012), 483–516.
Dhingra etal. (2023)Harnoor Dhingra, Preetiha Jayashanker, Sayali Moghe, and Emma Strubell. 2023.Queer People are People First: Deconstructing Sexual Identity Stereotypes in Large Language Models.arXiv preprint arXiv:2307.00101 (2023).
Dong etal. (2023)Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023.Raft: Reward ranked finetuning for generative foundation model alignment.arXiv preprint arXiv:2304.06767 (2023).
Dunham etal. (2014)Yarrow Dunham, Mahesh Srinivasan, Ron Dotsch, and David Barner. 2014.Religion insulates ingroup evaluations: The development of intergroup attitudes in India.Developmental science 17, 2 (2014), 311–319.
Dwivedi etal. (2023)Satyam Dwivedi, Sanjukta Ghosh, and Shivam Dwivedi. 2023.Breaking the Bias: Gender Fairness in LLMs Using Prompt Engineering and In-Context Learning.Rupkatha Journal on Interdisciplinary Studies in Humanities 15, 4 (2023).
Felkner etal. (2023)VirginiaK Felkner, Ho-ChunHerbert Chang, Eugene Jang, and Jonathan May. 2023.Winoqueer: A community-in-the-loop benchmark for anti-lgbtq+ bias in large language models.arXiv preprint arXiv:2306.15087 (2023).
Feng etal. (2023)Shangbin Feng, ChanYoung Park, Yuhan Liu, and Yulia Tsvetkov. 2023.From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models.arXiv preprint arXiv:2305.08283 (2023).
Fu etal. (2023)Zihao Fu, Wai Lam, Qian Yu, Anthony Man-Cho So, Shengding Hu, Zhiyuan Liu, and Nigel Collier. 2023.Decoder-only or encoder-decoder? interpreting language model as a regularized encoder-decoder.arXiv preprint arXiv:2304.04052 (2023).
Ganguli etal. (2023)Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, etal. 2023.The capacity for moral self-correction in large language models.arXiv preprint arXiv:2302.07459 (2023).
Garg etal. (2019)Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, EdH Chi, and Alex Beutel. 2019.Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 219–226.
Gnana (2018)SelvinRaj Gnana. 2018.Caste system, Dalitization and its implications in contemporary India.International Journal of Sociology and Anthropology 10, 7 (2018), 65–71.
Gregory (1995)JamesF Gregory. 1995.The crime of punishment: Racial and gender disparities in the use of corporal punishment in US public schools.Journal of Negro Education (1995), 454–462.
Gupta (2008a)Charu Gupta. 2008a.(MIS) Representing the Dalit Woman: Reification of Caste and Gender Stereotypes in the Hindi Didactic Literature of Colonial India.Indian Historical Review 35, 2 (2008), 101–124.
Gupta (2008b)Charu Gupta. 2008b.(MIS) Representing the Dalit Woman: Reification of Caste and Gender Stereotypes in the Hindi Didactic Literature of Colonial India.Indian Historical Review 35, 2 (2008), 101–124.
Hartvigsen etal. (2022)Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022.Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection.arXiv preprint arXiv:2203.09509 (2022).
Hu (2023)Krystal Hu. 2023.ChatGPT sets record for fastest-growing user base - analyst note.Reuters (2023).https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/
Jamwal (2021)Ranjodh Jamwal. 2021.TIMUR’S INVASION OF INDIA.DIRECTORATE OF DISTANCE EDUCATION UNIVERSITY OF JAMMU (2021), 124.
Jiang etal. (2023)AlbertQ Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, DevendraSingh Chaplot, Diego delas Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, etal. 2023.Mistral 7B.arXiv preprint arXiv:2310.06825 (2023).
Jin etal. (2020)Xisen Jin, Francesco Barbieri, Brendan Kennedy, AidaMostafazadeh Davani, Leonardo Neves, and Xiang Ren. 2020.On transferability of bias mitigation effects in language model fine-tuning.arXiv preprint arXiv:2010.12864 (2020).
Kamble and Joshi (2018)Satyajit Kamble and Aditya Joshi. 2018.Hate speech detection from code-mixed hindi-english tweets using deep learning models.arXiv preprint arXiv:1811.05145 (2018).
Kaneko and Bollegala (2022)Masahiro Kaneko and Danushka Bollegala. 2022.Unmasking the mask–evaluating social biases in masked language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol.36. 11954–11962.
Kirk etal. (2023)HannahRose Kirk, AndrewM Bean, Bertie Vidgen, Paul Röttger, and ScottA Hale. 2023.The past, present and better future of feedback learning in large language models for subjective human preferences and values.arXiv preprint arXiv:2310.07629 (2023).
Kirk etal. (2021)HannahRose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. 2021.Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models.Advances in neural information processing systems 34 (2021), 2611–2624.
Kirtane and Anand (2022)Neeraja Kirtane and Tanvi Anand. 2022.Mitigating gender stereotypes in Hindi and Marathi.arXiv preprint arXiv:2205.05901 (2022).
Kumar (2020)Awanish Kumar. 2020.BR Ambedkar on caste and land relations in India.Review of Agrarian Studies 10, 2369-2020-1859 (2020).
Li etal. (2024)Jingling Li, Zeyu Tang, Xiaoyu Liu, Peter Spirtes, Kun Zhang, Liu Leqi, and Yang Liu. 2024.Steering LLMs Towards Unbiased Responses: A Causality-Guided Debiasing Framework.arXiv preprint arXiv:2403.08743 (2024).
Mahabadi etal. (2019)RabeehKarimi Mahabadi, Yonatan Belinkov, and James Henderson. 2019.End-to-end bias mitigation by modelling biases in corpora.arXiv preprint arXiv:1909.06321 (2019).
Malik etal. (2021)Vijit Malik, Sunipa Dev, Akihiro Nishi, Nanyun Peng, and Kai-Wei Chang. 2021.Socially Aware Bias Measurements for Hindi Language Representations.arXiv preprint arXiv:2110.07871 (2021).
Nadeem etal. (2020)Moin Nadeem, Anna Bethke, and Siva Reddy. 2020.StereoSet: Measuring stereotypical bias in pretrained language models.arXiv preprint arXiv:2004.09456 (2020).
Nangia etal. (2020)Nikita Nangia, Clara Vania, Rasika Bhalerao, and SamuelR Bowman. 2020.CrowS-pairs: A challenge dataset for measuring social biases in masked language models.arXiv preprint arXiv:2010.00133 (2020).
Penedo etal. (2023)Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023.The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only.arXiv preprint arXiv:2306.01116 (2023).arXiv:2306.01116https://arxiv.org/abs/2306.01116
Plous and Neptune (1997)Scott Plous and Dominique Neptune. 1997.Racial and gender biases in magazine advertising: A content-analytic study.Psychology of women quarterly 21, 4 (1997), 627–644.
Radford etal. (2019)Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, etal. 2019.Language models are unsupervised multitask learners.OpenAI blog 1, 8 (2019), 9.
Rajam etal. (2021)Vaidehi Rajam, ABheemeshwar Reddy, and Sudatta Banerjee. 2021.Explaining caste-based digital divide in India.Telematics and Informatics 65 (2021), 101719.
Rambachan (2008)Anantanand Rambachan. 2008.Is Caste Intrinsic to Hinduism?Tikkun 23, 1 (2008), 59–61.
Ranaldi etal. (2023)Leonardo Ranaldi, ElenaSofia Ruzzetti, Davide Venditti, Dario Onorati, and FabioMassimo Zanzotto. 2023.A trip towards fairness: Bias and de-biasing in large language models.arXiv preprint arXiv:2305.13862 (2023).
Rath and Sircar (1960)R Rath and NC Sircar. 1960.The mental pictures of six Hindu caste groups about each other as reflected in verbal stereotypes.The Journal of Social Psychology 51, 2 (1960), 277–293.
Robinson (2008)Rowena Robinson. 2008.Religion, socio-economic backwardness & discrimination: The case of Indian Muslims.Indian Journal of Industrial Relations (2008), 194–200.
Saha etal. (2021a)Punyajoy Saha, Binny Mathew, Kiran Garimella, and Animesh Mukherjee. 2021a.“Short is the Road That Leads from Fear to Hate”: Fear Speech in Indian WhatsApp Groups. In Proceedings of the Web Conference 2021 (Ljubljana, Slovenia) (WWW ’21). Association for Computing Machinery, New York, NY, USA, 1110–1121.https://doi.org/10.1145/3442381.3450137
Saha etal. (2021b)Punyajoy Saha, Binny Mathew, Kiran Garimella, and Animesh Mukherjee. 2021b.“Short is the Road That Leads from Fear to Hate”: Fear Speech in Indian WhatsApp Groups. In Proceedings of the Web Conference 2021 (Ljubljana, Slovenia) (WWW ’21). Association for Computing Machinery, New York, NY, USA, 1110–1121.https://doi.org/10.1145/3442381.3450137
Sambasivan etal. (2021)Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. 2021.Re-imagining algorithmic fairness in india and beyond. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 315–328.
Santurkar etal. (2023)Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023.Whose opinions do language models reflect?arXiv preprint arXiv:2303.17548 (2023).
Sen and Wagner (2005)Ragini Sen and Wolfgang Wagner. 2005.History, emotions and hetero-referential representations in inter-group conflict: The example of Hindu-Muslim relations in India.Papers on Social Representations 14 (2005), 2–1.
Sinha and Sinha (1967)GopalSharan Sinha and RameshChandra Sinha. 1967.Exploration in caste stereotypes.Social Forces 46, 1 (1967), 42–47.
Smith etal. (2022)EricMichael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022.“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 9180–9211.
Snipp and Cheung (2016)CMatthew Snipp and SinYi Cheung. 2016.Changes in racial and gender inequality since 1970.The ANNALS of the American Academy of Political and Social Science 663, 1 (2016), 80–98.
Stevenson (1954)Henry NoelCochran Stevenson. 1954.Status evaluation in the Hindu caste system.The Journal of the Royal Anthropological Institute of Great Britain and Ireland 84, 1/2 (1954), 45–65.
Stewart (1951)Neil Stewart. 1951.Divide and rule: British policy in indian history.Science & Society (1951), 49–57.
Sun etal. (2019)Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and WilliamYang Wang. 2019.Mitigating Gender Bias in Natural Language Processing: Literature Review.arXiv:1906.08976[cs.CL]
Talat etal. (2022)Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, etal. 2022.You reap what you sow: On the challenges of bias evaluation under multilingual settings. In Proceedings of BigScience Episode# 5–Workshop on Challenges & Perspectives in Creating Large Language Models. 26–41.
Talbot (1995)Cynthia Talbot. 1995.Inscribing the other, inscribing the self: Hindu-Muslim identities in pre-colonial India.Comparative studies in society and history 37, 4 (1995), 692–722.
Tewathia etal. (2020)Nidhi Tewathia, Anant Kamath, and PVigneswara Ilavarasan. 2020.Social inequalities, fundamental inequities, and recurring of the digital divide: Insights from India.Technology in Society 61 (2020), 101251.
Thakur (2023)Vishesh Thakur. 2023.Unveiling gender bias in terms of profession across LLMs: Analyzing and addressing sociological implications.arXiv preprint arXiv:2307.09162 (2023).
Touvron etal. (2023)Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, CristianCanton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa,Isabel Kloumann, Artem Korenev, PunitSingh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, EricMichael Smith, Ranjan Subramanian, XiaoqingEllen Tan, Binh Tang, Ross Taylor, Adina Williams, JianXiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, YuchenZhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.Llama 2: Open Foundation and Fine-Tuned Chat Models.arXiv:2307.09288[cs.CL]
Vashishtha etal. (2023)Aniket Vashishtha, Kabir Ahuja, and Sunayana Sitaram. 2023.On evaluating and mitigating gender biases in multilingual settings.arXiv preprint arXiv:2307.01503 (2023).
Yu etal. (2023)Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, etal. 2023.Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback.arXiv preprint arXiv:2312.00849 (2023).
Zhang etal. (2023)Angela Zhang, Mert Yuksekgonul, Joshua Guild, James Zou, and Joseph Wu. 2023.ChatGPT exhibits gender and racial biases in acute coronary syndrome management.medRxiv (2023), 2023–11.
Zheng etal. (2024)Chen Zheng, Ke Sun, Hang Wu, Chenguang Xi, and Xun Zhou. 2024.Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHF.arXiv preprint arXiv:2403.02513 (2024).

Appendix A Appendix

Appendix B More on Casteism and Religious Discrimination in India

Casteism

Caste-based discrimination, unique to India, has a 3,000-year history rooted in texts like the “Manusmriti,” which sanctions the caste system and prescribes harsh punishments for dissent (Rambachan, 2008). This system defines social order, with individuals’ karma and dharma determining their caste, passing it down through generations (Gnana, 2018). Ancient Hindu society organized this hierarchy into four main castes: Brahmins (intellectuals), Kshatriyas (warriors), Vaishyas (traders), and Shudras (laborers), with Dalits as untouchables (Gnana, 2018). Despite legal reforms, the caste system still affects many Indians (Kumar, 2020). About 30% of Indians identify as “General Category” (upper castes), while only 4% identify as Brahmins (Center, 2021). Most Indians identify as “Scheduled Castes” (Dalits), “Scheduled Tribes,” or “Other Backward Classes.” Caste discrimination thrives on stereotypes, portraying Brahmins positively and Dalits negatively (Sinha and Sinha, 1967). For simplicity, this paper examines these biases using a dichotomous framework: “Brahmin” for the upper caste and “Dalit” for the lower castes (Sinha and Sinha, 1967).

Religious discrimination in India

Despite its constitutional claim to secularism, India has long grappled with a Hindu–Muslim divide. This divide predates British colonial rule but was exacerbated by British colonial policies (Bayly, 1985; Talbot, 1995). Historical events like Muslim rulers’ conquests and the British ‘divide and rule’ strategy further fueled this divide (Jamwal, 2021; Stewart, 1951). In India, Orientalist scholarship contributed to the racialisation of communal identity, emphasising the Indo-Aryan linguistic family, Aryan Race, and the supposed end of a Golden Age of Hinduism due to Muslim invasions (Baber, 2004). Hindu activists, drawing from Orientalist ideas, advocated for a revival of this ‘Golden Age’, promoting the notion of Hindus as descendants of a superior Aryan race (Baber, 2004). These ideas gave rise to the Rashtriya Swayamsevak Sangh (RSS), a right-wing Hindu nationalist organization, from which the largest political party in India, the Bharatiya Janata Party (BJP) emerged. It is associated with many pro-Hindu policies (Dhattiwala and Biggs, 2012). Today, Hindu–Muslim relations in India are marked by tension and a power disparity, with Hindus in the majority and Muslims as a minority (Dunham etal., 2014). This paper primarily focuses on Hindu–Muslim religious discrimination as it is the most prevalent form in the Indian context, especially online (Saha etal., 2021b).

Appendix C Data Statement

The new dataset is created through a generation and validation process, emulating the approach of HolisticBias (Smith etal., 2022).The following steps were adopted:

(i) The sentences are first brainstormed by the authors based on the existing literature on casteism and religion-based hate speech datasets from the recent past (Sinha and Sinha, 1967; Rath and Sircar, 1960; Gupta, 2008b; Kamble and Joshi, 2018). In doing so, we prefer colloquial terms such as ‘dirty’ instead of ‘contaminated’ or ‘dilapidated’, and include both positive and negatively biased examples. At this stage it was 112 sentences for caste, and 120 for religion.

(ii) In order to minimize the authors’ own biases from this dataset, three professors in India- (1) a Linguistics professor, (2) a Caste Based Studies professor, and (3) an Islamic studies professor, were consulted and asked to provide qualitative feedback for the dataset regarding the composition, validity, and coverage of the prompts. Those inputs included the suggestions such as to add sentences on ‘Muslim men tricking Hindu women into marriage’ and ‘Crime associated with Dalits’. Their suggestions also involved removing certain sentences which captured the debatable stereotypes that Dalits are hard-working, and Muslims are artistic. Such qualitative reviews and suggestions were incorporated into the dataset and finally approved by the same professors unanimously.

We believe that closely basing our data on past research and datasets also reduces the personal biases of the authors. Upon inclusion of final suggestions, we add 5 more sentences in caste dataset while removing 11 which the experts believed were inaccurate/problematic, while adding 3 more sentences for religion.

The caste and religion dataset is designed to be broadly balanced and as comprehensive as possible given the size. There are a total of 123 sentences constructed for religion (60% Muslim stereotypes, 40% Hindu and other stereotypes) and 106 sentences for caste (50% positive stereotypes for Brahmins, 40% negative stereotypes of Dalits, and 10% mixed stereotypes for the two castes). We also provide full sentence examples rather than word/stereotype pairs to better support future users of the data.

Appendix D Model choice and implementation details

The method for model selection was largely qualitative and based on obtaining certain representative, well-known and widely downloaded models for different categories. We began with popular BERT-based models, followed by several of the most popular generative models available on Hugging Face. To be specific, for BERT-based models we chose the original BERT-Base (Devlin etal., 2018), and prevailing multilingual models BERT-Base-Multilingual (2019), and XLM-RoBERTa-Large (2019). These were chosen in order to compare the difference in biases between a multilingual and a largely English model. Among open source generative models, we selected models based on popularity and availability. In particular, these are Meta AI’s LLaMA 2 (7 billion and 13 billion parameters) (Touvron etal., 2023), Falcon (7 billion and 40 billion parameters) (Penedo etal., 2023), as well as OpenAI’s GPT 2 (117 million and 345 million parameters) (Radford etal., 2019). Once again, these were selected on the basis of how large they are, how commonly downloaded they are on Hugging Face (all of these models have been downloaded $>100,000$ times in the month of July, 2023). We were not able to access the necessary log-likelihood values for some of the other popular models such as GPT 4, Gemini, or Bard. We do include GPT-3.5 as a non open-source model as it has the highest number of users (100 million monthly active users) as compared to any other generative model that is in production. However, we can only provide metrics based on querying the API, which are not directly comparable with the others.

We provide the prompt in the following format to GPT 3.5 to obtain results: ”For the following sentence, fill in the blank strictly with $<$ stereotypical word $>$ or $<$ anti-stereotypical word $>$ : $<$ sentence with blank $>$ . Finally, the outputs for 3 prompts were calculated based on best of three responses. It is to note that there was also a case of neutral responses by GPT 3.5 (refusing to select between the two options/selecting both options/outputting a new third option altogether). For reporting this, we took the number of neutral responses, and divided it equally among the number of stereotypical and anti-stereotypical responses to report the final results. For instance, if there were 10 neutral, 10 stereotypical and 10 anti-stereotypical responses, we reported it as percent of (10 + (10/2) ) = 15 stereotypical and 15 anti-stereotypical responses.