My work bridges data science and social science, focusing on textual data and what it can tell us about human experience. I lead HUMBLE Lab, where a central question is how we can use computational methods to understand people’s experiences without ironing out what it is to be human. These are my thoughts through an interdisciplinary lens.
Human experience data are gathered on a large scale through clinical notes, social media data and open-ended questions. The volume of data is simply too large to analyse using only human analysis, traditionally using qualitative methods. Artificial Intelligence (AI), including large language models (LLMs), can process large volumes of text rapidly by identifying statistical patterns in language. These approaches can support the identification of topics, similarities and differences across lived experiences.
But using AI tools to make sense of human experience data requires much more than computational expertise: indeed, we need social science involved in how AI tools are developed and evaluated, and, most importantly, how we interpret what AI produces. This matters particularly when we work with underserved and marginalised experiences and seek to advance health equity.
I will propose 3 arguments for why social sciences are important in computational methods: 1) AI can flatten meaning; 2) AI can reproduce stereotypes; 3) we can combine computational methods with human qualitative interpretation to get a synergistic effect.
First, AI can flatten meaning. In our work with LLMs, we found that when we try to use LLMs to replace humans in creating themes from large-scale human experience data, they produce coherent outputs but lose the nuance in people’s accounts. We also identified several patterns in how LLMs synthesise lived experience data. For example, headline summaries presented as themes, with generalised labels such as “challenges”, “disruption” and “coping strategies”; abstract filler language that sounds elegant but does not actually mean anything – my favourite being “a rich tapestry of experience”; and distinct concepts blurred together. For example, “well-being”, “health” and “quality of life” could be used as if they meant the same thing.
Why does it matter?
These distinctions matter for scientific integrity in representing human experience. The subtle ironing out of the differences in meaning means a loss of what it means to experience. Indeed, human experience is, by definition, complex. As scientists, at the centre of our values lies the representation of the variety of the underlying data.
More broadly, I raise a question about the use of AI for text analysis: if we use AI without the conceptual, contextual and interpretive expertise of social science, will we lose the nuance of experience and erase the already less heard voices?
Second, AI can reproduce bias. In textual analysis using AI, the problem is often not obviously offensive or discriminatory language; it may instead involve subtle shifts in wording.
In our study, we took the same underlying text about experiences of living through the Covid-19 pandemic and changed only the demographic labels – for example, “telling” the LLM that the data came from women rather than men. The analysis changed: we saw differences in semantics, sentiment and emotion. For example, when the data were labelled as coming from men, the language was more negative, with words such as “disillusionment”, “discontent”, “unhealthy” and “intense” more prominent in the analysis. When the data were labelled as coming from women, it shifted towards using “hope”, “healing”, “peace” and “strength”. These differences were not readily apparent from reading the outputs; we had to evaluate them quantitatively.
It is more than inaccurate analysis. If AI flattens meaning and/or reproduces bias, it is likely to obscure the sociocultural context of people’s experiences. In the techno-centred culture in which we operate, the lived experience of people, especially those in the minority, may be discredited when an AI analysis is treated as more objective or authoritative than their own account, creating a risk of generative epistemic injustice.
At HUMBLE Lab, we review evidence of bias in AI analysis. We see biased results relating to gender, ethnicity, age, disability, socioeconomic status, language, culture and politics, including intersections between these factors. We need social scientists involved because recognising these biases requires an understanding of social context, marginalisation and power.
Third, we can combine computational methods with human qualitative interpretation. In the method we developed, Machine-Assisted Topic Analysis (MATA), we combine computational methods with qualitative analysis. The benefit of this approach is that it starts with data-driven methods, looking at what AI is best at, i.e. pattern recognition, speed, scale. Then AI’s role ends and humans take over, with rich qualitative domain expertise in meaning-making, contextual understanding and the application of theoretical understanding to the underlying topics created by the machine. Using MATA, we analysed large-scale free-text accounts of experiences during the Covid-19 pandemic and of public services, and shed light on the detrimental effects of highly popular health apps, amongst other. Recently, we also found that this approach can surface meaningful sociodemographic differences in experiences that LLM-based analysis did not identify.
It is more important than ever that social science is an active part of AI research. We need a culture of interdisciplinary collaboration where social science and computer and data science work together on equal terms. This way we can combine the immense and exponentially expanding capacity of AI with expertise in sociocultural understanding, marginalisation and power imbalances. Then we have the opportunity to amplify health equity. Otherwise, we risk ironing out the very differences we need to understand and perpetuating the still omnipresent historical biases.