top of page
Search

The Statistical Ghost: Can AI Ever Truly Be Inclusive in Mental Health?

  • 23 hours ago
  • 7 min read

Written by Kevin Flanagan



At a recent virtual coffee morning we discussed how as artificial intelligence is increasingly integrated into healthcare systems, there’s a tendency to view its deployment as a technical challenge. However, deploying algorithms into mental healthcare is not a neutral technological upgrade. It is the active encoding of a very specific worldview. My hypothesis is we may be facing a Black Swan type event in mental health AI, driven by deeply flawed baseline data, algorithmic degradation, and a dangerous illusion of competence. To truly humanise mental health AI, we must fundamentally rethink what we consider "ground truth."


Key Assumptions and Perspective: I am taking a systems perspective and I am also making several key assumptions:


  • There are no sources of high quality consistent data for mental health at scale available to train the models

  • English is the dominant language for training AI models

  • Training sets for foundation models have at least 10% of bad data

  • Foundation models have been trained on at least 10% of LLM generated output

  • The ratio of problematic training data to good is approx. 1:6

  • Statistical independence may not be guaranteed


Linguistic and Cultural Hegemony The epistemological foundation of our current AI models is heavily skewed. Approximately 90% of all AI training data is in English. This is not merely a problem of translation; language structurally encodes a worldview.


When an AI is trained primarily on English, it inherits Western, Educated, Industrialised, Rich, and Democratic (WEIRD) assumptions about human psychology. For instance, English grammar inherently emphasises individual agency, a linear progression of time and an egocentric spatial reference whereas other languages might centre communal relationships or cyclical time. Concepts like "privacy" mean something entirely different in Silicon Valley compared to a collectivist culture. Consequently, AI systems trained on this linguistic logic risk naturally pathologising non-Western behaviours simply because they do not fit the cultural architecture of the training data.


The Clinician Filter vs. Patient Reality This demographic skew is compounded by an epistemological gap in mental health data itself, clinical records are not the objective ground truth of a patient's phenomenological experience.


Instead, clinical records are filtered interpretations shaped by clinician cognition, institutional pressures, and individual psychology. Furthermore, clinicians document presentations using diagnostic grammars like the DSM and ICD, which are themselves English-language constructs that assume/inherit individualist agency and linear causality. We are training models on a highly legible, reductionist biomedical model of illness, completely missing the fact that human distress is a complex, biopsychosocial problem.


Model Collapse: Erasing the Margins These initial data flaws are drastically amplified by a phenomenon known as "model collapse." A 2024 peer-reviewed paper in Nature demonstrated that when generative AI models are trained recursively on AI-generated data, they undergo irreversible degradation.


As models collapse, the first thing they lose are the "tails" of the distribution, the rare, low-probability events. In marketing algorithms, a statistical tail is just noise. But in mental health, the tail is where the crisis lives. These margins represent complex comorbidities, atypical presentations suicidal ideation but also culturally specific expressions of distress.


Model collapse actively erases this vital variance, converging on a narrow, low-variance mean. A system that optimises for the "average" patient is therefore structurally discriminating against the most vulnerable patients with the highest clinical risk. Crucially, as some critiques note, collapse erases what little variance remains in a system that already excluded non-Western experiences from the start.


The Illusion of Competence Why is this erasure so difficult to detect? Because these models perform with a dangerous "illusion of competence".


AI systems trigger the "fluency heuristic", because they sound highly articulate, humans naturally assume they are credible. Humans are cognitively lazy, we readily accept plausible sounding AI outputs because verifying or challenging them requires non-trivial effort. At the same time, commercial pressures actively reward this confident plausibility and penalise the calibrated uncertainty that responsible clinicians rely upon.


This creates a dangerous feedback loop where models confidently fail silently on complex cases, creating the architecture for a slow-moving systemic crisis measured in human suffering at scale.


Black Swan Scenario: Mental health is already experiencing what might be characterised as a slow-moving Black Swan, a global prevalence crisis.


It is our biggest health problem by impact whose causes remain poorly understood and whose dominant interventions show only modest efficacy at the population level. The application of a poorly fitted model, amplified and validated at scale by LLMs that systematically favour its framing, risks cementing that model further into clinical practice, policy, and public understanding precisely at the moment when genuine intellectual openness to alternative frameworks is most needed.


Serious Implications: The biopsychosocial model requires holding complexity, uncertainty, and individual variation simultaneously. It is cognitively demanding, commercially inconvenient, and difficult to operationalise at scale. It sits at the opposite end of the spectrum from what the laziness-plausibility-collapse dynamic selects for. The systems argument therefore predicts not merely stagnation in mental health understanding, but active regression. That’s a narrowing of the epistemic space available for thinking about the problem, occurring quietly and without any identifiable moment of failure, which is precisely what makes it a Black Swan condition.


It is a domain where the consequences of that narrowing are measured in human suffering at scale.


The Path Forward: The N=1 Blueprint and Multimodality To step off this catastrophic path, we must shift our definition of ground truth away from a flawed, collapsing global average. The future of mental health AI lies in the "N=1 Blueprint" to finally realise the original technologists aim of “pertinence”. This means establishing individual baselines and comparing a patient to themselves over time, as the true clinical signal is found in a patient's deviation from their own self, not a synthetic norm.


Executing this blueprint requires moving beyond text. Because text is so heavily subject to linguistic hegemony and model collapse, we must embrace multimodal signals. Relying on the electronic patient record is flawed for the reasons outlined before but also because historically up to 30% of records may be incomplete. By integrating voice prosody, tone, and physiological stress markers, we can capture the "unwritten" context of a patient's reality. This allows us to bypass the epistemic filters of clinical text, offering a more direct, localised, and human window into psychological distress.


Discussion: We had a very lively discussion centring around ways in which we might be able to use Large Language Models (LLMs) in better ways. The consensus was that LLMs are a useful tool and should be seen as just this and not some silver bullet and we had some examples from participants of this. There was general acceptance that LLMs are definitely better than nothing in moments of distress. This raised the system question of what happens next and that perhaps this just moves the bottleneck along without solving the underlying problem if the rest of the system is not ready.


A last important point was made about linguistic hegemony that because a lot of people with mental health issues use language in subtly different way than others do, the assumption of “standardised” language may be even more disenfranchising. We all agreed that this is a much bigger subject that deserves to be discussed further so keep a watch on the OHT events page for the next instalment…



References:


Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. & Gal, Y. (2024). 'AI models collapse when trained on recursively generated data.' Nature, Vol 631, pp. 755–759. https://doi.org/10.1038/s41586-024-07566-y


Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2024). "Will we run out of data? Limits of LLM scaling based on human-generated data." arXiv:2211.04325 [cs.LG].


Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901. https://arxiv.org/abs/2005.14165


Touvron et al. (2023) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., … Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models. https://arxiv.org/abs/2307.09288


Penedo et al. (2023) Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Beguelin, G., Kegl, B., & Launay, J. (2023). The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data, and web data only. arXiv preprint. https://arxiv.org/abs/2306.01116


Cipriani, A., Furukawa, T. A., Salanti, G., Chaimani, A., Atkinson, L. Z., Ogawa, Y., Leucht, S., Ruhe, H. G., Turner, E. H., Higgins, J. P. T., Egger, M., Takeshima, N., Hayasaka, Y., Imai, H., Shinohara, K., Tajika, A., Ioannidis, J. P. A., & Geddes, J. R. (2018). Comparative efficacy and acceptability of 21 antidepressant drugs for the acute treatment of adults with major depressive disorder: A systematic review and network meta-analysis. The Lancet, 391(10128), 1357–1366. https://doi.org/10.1016/S0140-6736(17)32802-7


Haidt, J. (2024). The anxious generation: How the great rewiring of childhood is causing an epidemic of mental illness.Penguin Press. ISBN 9780593655030 (US hardcover); Allen Lane. ISBN 9780241647660 (UK hardcover).


Taleb, N. N. (2007). The black swan: The impact of the highly improbable. Random House. ISBN 9781400063512 (US hardcover); Allen Lane/Penguin Books. ISBN 9780713999952 (UK hardcover).


Alter, A. L., & Oppenheimer, D. M. (2009). Uniting the tribes of fluency to form a metacognitive nation. Personality and Social Psychology Review, 13(3), 219–235. https://doi.org/10.1177/1088868309341564


Kahneman, D. (2011). Thinking, fast and slow. Farrar, Straus and Giroux. ISBN 9780374275631 (US hardcover); Allen Lane. ISBN 9781846140556 (UK hardcover).


Weizenbaum, J. (1966). ELIZA — a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1), 36–45. https://doi.org/10.1145/365153.365168

 
 
bottom of page