An intentional approach to managing bias in general purpose embedding models

Wei Hung Weng, Andrew Sellergen, Atilla P. Kiraly, Alexander D'Amour, Jungyeon Park, Rory Pilgrim, Stephen Pfohl, Charles Lau, Vivek Natarajan, Shekoofeh Azizi, Alan Karthikesalingam, Heather Cole-Lewis, Yossi Matias, Greg S. Corrado, Dale R. Webster, Shravya Shetty, Shruthi Prabhakara, Krish Eswaran, Leo A.G. Celi, Yun Liu*

*Corresponding author for this work

Research output: Contribution to journalReview articlepeer-review

Abstract

Advances in machine learning for health care have brought concerns about bias from the research community; specifically, the introduction, perpetuation, or exacerbation of care disparities. Reinforcing these concerns is the finding that medical images often reveal signals about sensitive attributes in ways that are hard to pinpoint by both algorithms and people. This finding raises a question about how to best design general purpose pretrained embeddings (GPPEs, defined as embeddings meant to support a broad array of use cases) for building downstream models that are free from particular types of bias. The downstream model should be carefully evaluated for bias, and audited and improved as appropriate. However, in our view, well intentioned attempts to prevent the upstream components—GPPEs—from learning sensitive attributes can have unintended consequences on the downstream models. Despite producing a veneer of technical neutrality, the resultant end-to-end system might still be biased or poorly performing. We present reasons, by building on previously published data, to support the reasoning that GPPEs should ideally contain as much information as the original data contain, and highlight the perils of trying to remove sensitive attributes from a GPPE. We also emphasise that downstream prediction models trained for specific tasks and settings, whether developed using GPPEs or not, should be carefully designed and evaluated to avoid bias that makes models vulnerable to issues such as distributional shift. These evaluations should be done by a diverse team, including social scientists, on a diverse cohort representing the full breadth of the patient population for which the final model is intended.

Original languageEnglish
Pages (from-to)e126-e130
JournalThe Lancet Digital Health
Volume6
Issue number2
DOIs
StatePublished - Feb 2024
Externally publishedYes

Funding

FundersFunder number
Boris Babenko
Naama Hammel
Google

    Fingerprint

    Dive into the research topics of 'An intentional approach to managing bias in general purpose embedding models'. Together they form a unique fingerprint.

    Cite this