Research indicates that more than half of the Council of England’s use of artificial intelligence tools minimizes women’s physical and mental health issues, raising concerns about potential gender bias in care decisions. The study revealed that when generating and summarizing identical case notes using Google’s AI tool “Gemma,” terms like “invalid,” “impossible,” and “complex” appeared significantly more often in descriptions of males than females.
Conducted by the London School of Economics and Political Science (LSE), the study found that comparable care needs in women were more likely to be overlooked or inadequately explained. Dr. Samurikman, the report’s lead author and a researcher at LSE’s Care Policy and Assessment Centre, emphasized that AI could result in “unequal care provision for women.” He noted, “These models are widely used, yet our findings reveal significant disparities regarding bias across different models. Specifically, Google’s models understate women’s physical and mental health needs compared to those for men.”
Furthermore, he pointed out that the care received is often determined by perceived needs, which could lead to women receiving inadequate care if a biased model is in use—although it remains unclear which model is currently being applied.
As AI tools grow in popularity among local authorities, the LSE study analyzed real case notes from 617 adult social care users. These notes were anonymized by gender and input multiple times into various major language models (LLM). Researchers examined a summary of 29,616 pairs to assess how male and female cases were treated differently by the AI model.
One example highlighted that the Gemma model summarized case notes as follows: “Mr. Smith is an 84-year-old man living alone with a complicated medical history, a care package, and poor mobility.” Conversely, when the gender was swapped, the summary read: “Mrs. Smith is an 84-year-old resident. Despite her limitations, she is independent and can maintain personal care.” In another instance, the summary stated that Mrs. Smith “has no access to the community,” while Mr. Smith “has managed to manage her daily activities.”
Among the AI models assessed, Google’s Gemma exhibited a more significant gender-based disparity compared to other models. The study noted that Meta’s Llama 3 model did not differentiate its language based on gender.
Dr. Rickman commented that although the tool “is already in use in the public sector, it should not compromise fairness.” He added, “My research sheds light on the issues posed by a single model, but with many models continuously being deployed, it is imperative that all AI systems are transparent, rigorously tested for bias, and subject to stringent legal oversight.”
The paper concludes that to prioritize “algorithm equity,” regulators should mandate measures of bias in LLMs used in long-term care. Concerns regarding racial and gender bias in AI tools have persisted for an extended period, as machine learning technology tends to absorb biases present in human languages. Our research analyzed 133 AI systems across various industries, revealing that approximately 44% exhibited gender bias, while 25% showed both gender and racial biases.
According to Google, the team is reviewing the report’s findings. The researcher assessed the initial generation of the GEMMA model, which is currently in its third generation and is expected to show improved performance; however, it should not be utilized for medical purposes.
Source: www.theguardian.com
