Certain demographics of people run the risk of receiving inaccurate or outdated medical information when asking chatbots for advice – and so should remain wary, an expert has said
While artificial intelligence continues to rise in prominence and people turn to the technology for medical help, women and people of colour have been warned about following advice from chatbots.
Magali Geens, who led the STADA Health Report 2025 on health trends in the UK and Europe, explained that AI will not always provide accurate health advice for women and anyone who isn’t white.
When we ask AI systems like Chat GPT for guidance, it will pull together studies and data from the internet to provide its response. However, this existing knowledge is often outdated.
A large amount of the research into health issues has historically been centred around men, with them often being the default subjects in studies. Caucasians are also overrepresented in the available studies on the internet that Chat GPT uses to form its advice.
Geens explained: “If [women] use Chat GPT and put in symptoms, you will get data and information that is not for you.”
She said that to get the best out of AI when it comes to health advice, you will need tools that are up-to-date, relevant, and that eliminate bias. Geens used specific chatbots designed for mental health problems as a good example.
READ MORE: ‘Super humans’ fears as scientists aim to create first human genome in a labREAD MORE: ‘I use AI to make me a better dad – it helps me settle arguments and is a great teacher’
Using AI like a search engine to ask for medical advice will result in you getting inaccurate information, or results that will confirm bias. You would need to provide more details, such as your height, weight, family history, and gender to get a tailored response – but many AI users are unaware of this.
While AI can be a useful tool for those in rural areas to access health advice, it is vital for users to remain critical of the information they are presented with. Geens told us: “This is a challenge that will take a few more years to crack.”
Peter Goldschmidt, CEO of pharmaceutical company STADA, added that AI can provide “information, but not competent judgement”.
Women already have less trust in healthcare systems than men. According to the STADA survey of 27,000 people from 22 counties, 55% of women trust healthcare systems, compared to 61% of men.
AI continues to rise in popularity and play a role in education, science, and now, healthcare. Of those surveyed, 39% said they would consider getting a medical consultation from AI instead of seeing a doctor – with most citing accessibility and convenience as the reasons why.
The technology is already in use in the UK healthcare system, with the UK Health Security Agency harnessing AI to understand patient experiences, detect food-borne illness outbreaks, and make public health guidance more consistent.
Dr Nick Watkins, chief data scientist at UKHSA, said that AI could “enhance public health protection”.
He added: “As we continue to develop and refine these systems, we maintain a careful balance between embracing innovation and ensuring robust validation of AI outputs. This approach helps us harness AI’s potential while maintaining the high standards expected of a national public health agency.”