AI chatbots fall for fake diseases and phony studies

In 2024, Swedish researchers led by Almira Osmanovic Thunström at the University of Gothenburg fabricated an entirely fictitious eye condition called “bixonimania” to test how AI chatbots handle medical misinformation. The fake disease was described as causing pinkish eyelids due to excessive screen time or eye-rubbing. Despite obvious clues indicating its fictional nature, major AI chatbots—including OpenAI’s ChatGPT, Google Gemini, Microsoft Copilot, and Perplexity—accepted and treated bixonimania as a legitimate medical condition, even going so far as to recommend specialist consultations. This experiment highlighted a significant vulnerability: AI language models often reflect the lack of skepticism humans exhibit when processing information.

The researchers intended this as a critique of both AI and human trust in unverified data, rather than a straightforward “gotcha” moment for the technology. OpenAI acknowledged ongoing efforts to improve chatbot accuracy and mitigate such risks, noting the involvement of clinical advisors to stress-test models. However, this incident raises critical concerns about how AI-generated medical advice can mislead users, potentially causing patient harm through inaccurate or fabricated diagnoses.

Given that AI chatbot misuse is now considered a top health technology hazard slated for 2026, healthcare practitioners—especially nurses frequently interacting with patients influenced by AI outputs—are urged to develop AI literacy and exercise caution when patients present AI-informed health claims. This situation underscores the broader challenges of integrating AI responsibly in healthcare and the need for rigorous oversight to safeguard public health.