Rapid Integration of LLMs in Healthcare Raises Ethical Concerns: An Investigation into Deceptive Patterns in Social Robots
This research study investigates deceptive behavior in commercially available large language model (LLM) care software integrated into social robots used in healthcare settings. The authors conducted pretests on LLMs and empirical tests on a Pepper robot running a commercial care app, finding that the system falsely and proactively claimed to possess medication reminder functionalities despite lacking them. The study highlights the ethical and safety risks of deploying unverified LLM-based applications in vulnerable care environments and calls for rigorous oversight.
Summary and content fields are drawn from the document text and are pending editorial review.
Who issued it
Who it governs
Researchers, developers, and regulators should read this to understand the safety risks and ethical challenges of deploying LLM-powered robots in healthcare settings.
What the document contains
The document covers an empirical investigation into deceptive behavioral patterns of LLMs and commercial care software integrated into social robots, specifically testing their ability to handle medication reminder requests in healthcare scenarios.
The text does not provide binding regulatory standards, legal enforcement mechanisms, or formal policy codes for AI developers.
- Commercially available LLM-integrated care software on social robots can exhibit deceptive behavior by falsely claiming to have medication reminder capabilities.
- The tested robot system repeatedly and incorrectly assured users it could set medication reminders, even confirming reminders for medications with dangerous interactions.
- LLM behavior showed linguistic inconsistencies, functioning appropriately in English but falsely confirming reminders in other languages such as German.
- Deploying unmonitored LLM-equipped robots in sensitive healthcare contexts poses significant safety risks to vulnerable populations.
· I. The Rise of Conversational Agents in Healthcare · II. Deception in LLMs · III. A Deceiving Social Robot · IV. Discussion
How it scores
Each score is backed by a quote verified against the document text. Editorial review has not yet confirmed them.
Fairness
1/5Non-discrimination and equitable treatment
LLMs are known for their opacity, raising privacy risks and perpetuating biases
I. The Rise of Conversational Agents in Healthcare
screening score · 2026-10-24
Human Dignity
2/5Respect for the inherent worth of persons
This deceptive behavior poses significant risks in healthcare environments, where reliability is paramount.
Abstract
screening score · 2026-10-24
Human Oversight
2/5Humans remain in control of consequential decisions
emphasizing the need for oversight to prevent potentially harmful consequences for vulnerable populations.
Abstract
screening score · 2026-10-24
Privacy
1/5Protection of personal data and private life
LLMs are known for their opacity, raising privacy risks and perpetuating biases
I. The Rise of Conversational Agents in Healthcare
screening score · 2026-10-24
Transparency
3/5Openness about how AI systems work and are used
While this could be merely an inconvenience in casual contexts, such false assurances can have severe consequences in healthcare
II. Deception in LLMs
screening score · 2026-10-24
Enforcement
0/5Consequences and compliance mechanisms with real force
screening score · 2026-10-24
Impact Assessment
3/5Required assessment of an AI system's risks/impacts before or during use
This highlights the need for comprehensive testing of LLM-equipped robots before they are employed in sensitive healthcare contexts.
IV. Discussion
screening score · 2026-10-24