Record STMT-WFFFF7AVoluntary

Grounded Instruction Understanding with Large Language Models: Toward Trustworthy Human-Robot Interaction

Report Institutional policyGlobaladopted

This research paper investigates the use of large language models and vision-language models to interpret unconstrained natural language instructions in human-robot interaction scenarios. The authors evaluate three classification approaches—a traditional neural network, text-only GPT-4, and multimodal GPT-4 Vision—using a newly constructed dataset called SCOUT++. The findings highlight both the potential and limitations of foundation models for grounding language in visual context within safety-critical robotics settings.

Summary and content fields are drawn from the document text and are pending editorial review.

Who issued it

Association for the Advancement of Artificial IntelligenceAcademic

Publisher

University of Maryland, Baltimore CountyAcademic

Lead issuer

Legal forceVoluntary

Under whose authorityInstitutional policy

Judged fromfull text

Who it governs

Researchers and academicsAI developers and providersAI deployers and users

AI researchers and robotics developers studying grounded language understanding and vision-language models for autonomous systems.

Sectors

Research and science · artificial intelligence and robotics researchDefence and security · urban disaster response

What the document contains

What's covered

The paper investigates instruction-response classification for human-robot dialogue using neural networks and GPT-4 models. It details the construction of the SCOUT++ multimodal dataset and compares text-only versus multimodal classification approaches.

What's missing

The document does not provide legislative regulations, enforcement mechanisms, or formal safety standards for commercial robot deployment.

Key takeaways

Large language models can serve as a representational bridge connecting natural language instructions to robot actions.
Extending the SCOUT corpus into SCOUT++ provides over 11,000 time-aligned visual frames paired with language commands.
Text-only GPT-4 achieved higher raw instruction classification accuracy than traditional neural networks, though with lower precision across diverse labels.
The multimodal variant of GPT-4 Vision underperformed in raw classification accuracy and exhibited challenges with hallucinations and vague outputs.
Informational asymmetries between human commanders and automated vision models complicate direct performance comparisons in robotic tasks.

Outline

· Abstract
· Introduction
· Background: The SCOUT Corpus
· Related Work
· Approach
· Results
· Discussion
· Future Work
· Conclusion

Issued2025

Length6,072 words · 9 pages

FormatNot recorded

Scopeglobal

Physical AI Governancecore

human_robot_interaction large_language_models vision_language_models multimodal_perception robotics_safety