Anthropomorphic AI: A Toolkit for Authoring and Interacting with Intelligent Virtual Agents for Extended Reality

DOI

Intelligent Virtual Agents (IVAs), which embody an artificial intelligence (AI) in a humanoid representation, have enormous potential for immersive extended reality (XR) environments to enable natural and engaging human-AI interactions. With recent advances in large language models (LLMs) in simulating human-like text responses, interest in anthropomorphic embodied IVAs has grown across XR research and application domains. However, toolkits for authoring and interacting with IVAs in research remain sparse.

Therefore, we present Anthropomorphic AI, a flexible and scalable open-source research toolkit for authoring and interacting with embodied IVAs with rich multimodal capabilities, including speech, gaze, gestures, facial expressions, and vision. Our system enables developers to create various embodied anthropomorphic IVAs by customizing behavior through expressive nonverbal cues, selecting and combining different foundation models, speech-to-text (STT) and text-to-speech (TTS) methods, and adapting the system prompt to guide interaction. We also integrate various features such as proximity detection, trajectory-based action recognition, and vision-based multimodal prompting for supporting natural human-IVA interaction in immersive XR. We evaluate the toolkit through four use case demonstrations, a pilot developer evaluation, and an end-user study in immersive VR, showing its capability in generating socially engaging, highly usable, and likable anthropomorphic IVAs for immersive XR applications.

Identifier
DOI https://doi.org/10.25592/uhhfdm.17776
Related Identifier IsPartOf https://doi.org/10.25592/uhhfdm.17775
Metadata Access https://www.fdr.uni-hamburg.de/oai2d?verb=GetRecord&metadataPrefix=oai_datacite&identifier=oai:fdr.uni-hamburg.de:17776
Provenance
Creator Li, Ke; Mostajeran, Fariba; Rings, Sebastian; Hertel, Julia; Schmidt, Susanne; Arz, Michael; Steinicke, Frank
Publisher Universität Hamburg
Publication Year 2025
Rights Closed Access; info:eu-repo/semantics/closedAccess
OpenAccess false
Representation
Resource Type Journal article; Text
Discipline Other