Integrating Large Language Models with Multimodal Virtual Reality Interfaces To Support Collaborative Human-Robot Construction Work

A computer-implemented method comprising:

• accepting input commands comprising speech and gesture inputs to specify task parameters within a virtual environment;
• combining the speech inputs with gesture-based inputs to generate integrated task instructions;
• retrieving object and environmental data from a building information model associated with the virtual environment;
• displaying a confirmation prompt to verify parameters interpreted from the integrated task instructions;
• validating constraints related to the task instructions using object and environmental data;
• and
• transmitting confirmed task parameters to a robotic control system for execution within a construction simulation.

A computer-readable medium having stored thereon instructions that, when executed, cause a computer to:

• accept user commands comprising speech and gesture inputs to specify virtual tasks;
• process and integrate speech and gesture inputs into task instructions;
• retrieve relevant object and contextual data from a building information model;
• confirm user intentions by verifying interpreted task details through user interaction;
• validate the task against operational constraints;
• and
• output commands to a robotic system for simulated task execution.