# US 12,602,866 B2 · Digital twin authoring and editing environment for creation of AR/VR and video instructions from a single demonstration > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,866 B2 - Document ID: US12602866B2 - Status: Granted - Kind: B2 - Application number: 18/480,173 - Filed: 2023-10-03 - Published: 2024-04-04 - Granted: 2026-04-14 - Earliest claimed: 2022-10-03 - Assignees: Purdue Research Foundation - Applicants: Purdue Research Foundation - Inventors: Karthik Ramani; Subramaniam Chidambaram; Sai Swarup Reddy - Web page: https://patentgrep.com/patent/US12602866B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/73: using feature-based methods - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06V 40/11: {Hand-related biometrics; Hand pose recognition} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G09B 5/06: with both visual and audible presentation of the material to be studied - G09B 19/003: {Repetitive work cycles; Sequence of movements} - H04N 23/611: where the recognised objects include parts of the human body - G06T 2207/30196: Human being; Person - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system and method for multi-media instruction authoring is disclosed. The multi-media instruction authoring system and method provide a unified and efficient system that supports the simultaneous creation of instructional media in AR, VR, and video-based media formats by subject matter experts. The multi-media instruction authoring system and method require only a single demonstration of the task by the subject matter expert and does not require any technical expertise of the subject matter expert. The multi-media instruction authoring system and method incorporate a 3D editing interface for free hand, in-headset interaction to create 2D videos from 3D recordings, by exploring novel virtual camera interactions. ## Claims ### Claim 1 A method for generating multi-media instructional content, the method comprising: recording, with at least one sensor, a demonstration by a user of a task within a real-world environment; determining, with a processor, a first sequence of pose data for a virtual avatar based on the recorded demonstration; generating, with the processor, based on the first sequence of pose data, at least one of augmented reality content and virtual reality content that provides instructions to perform the task, the at least one of the augmented reality content and the virtual reality content including a virtual model of the virtual avatar associated with the first sequence of pose data; and generating, with the processor, based on the first sequence of pose data, a two-dimensional video that provides instructions to perform the task, the two-dimensional video being generated using a virtual camera that views the virtual model of the virtual avatar within a virtual environment and by animating the virtual model of the virtual avatar to move according to the first sequence of pose data. ### Claim 2 (depends on 1) The method according to claim 1, the generating the at least one of the augmented reality content and the virtual reality content comprising: generating both of the augmented reality content and the virtual reality content. ### Claim 3 (depends on 1) The method according to claim 1, wherein the at least one sensor includes a sensor of a head-mounted augmented reality device, recorded sensor data from which is used to determine the first sequence of pose data. ### Claim 4 (depends on 1) The method according to claim 1, wherein the first sequence of pose data includes hand poses for the virtual avatar. ### Claim 5 (depends on 1) The method according to claim 1 further comprising: determining, with the processor, a second sequence of pose data for a virtual object based on the recorded demonstration, the demonstration including the user performing an operation using a real-world object corresponding to the virtual object. ### Claim 6 (depends on 5) The method according to claim 5, wherein the at least one sensor includes a sensor attached directly to the real-world object, recorded sensor data from which is used to determine the second sequence of pose data. ### Claim 7 (depends on 5) The method according to claim 5 further comprising: detecting that the user interacts with the real-world object, wherein the recording the demonstration automatically begins in response to detecting that the user has interacted with the real-world object. ### Claim 8 (depends on 1) The method according to claim 1 further comprising: recording, with a camera, during the demonstration, a two-dimensional video of the demonstration, wherein at least one of (i) the augmented reality content and (ii) the virtual reality content includes the two-dimensional video embedded therein. ### Claim 9 (depends on 1) The method according to claim 1 further comprising: recording, with a camera, a hand gesture performed by the user; pairing, with the processor, based on user inputs, the hand gesture with a user-selected operation; detecting, with the processor, after the pairing, performance of the hand gesture; and performing, with the processor, the user-selected operation in response to detecting the performance of the hand gesture. ### Claim 10 (depends on 1) The method according to claim 1, wherein the augmented reality content includes a plurality of virtual models and a plurality of time sequences of pose data, the plurality of virtual models including the virtual model of the virtual avatar, each respective virtual model of the plurality of virtual models being associated with a respective sequence of pose data of the plurality of time sequences of pose data, the plurality of time sequences of pose data including the first sequence of pose data, the method further comprising: displaying, on a display of an augmented reality device, each respective virtual model of the plurality of virtual models superimposed on the real-world environment and animated to move according to the respective sequence of pose data of the plurality of time sequences of pose data. ### Claim 11 (depends on 1) The method according to claim 1, wherein the virtual reality content includes a plurality of virtual models and a plurality of time sequences of pose data, the plurality of virtual models including the virtual model of the virtual avatar, each respective virtual model of the plurality of virtual models being associated with a respective sequence of pose data of the plurality of time sequences of pose data, the plurality of time sequences of pose data including the first sequence of pose data, the method further comprising: displaying, on a display of a virtual reality device, each respective virtual model of the plurality of virtual models within a virtual environment and animated to move according to the respective sequence of pose data of the plurality of time sequences of pose data. ### Claim 12 (depends on 1) The method according to claim 1, the generating the two-dimensional video further comprising: generating, with the processor, the two-dimensional video using the virtual camera, which views a plurality of virtual models within the virtual environment, the plurality of virtual models including the virtual model of the virtual avatar, wherein, in the two-dimensional video, each respective virtual model of the plurality of virtual models is animated to move according to a respective time sequence of pose data of a plurality of time sequences of pose data, the plurality of time sequences of pose data including the first sequence of pose data. ### Claim 13 (depends on 12) The method according to claim 12, the generating the two-dimensional video further comprising: displaying, on a display of at least one of an augmented reality device and a virtual reality device, a graphical user interface; defining, based on user inputs, at least one pose of the virtual camera; and generating, with the processor, the two-dimensional video using the virtual camera having the defined camera pose. ### Claim 14 (depends on 13) The method according to claim 13, the generating the two-dimensional video further comprising: defining, based on user inputs, a timeline of the two-dimensional video including a first time segment of the two-dimensional video and a second time segment of the two-dimensional video; defining, based on user inputs, for the first time segment of the timeline, a first pose of the virtual camera and a first time segment of the plurality of time sequences of pose data; defining, based on user inputs, for the second time segment of the timeline, a second pose of the virtual camera and a second time segment of the plurality of time sequences of pose data; and generating (i) the first time segment of the two-dimensional video using the virtual camera having the first camera pose and with the plurality of virtual models animated to move according to the first time segment of the plurality of time sequences of pose data and (ii) the second time segment of the two-dimensional video using the virtual camera having the second camera pose and with the plurality of virtual models animated to move according to the second time segment of the plurality of time sequences of pose data. ### Claim 15 (depends on 13) The method according to claim 13, wherein, for a first time segment of the two-dimensional video, the virtual camera has a first camera pose in which the virtual camera has a static position and orientation within the virtual environment. ### Claim 16 (depends on 13) The method according to claim 13, wherein, for a second time segment of the two-dimensional video, the virtual camera has a second camera pose in which the virtual camera moves dynamically through the virtual environment over time according to a user-defined trajectory. ### Claim 17 (depends on 13) The method according to claim 13, wherein, for a third time segment of the two-dimensional video, the virtual camera has a third camera pose in which the virtual camera moves dynamically through the virtual environment over time according to a trajectory that depends on a current state of at least one virtual model of the plurality of virtual models. ### Claim 18 (depends on 13) The method according to claim 13 further comprising: displaying, within the graphical user interface on the display, a preview of the two-dimensional video. ### Claim 19 (depends on 13) The method according to claim 13 further comprising: displaying, within the graphical user interface on the display, the plurality of virtual models within the virtual environment, each respective virtual model of the plurality of virtual models being animated to move according to a respective time sequence of pose data of the plurality of time sequences of pose data. ### Claim 20 (depends on 19) The method according to claim 19, the displaying the plurality of virtual models within the virtual environment further comprising: selecting, based on user inputs, a selected point in time of the plurality of time sequences of pose data; and displaying, within the graphical user interface on the display, a state of the plurality of virtual models at the selected point in time.