# US 2021/0279935 A1 · Motion Tracking and Image Recognition of Hand Gestures to Animate a Digital Puppet, Synchronized with Recorded Audio > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279935 A1 - Document ID: US20210279935A1 - Status: Published application - Kind: A1 - Application number: 17/328,137 - Filed: 2021-05-24 - Published: 2021-09-09 - Earliest claimed: 2013-12-06 - Assignees: Disney Enterprises, Inc. - Applicants: Disney Enterprises, Inc. - Inventors: Malcolm Murdock; Michael Hy; Armen Mkrtchyan - Web page: https://patentgrep.com/patent/US20210279935A1 ## CPC - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G10L 2021/105: {Synthesis of the lips movements from speech, e.g. for talking heads} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract There is provided a system and method for creating a digital puppet show by animating a digital puppet using gestures and sound. The method includes presenting the digital puppet to a user on a display. The method further includes receiving motion data corresponding to a gesture, using a camera, from the user, translating the motion data into digital data using a motion tracking algorithm, and animating the digital puppet, on the display, using the digital data. The method can further include receiving audio data from the user using a microphone and playing the audio data, using a speaker, while animating the digital puppet on the display to create the digital puppet show. ## Claims ### Claim at position 1 1-20. (canceled) ### Claim at position 2 21: A computer-readable non-transitory medium having stored thereon a digital puppet application, which when executed by a hardware processor, performs a method comprising: presenting a three-dimensional (3-D) digital puppet to a user on a display; receiving motion data corresponding to a gesture made by the user; receiving audio data corresponding to voice of the user for use in a 3-D digital puppet show; translating the motion data into digital data using a motion tracking algorithm; and creating the 3-D digital puppet show using the 3-D digital puppet by: animating the 3-D digital puppet, on the display, using the digital data; and playing the audio data at the same time as the animating of the 3-D digital puppet on the display, wherein playing the audio data includes synchronizing the audio data with the animating of the 3-D digital puppet. ### Claim at position 3 (depends on 21) 22: The computer-readable non-transitory medium of claim 21, wherein when the digital puppet application is executed by the hardware processor, further performs the method comprising: before presenting the 3-D digital puppet to the user, presenting a plurality of 3-D digital puppets to the user on the display; and receiving a user input for selecting the 3-D digital puppet from the plurality of 3-D digital puppets. ### Claim at position 4 (depends on 21) 23: The computer-readable non-transitory medium of claim 21, wherein the gesture made by the user is a hand gesture. ### Claim at position 5 (depends on 23) 24: The computer-readable non-transitory medium of claim 23, wherein a physical accessory corresponding to the 3-D digital puppet is used by the user for the hand gesture. ### Claim at position 6 (depends on 23) 25: The computer-readable non-transitory medium of claim 23, wherein a movement of fingers and a thumb of the hand gesture animates a mouth of the 3-D digital puppet. ### Claim at position 7 (depends on 21) 26: The computer-readable non-transitory medium of claim 21, wherein the digital data further includes rotational data. ### Claim at position 8 (depends on 21) 27: The computer-readable non-transitory medium of claim 21, wherein when the digital puppet application is executed by the hardware processor, further performs the method comprising: recording the animation of the 3-D digital puppet in a memory to create the 3-D digital puppet show. ### Claim at position 9 (depends on 21) 28: The computer-readable non-transitory medium of claim 21, wherein the digital data further includes mouth control data. ### Claim at position 10 29: A method comprising: presenting a three-dimensional (3-D) digital puppet to a user on a display; receiving motion data corresponding to a gesture made by the user; receiving audio data corresponding to voice of the user for use in a 3-D digital puppet show; translating the motion data into digital data using a motion tracking algorithm; and creating the 3-D digital puppet show using the 3-D digital puppet by: animating the 3-D digital puppet, on the display, using the digital data; and playing the audio data at the same time as the animating of the 3-D digital puppet on the display, wherein playing the audio data includes synchronizing the audio data with the animating of the 3-D digital puppet. ### Claim at position 11 (depends on 29) 30: The method of claim 29, further comprising: before presenting the 3-D digital puppet to the user, presenting a plurality of 3-D digital puppets to the user on the display; and receiving a user input for selecting the 3-D digital puppet from the plurality of 3-D digital puppets. ### Claim at position 12 (depends on 29) 31: The method of claim 29, wherein the gesture made by the user is a hand gesture. ### Claim at position 13 (depends on 31) 32: The method of claim 31, wherein a physical accessory corresponding to the 3-D digital puppet is used by the user for the hand gesture. ### Claim at position 14 (depends on 31) 33: The method of claim 31, wherein a movement of fingers and a thumb of the hand gesture animates a mouth of the 3-D digital puppet. ### Claim at position 15 (depends on 29) 34: The method of claim 29, wherein the digital data further includes rotational data. ### Claim at position 16 (depends on 29) 35: The method of claim 29, further comprising: recording the animation of the 3-D digital puppet in a memory to create the 3-D digital puppet show. ### Claim at position 17 (depends on 29) 36: The method of claim 29, wherein the digital data further includes mouth control data. ### Claim at position 18 37: A system for animating a three-dimensional (3-D) digital puppet, the system comprising: a display; and a processor configured to: present a three-dimensional (3-D) digital puppet to a user on a display; receive motion data corresponding to a gesture made by the user, receive audio data corresponding to voice of the user for use in a 3-D digital puppet show; translate the motion data into digital data using a motion tracking algorithm; and create the 3-D digital puppet show using the 3-D digital puppet by: animating the 3-D digital puppet, on the display, using the digital data; and playing the audio data at the same time as the animating of the 3-D digital puppet on the display, wherein playing the audio data includes synchronizing the audio data with the animating of the 3-D digital puppet. ### Claim at position 19 (depends on 37) 38: The system of claim 37, wherein the processor is further configured to: before presenting the 3-D digital puppet to the user, present a plurality of 3-D digital puppets to the user on the display; and receive a user input for selecting the 3-D digital puppet from the plurality of 3-D digital puppets. ### Claim at position 20 (depends on 37) 39: The system of claim 37, wherein the gesture made by the user is a hand gesture. ### Claim at position 21 (depends on 39) 40: The system of claim 39, wherein a physical accessory corresponding to the 3-D digital puppet is used by the user for the hand gesture.