# US 11,029,767 B2 · System and method for determining 3D orientation of a pointing device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,029,767 B2 - Document ID: US11029767B2 - Status: Granted - Kind: B2 - Application number: 16/380,455 - Filed: 2019-04-10 - Published: 2019-08-01 - Granted: 2021-06-08 - Earliest claimed: 2002-02-07 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Andrew Wilson - Web page: https://patentgrep.com/patent/US11029767B2 ## CPC - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry - G08C 17/00: Arrangements for transmitting signals characterised by the use of a wireless electrical link - G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer - G08C 2201/31: Voice input - G08C 2201/32: Remote control based on movements, attitude of remote control device - G08C 2201/41: Remote control of gateways - G08C 2201/50: Receiving or transmitting feedback, e.g. replies, status updates, acknowledgements, from the controlled devices - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G08C: TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS (fluid pressure transmitting systems F15B; mechanical means for transferring the output of a sensing member into a different variable G01D5/00; mechanical control systems G05G) ## Abstract The present invention is directed toward a system and process that controls a group of networked electronic components using a multimodal integration scheme in which inputs from a speech recognition subsystem, gesture recognition subsystem employing a wireless pointing device and pointing analysis subsystem also employing the pointing device, are combined to determine what component a user wants to control and what control action is desired. In this multimodal integration scheme, the desired action concerning an electronic component is decomposed into a command and a referent pair. The referent can be identified using the pointing device to identify the component by pointing at the component or an object associated with it, by using speech recognition, or both. The command may be specified by pressing a button on the pointing device, by a gesture performed with the pointing device, by a speech recognition event, or by any combination of these inputs. ## Claims ### Claim 1 A system for gesture recognition, comprising: a camera configured to image a real-world environment; a computing device communicatively coupled with the camera, the computing device configured to: process images of the real-world environment captured by the camera to track a three-dimensional movement of a handheld device separate from the computing device; receive, from the handheld device, movement data measured by an inertial measurement unit (IMU) of the handheld device during the three-dimensional movement of the handheld device; based on the images of the real-world environment captured by the camera and the movement data received from the handheld device, classify the three-dimensional movement of the handheld device as a recognized gesture; and at the computing device, perform a control action in response to the recognized gesture of the handheld device. ### Claim 2 (depends on 1) The system of claim 1, where the computing device is further configured to receive a button actuation status of one or more physical buttons of the handheld device, and further classify the three-dimensional movement of the handheld device as the recognized gesture based on the button actuation status. ### Claim 3 (depends on 1) The system of claim 1, further comprising a microphone communicatively coupled with the computing device, the microphone configured to record sounds in the real-world environment, and where the computing device is further configured to classify the three-dimensional movement of the handheld device as the recognized gesture based on a verbal user command recorded by the microphone. ### Claim 4 (depends on 1) The system of claim 1, where said classifying the three-dimensional movement of the handheld device as the recognized gesture includes comparing the movement data received from the handheld device to a prototype sequence of a first movement data corresponding to the recognized gesture. ### Claim 5 (depends on 4) The system of claim 4, where the prototype sequence of the first movement data corresponding to the recognized gesture corresponds to a training three-dimensional movement of the handheld device performed during a training phase. ### Claim 6 (depends on 4) The system of claim 4, where said comparing the movement data received from the handheld device to the prototype sequence of the first movement data includes generating a plurality of modified versions of the prototype sequence by applying one or both of a plurality of scaling factors and a plurality of warping factors to the prototype sequence. ### Claim 7 (depends on 4) The system of claim 4, where the computing device maintains a plurality of different prototype sequences of a plurality of different movement data corresponding to a plurality of different candidate gestures. ### Claim 8 (depends on 7) The system of claim 7, where said classifying the three-dimensional movement of the handheld device as the recognized gesture includes calculating a similarity indicator between the movement data received from the handheld device and each different movement data of the plurality of different movement data of each of the plurality of prototype sequences corresponding to the plurality of different candidate gestures, and selecting a candidate gesture from the plurality of different candidate gestures having a highest similarity indicator as the recognized gesture. ### Claim 9 (depends on 8) The system of claim 8, where the candidate gesture having the highest similarity indicator is selected as the recognized gesture after determining that the highest similarity indicator exceeds a predetermined similarity threshold. ### Claim 10 (depends on 1) The system of claim 1, where the movement data received from the handheld device includes data measured by one or more of an accelerometer, gyroscope, and magnetometer of the IMU. ### Claim 11 (depends on 1) The system of claim 1, where the camera is further configured to detect Infrared (IR) light output by an IR emitter of the handheld device. ### Claim 12 (depends on 1) The system of claim 1, where said processing the images of the real-world environment captured by the camera includes identifying a pointing direction of the handheld device, and where the three-dimensional movement of the handheld device is further classified as the recognized gesture based on the pointing direction. ### Claim 13 (depends on 1) The system of claim 1, where the control action performed by the computing device controls an electronic device separate from the computing device. ### Claim 14 A method for recognizing gestures, comprising: at a computing device, processing images of a real-world environment captured by a camera to track a three-dimensional movement of a handheld device separate from the computing device; receiving movement data from the handheld device measured by an inertial measurement unit (IMU) of the handheld device during the three-dimensional movement of the handheld device; based on the images of the real-world environment captured by the camera and the movement data received from the handheld device, classifying the three-dimensional movement of the handheld device as a recognized gesture; and at the computing device, performing a control action in response to the recognized gesture of the handheld device. ### Claim 15 (depends on 14) The method of claim 14, further comprising receiving a button actuation status of one or more physical buttons of the handheld device, and where the three-dimensional movement of the handheld device is further classified as the recognized gesture based on the button actuation status. ### Claim 16 (depends on 14) The method of claim 14, further comprising, via a microphone, recording sounds in the real-world environment, and where the three-dimensional movement of the handheld device is classified as the recognized gesture based on a verbal user command recorded by the microphone. ### Claim 17 (depends on 14) The method of claim 14, where said classifying the three-dimensional movement of the handheld device as the recognized gesture includes comparing the movement data received from the handheld device to a prototype sequence of a first movement data corresponding to the recognized gesture. ### Claim 18 (depends on 14) The method of claim 14, where the movement data received from the handheld device includes data measured by one or more of an accelerometer, gyroscope, and magnetometer of the IMU. ### Claim 19 (depends on 14) The method of claim 14, where said processing the images of the real-world environment captured by the camera includes identifying a pointing direction of the handheld device, and where the three-dimensional movement of the handheld device is further classified as the recognized gesture based on the pointing direction. ### Claim 20 A system for gesture recognition, comprising: a camera configured to image a real-world environment; a microphone configured to record sounds of the real-world environment; a computing device communicatively coupled with the camera and microphone, the computing device configured to: process images of the real-world environment captured by the camera to track a three-dimensional movement of a handheld device separate from the computing device; process sounds recorded by the microphone to identify a verbal user command; receive, from the handheld device, movement data measured by an inertial measurement unit (IMU) of the handheld device during the three-dimensional movement of the handheld device; based on the images of the real-world environment captured by the camera and the movement data received from the handheld device, classify the three-dimensional movement of the handheld device as a recognized gesture at least by comparing the movement data to a prototype sequence of a first movement data corresponding to the recognized gesture; and at the computing device, perform a control action in response to the recognized gesture of the handheld device.