# US 11,029,753 B2 · Human computer interaction system and human computer interaction method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,029,753 B2 - Document ID: US11029753B2 - Status: Granted - Kind: B2 - Application number: 16/673,987 - Filed: 2019-11-05 - Published: 2021-05-06 - Granted: 2021-06-08 - Earliest claimed: 2019-11-05 - Assignees: Xrspace Co., Ltd. - Applicants: Xrspace Co., Ltd. - Inventors: Yen-Hung Lin; Chieh-Kai Wang; Shih-Hao Ke; Wei-Chi Yen - Web page: https://patentgrep.com/patent/US11029753B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 2207/30196: Human being; Person - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A human computer interaction system and a human computer interaction method are provided. The human computer interaction system includes a first sensor, a second sensor and a processor. The processor obtains first motion sensing data of the first operating portion from the first sensor, obtains second motion sensing data of the second operating portion from the second sensor different from the first operating portion, and determines an event according to both the first motion sensing data and the second motion sensing data. Accordingly, a hybrid motion tracking method for multiple operating portion of the user is provided. ## Claims ### Claim 1 A human computer interaction method, comprising: obtaining first motion sensing data based on a first sensor, wherein the first motion sensing data is related to a first operating portion of a user and generated based on a determination whether an analysis result of a motion of the first operating portion based on one of a first motion sensor and a first image sensor is reliable, the first sensor comprises the first image sensor and the first motion sensor, the first motion sensor is acted with the first operating portion of the user, and the first image sensor captures images of the first operating portion of the user, and the step of obtaining the first motion sensing data further comprises: using the images captured from the first image sensor as the first motion sensing data in response to the analysis result of the first operating portion of the user based on the first image sensor being reliable; and using data obtained from the first motion sensor as the first motion sensing data in response to the analysis result of the first operating portion of the user based on the first image sensor being unreliable; obtaining second motion sensing data, wherein the second motion sensing data is related to a second operating portion of the user different from the first operating portion and generated based on a second sensor; and triggering an event according to both the first motion sensing data and the second motion sensing data. ### Claim 2 (depends on 1) The human computer interaction method according to claim 1, wherein the second sensor is an image sensor, and the second sensor captures image of the second operating portion of the user. ### Claim 3 (depends on 1) The human computer interaction method according to claim 1, wherein both of the images captured from the first image sensor and the data obtained from the first motion sensor are used as the first motion sensing data. ### Claim 4 (depends on 3) The human computer interaction method according to claim 3, wherein the second sensor comprises a second motion sensor and a second image sensor, the second motion sensor is acted with the second operating portion of the user, the second image sensor captures image of the second operating portion of the user, and both of images captured from the second image sensor and data obtained from the second motion sensor are used as the second motion sensing data. ### Claim 5 (depends on 1) The human computer interaction method according to claim 1, wherein the analysis result comprises a first image analysis result, and the step of obtaining the first motion sensing data further comprises: determining whether the first image analysis result is reliable, wherein the first image analysis result is related to the motion of the first operating portion of the user and determined based on the images captured from the first image sensor; using the images captured from the first image sensor as the first motion sensing data in response to the first image analysis result of the first operating portion of the user being reliable; and using the data obtained from the first motion sensor as the first motion sensing data in response to the first image analysis result of the first operating portion of the user being unreliable. ### Claim 6 (depends on 5) The human computer interaction method according to claim 5, wherein the second sensor comprise a second motion sensor and a second image sensor, the second motion sensor is acted with the second operating portion of the user, the second image sensor captures images of the second operating portion of the user, and the step of obtaining the second motion sensing data comprises: determining whether a second image analysis result is reliable, wherein the second image analysis result is related to the motion of the second operating portion of the user and determined based on the images captured from the second image sensor; using the images captured from the second image sensor as the second motion sensing data in response to the second image, analysis result of the second operating portion of the user being reliable; and using data obtained from the second motion sensor as the second motion sensing data in response to the second image analysis result of the second operating portion of the user being unreliable. ### Claim 7 (depends on 5) The human computer interaction method according to claim 5, wherein the second sensor comprise a second motion sensor and a second image sensor, the second motion sensor is acted with the second operating portion of the user, the second image sensor captures images of the second operating portion of the user, and the step of obtaining the second motion sensing data comprises: determining whether a first data analysis result is reliable, wherein the first data image analysis result is related to the motion of the second operating portion of the user and determined based on the images captured from the second image sensor; using the data captured from the second motion sensor as the second motion sensing data in response to the first data analysis result of the second operating portion of the user being reliable; and using the images captured from the second image sensor as the second motion sensing data in response to the first data analysis result of the second operating portion of the user being unreliable. ### Claim 8 (depends on 1) The human computer interaction method according to claim 1, wherein the analysis result comprises a second data analysis result, and the step of obtaining the first motion sensing data further comprises: determining whether the second data analysis result is reliable, wherein the second data analysis result is related to the motion of the first operating portion of the user and determined based on data obtained from the first motion sensor; using the data obtained from the first motion sensor as the first motion sensing data in response to the second data analysis result of the first operating portion of the user being reliable; and using the images captured from the first image sensor as the first motion sensing data in response to the second data analysis result of the first operating portion of the user being unreliable. ### Claim 9 (depends on 8) The human computer interaction method according to claim 8, wherein the second sensor comprise a second motion sensor and a second image sensor, the second motion sensor is acted with the second operating portion of the user, the second image sensor captures images of the second operating portion of the user, and the step of obtaining the second motion sensing data comprises: determining whether a third data analysis result is reliable, wherein the third data analysis result is related to the motion of the second operating portion of the user and determined based on data obtained from the second motion sensor; using data obtained from the second data sensor as the second motion sensing data in response to the third data analysis result of the second operating portion of the user being reliable; and using the images captured from the second image sensor as the second motion sensing data in response to the third image analysis result of the second operating portion of the user being unreliable. ### Claim 10 (depends on 1) The human computer interaction method according to claim 1, wherein the event is related to at least one command for controlling a virtual object in a virtual environment or a real object in a real environment. ### Claim 11 A human computer interaction system, comprising: a first sensor, comprising a first image sensor and a first motion sensor, wherein the first motion sensor is acted with a first operating portion of a user, and the first image sensor captures images of the first operating portion of the user; a second sensor; and a processor, configured to perform: obtaining first motion sensing data of a first operating portion from the first sensor, wherein the first motion sensing data related to the first operating portion is generated based on a determination whether an analysis result of a motion of the first operating portion based on one of the first motion sensor and the first image sensor is reliable, and the processor is further configured to perform: using the images captured from the first image sensor as the first motion sensing data in response to the analysis result of the first operating portion of the user based on the first image sensor being reliable; and using data obtained from the first motion sensor as the first motion sensing data in response to the analysis result of the first operating portion of the user based on the first image sensor being unreliable; obtaining second motion sensing data of the second operating portion from the second sensor different from the first operating portion; and triggering an user input according to both the first motion sensing data and the second motion sensing data. ### Claim 12 (depends on 11) The human computer interaction system according to claim 11, wherein an image sensor, and the second sensor captures image of the second operating portion of the user. ### Claim 13 (depends on 11) The human computer interaction system according to claim 11, wherein both of the images captured from the first image sensor and the data obtained from the first motion sensor are used as the first motion sensing data. ### Claim 14 (depends on 13) The human computer interaction system according to claim 13, wherein the second sensor comprises a second motion sensor and a second image sensor, the second motion sensor is acted with the second operating portion of the user, the second image sensor captures image of the second operating portion of the user, and both of images captured from the second image sensor and data obtained from the second motion sensor are used as the second motion sensing data. ### Claim 15 (depends on 11) The human computer interaction system according to claim 11, wherein the analysis result comprises a first image analysis result, and the processor is configured to perform: determining whether the first image analysis result is reliable, wherein the first image analysis result is related to the motion of the first operating portion of the user and determined based on the images captured from the first image sensor; using the images captured from the first image sensor as the first motion sensing data in response to the first image analysis result of the first operating portion of the user being reliable; and using the data obtained from the first motion sensor as the first motion sensing data in response to the first image analysis result of the first operating portion of the user being unreliable. ### Claim 16 (depends on 15) The human computer interaction system according to claim 15, wherein the second sensor comprise a second motion sensor and a second image sensor, the second motion sensor is acted with the second operating portion of the user, the second image sensor captures images of the second operating portion of the user, and the processor is configured to perform: determining whether a second image analysis result is reliable, wherein the second image analysis result is related to the motion of the second operating portion of the user and determined based on the images captured from the second image sensor; using the images captured from the second image sensor as the second motion sensing data in response to the second image analysis result of the second operating portion of the user being reliable; and using data obtained from the second motion sensor as the second motion sensing data in response to the second image analysis result of the second operating portion of the user being unreliable. ### Claim 17 (depends on 5) The human computer interaction system according to claim 5, wherein the second sensor comprise a second motion sensor and a second image sensor, the second motion sensor is acted with the second operating portion of the user, the second image sensor captures images of the second operating portion of the user, and the processor is configured to perform: determining whether a first data analysis result is reliable, wherein the first data image analysis result is related to the motion of the second operating portion of the user and determined based on the images captured from the second image sensor; using the data captured from the second motion sensor as the second motion sensing data in response to the first data analysis result of the second operating portion of the user being reliable; and using the images captured from the second image sensor as the second motion sensing data in response to the first data analysis result of the second operating portion of the user being unreliable. ### Claim 18 (depends on 11) The human computer interaction system according to claim 11, wherein the analysis result comprises a second data analysis result, and the processor is configured to perform: determining whether the second data analysis result is reliable, wherein the second data analysis result is related to the motion of the first operating portion of the user and determined based on data obtained from the first motion sensor; using the data obtained from the first motion sensor as the first motion sensing data in response to the second data analysis result of the first operating portion of the user being reliable; and using the images captured from the first image sensor as the first motion sensing data in response to the second data analysis result of the first operating portion of the user being unreliable. ### Claim 19 (depends on 18) The human computer interaction system according to claim 18, wherein the second sensor comprise a second motion sensor and a second image sensor, the second motion sensor is acted with the second operating portion of the user, the second image sensor captures images of the second operating portion of the user, and the processor is configured to perform: determining whether a third data analysis result is reliable, wherein the third data analysis result is related to the motion of the second operating portion of the user and determined based on data obtained from the second motion sensor; using data obtained from the second data sensor as the second motion sensing data in response to the third data analysis result of the second operating portion of the user being reliable; and using the images captured from the second image sensor as the second motion sensing data in response to the third image analysis result of the second operating portion of the user being unreliable. ### Claim 20 (depends on 11) The human computer interaction system according to claim 11, wherein the event is related to at least one command for controlling a virtual object in a virtual environment or a real object in a real environment.