# US 10,372,325 B2 · Electromyographic based touch interaction processing method, device, and system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,372,325 B2 - Document ID: US10372325B2 - Status: Granted - Kind: B2 - Application number: 15/486,452 - Filed: 2017-04-13 - Published: 2017-08-03 - Granted: 2019-08-06 - Earliest claimed: 2014-10-16 - Assignees: Huawei Technologies Co., Ltd. - Applicants: Huawei Technologies Co., Ltd. - Inventors: Qiang Ding; Xiaorong Gao; Xiaoshan Huang - Web page: https://patentgrep.com/patent/US10372325B2 ## CPC - G06F 3/0418: {for error correction or compensation, e.g. based on parallax, calibration or alignment} - G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/041: Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means - G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 2203/04104: Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger - G06F 2203/04105: Pressure sensors for measuring the pressure or force exerted on the touch surface without providing the touch position - G06F 2203/04808: Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A touch interaction processing method, device, and system. The touch interaction processing method includes: receiving first information sent by an electromyographic signal collection device and second information sent by a location capturing device (101); if it is determined that a time gap between a first touch start time and a second touch start time is less than a preset threshold, and a quantity of touch points that is corresponding to a hand gesture is the same as a quantity of touch points that is included in the second information, generating a touch instruction, where the touch instruction includes a device identifier of the electromyographic signal collection device, the hand gesture, and coordinate information of each touch point (103); and performing an interaction operation corresponding to the touch instruction (105). ## Claims ### Claim 1 A touch interaction processing method, comprising: receiving first information sent by an electromyographic signal collection device and second information sent by a location capturing device, wherein the first information comprises a device identifier of the electromyographic signal collection device, a hand gesture in which a user performs a touch, and a first touch start time; the second information comprises a quantity of touch points at which the user performs the touch, a second touch start time, and coordinate information of each touch point; the first touch start time is a start time at which the user performs a touch operation and that is identified by the electromyographic signal collection device; and the second touch start time is a start time at which the user performs the touch operation and that is identified by the location capturing device; generating a touch instruction when it is determined that a time gap between the first touch start time and the second touch start time is less than a preset threshold, and a quantity of touch points that is corresponding to the hand gesture is the same as the quantity of the touch points that is comprised in the second information, wherein the touch instruction comprises the device identifier of the electromyographic signal collection device, the hand gesture, and the coordinate information of each touch point; and performing an interaction operation corresponding to the touch instruction. ### Claim 2 (depends on 1) The method according to claim 1, wherein the method further comprises: continuously receiving the hand gesture in which the user performs the touch and that is sent by the electromyographic signal collection device, and the coordinate information of each touch point that is sent by the location capturing device; and updating the touch instruction. ### Claim 3 (depends on 1) The method according to claim 1, wherein the first information further comprises operating force at which the user performs the touch, and the touch instruction further comprises the operating force. ### Claim 4 (depends on 3) The method according to claim 3, wherein the method further comprises: continuously receiving the hand gesture in which the user performs the touch and the operating force at which the user performs the touch that are sent by the electromyographic signal collection device, and the coordinate information of each touch point that is sent by the location capturing device; and updating the touch instruction. ### Claim 5 (depends on 4) The method according to claim 4, wherein the method further comprises: deleting the touch instruction when it is determined that the operating force is less than a second preset threshold. ### Claim 6 A touch interaction processing method, comprising: periodically collecting, by an electromyographic signal collection device, surface electromyographic signals S1 from multiple channels; determining, by the electromyographic signal collection device according to a time-frequency domain feature of the surface electromyographic signals S1, a hand gesture in which a user performs a touch; and determining a first touch start time according to the hand gesture, wherein the first touch start time is a start time at which the user performs a touch operation and that is identified by the electromyographic signal collection device; and sending, by the electromyographic signal collection device, first information to a processing device, such that the processing device generates a touch instruction according to the first information and second information that is sent by a location capturing device, and performs a corresponding interaction operation, wherein the first information comprises a device identifier of the electromyographic signal collection device, the hand gesture, and the first touch start time; the second information comprises a quantity of touch points at which the user performs the touch, a second touch start time, and coordinate information of each touch point; and the second touch start time is a start time at which the user performs the touch operation and that is identified by the location capturing device. ### Claim 7 (depends on 6) The method according to claim 6, wherein the determining, by the electromyographic signal collection device according to a time-frequency domain feature of the surface electromyographic signals S1, a hand gesture in which a user performs a touch comprises: determining, by the electromyographic signal collection device, the hand gesture according to an amplitude and a frequency that are of the surface electromyographic signals S1 based on a type of a hand gesture. ### Claim 8 (depends on 6) The method according to claim 6, further comprising: determining, by the electromyographic signal collection device according to the time-frequency domain feature of the surface electromyographic signals S1, operating force at which the user performs the touch, wherein the first information further comprises the operating force, and the determining a first touch start time according to the hand gesture is determining the first touch start time according to the hand gesture and the operating force. ### Claim 9 (depends on 8) The method according to claim 8, wherein the determining operating force at which the user performs the touch comprises: obtaining, by the electromyographic signal collection device, a single channel electromyographic signal S2 by performing superposition and averaging on the surface electromyographic signals S1, calculating an average amplitude of the single channel electromyographic signal S2 using a sliding time window, and using the average amplitude of the single channel electromyographic signal S2 as the operating force S. ### Claim 10 (depends on 8) The method according to claim 8, wherein the determining the first touch start time according to the hand gesture and the operating force comprises: determining the first touch start time by looking up a table after acquiring the hand gesture and the operating force, wherein a first preset threshold of operating force corresponding to each hand gesture is pre-stored in the table; and acquiring a current system time when the operating force acquired according to the surface electromyographic signals S1 is greater than the first preset threshold, and using the current system time as the first touch start time. ### Claim 11 A processing device, comprising: a non-transitory memory configured to store instructions; and a processor coupled to the memory and configured to execute the instructions to: receive first information sent by an electromyographic signal collection device and second information sent by a location capturing device, wherein the first information comprises a device identifier of the electromyographic signal collection device, a hand gesture in which a user performs a touch, and a first touch start time; the second information comprises a quantity of touch points at which the user performs the touch, a second touch start time, and coordinate information of each touch point; the first touch start time is a start time at which the user performs a touch operation and that is identified by the electromyographic signal collection device; and the second touch start time is a start time at which the user performs the touch operation and that is identified by the location capturing device; generate a touch instruction when it is determined that a time gap between the first touch start time and the second touch start time is less than a preset threshold, and a quantity of touch points that is corresponding to the hand gesture is the same as the quantity of the touch points that is comprised in the second information, wherein the touch instruction comprises the device identifier of the electromyographic signal collection device, the hand gesture, and the coordinate information of each touch point; and perform an interaction operation corresponding to the touch instruction. ### Claim 12 (depends on 11) The device according to claim 11, wherein the processor is further configured to: continuously receive the hand gesture in which the user performs the touch and that is sent by the electromyographic signal collection device, and the coordinate information of each touch point that is sent by the location capturing device; and update the touch instruction. ### Claim 13 (depends on 11) The device according to claim 11, wherein the first information further comprises operating force at which the user performs the touch, and the touch instruction further comprises the operating force. ### Claim 14 (depends on 13) The device according to claim 13, wherein the processor is further configured to: continuously receive the hand gesture in which the user performs the touch and the operating force at which the user performs the touch that are sent by the electromyographic signal collection device, and the coordinate information of each touch point that is sent by the location capturing device; and update the touch instruction. ### Claim 15 (depends on 14) The device according to claim 14, wherein the processor is further configured to: delete the touch instruction when it is determined that the operating force is less than a second preset threshold. ### Claim 16 A touch interaction processing device, comprising: a non-transitory memory configured to store instructions; and a processor coupled to the memory and configured to execute the instructions to: periodically collect surface electromyographic signals S1 from multiple channels; determine, according to a time-frequency domain feature of the surface electromyographic signals S1, a hand gesture in which a user performs a touch; and determine a first touch start time according to the hand gesture, wherein the first touch start time is a start time at which the user performs a touch operation and that is identified by the electromyographic signal collection device; and send first information to a processing device, such that the processing device generates a touch instruction according to the first information and second information that is sent by a location capturing device, and performs a corresponding interaction operation, wherein the first information comprises a device identifier of the electromyographic signal collection device, the hand gesture, and the first touch start time; the second information comprises a quantity of touch points at which the user performs the touch, a second touch start time, and coordinate information of each touch point; and the second touch start time is a start time at which the user performs the touch operation and that is identified by the location capturing device. ### Claim 17 (depends on 16) The device according to claim 16, wherein the processor is configured to: determining the hand gesture according to an amplitude and a frequency that are of the surface electromyographic signals S1 based on a type of a hand gesture. ### Claim 18 (depends on 16) The device according to claim 16, wherein the processor is further configured to: determine, according to the time-frequency domain feature of the surface electromyographic signals S1, operating force at which the user performs the touch, wherein the first information further comprises the operating force, and the processor is further configured to: determine the first touch start time according to the hand gesture and the operating force. ### Claim 19 (depends on 18) The device according to claim 18, wherein the processor is configured to: obtain a single channel electromyographic signal S2 by performing superposition and averaging on the surface electromyographic signals S1, calculate an average amplitude of the single channel electromyographic signal S2 using a sliding time window, and using the average amplitude of the single channel electromyographic signal S2 as the operating force S. ### Claim 20 (depends on 18) The device according to claim 18, wherein the processor is configured to: determine the first touch start time by looking up a table after acquiring the hand gesture and the operating force, wherein a first preset threshold of operating force corresponding to each hand gesture is pre-stored in the table; and acquire a current system time when the operating force acquired according to the surface electromyographic signals S1 is greater than the first preset threshold, and use the current system time as the first touch start time.