# US 12,112,756 B2 · Voice interaction wakeup electronic device, method and medium based on mouth-covering action recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,756 B2 - Document ID: US12112756B2 - Status: Granted - Kind: B2 - Application number: 17/616,075 - Filed: 2020-05-26 - Published: 2022-10-06 - Granted: 2024-10-08 - Earliest claimed: 2019-06-03 - Assignees: Tsinghua University - Applicants: Tsinghua University - Inventors: Chun Yu; Yuanchun Shi - Web page: https://patentgrep.com/patent/US12112756B2 ## CPC - G10L 15/25: using position of the lips, movement of the lips or face analysis - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06V 40/161: {Detection; Localisation; Normalisation} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 25/18: the extracted parameters being spectral information of each sub-band - H04R 1/1041: {Mechanical or electronic switches, or control elements (switches in general H01H)} - H04R 1/406: {microphones} - H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)} - G10L 2015/223: {Execution procedure of a spoken command} - H04R 2420/07: Applications of wireless loudspeakers or wireless microphones - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract An interaction method triggered by a mouth-covering gesture and an intelligent electronic device are provided. The interaction method is applied to an intelligent electronic device arranged with a sensor. The intelligent electronic device includes a sensor system for capturing a signal of a user putting one hand on a mouth to make a mouth-covering gesture. The interaction method includes: processing the signal to determine whether the user puts the hand on the mouth to make the mouth-covering gesture; and in a case that the user puts the hand on the mouth to make the mouth-covering gesture, determining a mouth-covering gesture input mode as an input mode for controlling an interaction to trigger a control command or trigger another input mode, by executing a program on the intelligent electronic device. ## Claims ### Claim 1 An intelligent electronic device, comprising: a sensor system, configured to capture a signal for determining that a user puts a hand on a mouth of the user to make a mouth-covering gesture; a processor; and a memory storing computer executable instructions, wherein the computer executable instructions, when executed by the processor, cause the processor, to perform a voice interaction wakeup method, and the voice interaction wakeup method comprises: processing the signal to determine whether the user puts the hand on the mouth to make the mouth-covering gesture; and in response to determining that the user puts the hand on the mouth to make the mouth-covering gesture, triggering to enter a voice input interaction, to process a voice input from the user keeping the mouth-covering gesture and respond to the voice input, by executing a program on the intelligent electronic device. ### Claim 2 (depends on 1) The intelligent electronic device according to claim 1, wherein the mouth-covering gesture is one of a mouth-covering gesture with a left hand and a mouth-covering gesture with a right hand. ### Claim 3 (depends on 1) The intelligent electronic device according to claim 1, wherein the mouth-covering gesture, according to a position of a palm relative to the mouth, is one of a mouth-covering gesture with the palm locating at a position between the mouth and a left ear, a mouth-covering gesture with the palm locating at a position between the mouth and a right ear, and a mouth-covering gesture with the palm locating at a position in front of the mouth. ### Claim 4 (depends on 1) The intelligent electronic device according to claim 1, wherein the mouth-covering gesture is one of a mouth-covering gesture with touching a face and a mouth-covering gesture without touching a face. ### Claim 5 (depends on 1) The intelligent electronic device according to claim 1, wherein the mouth-covering gesture comprises one of: a mouth-covering gesture with the palm covering the whole mouth; a mouth-covering gesture with a thumb being in contact with a side of the mouth, an index finger being on top of lips and the mouth exposed below the palm; a mouth-covering gesture with the thumb being in contact with a lower jaw, the index finger being on the top of the lips and the mouth exposed below the palm; and a mouth-covering gesture with the thumb being in contact with the side of the mouth, a little finger touching the lower jaw and the mouth exposed above the palm. ### Claim 6 (depends on 1) The intelligent electronic device according to claim 1, wherein the intelligent electronic device is configured, in a case that the intelligent electronic device determines that the mouth-covering gesture is a predetermined mouth-covering gesture, to execute a control command. ### Claim 7 (depends on 6) The intelligent electronic device according to claim 6, wherein the control command is executed to trigger the another input mode except the mouth-covering gesture input mode to process information inputted in the another input mode. ### Claim 8 (depends on 7) The intelligent electronic device according to claim 7, wherein the another input mode comprises one or a combination of a voice input mode, a non-mouth-covering gesture input mode, a sight input mode, a blink input mode, and a head movement input mode. ### Claim 9 (depends on 1) The intelligent electronic device according to claim 1, wherein the intelligent electronic device is configured to: process the signal to determine whether the user stops making the mouth-covering gesture; and end the voice input interaction in a case that the user stops making the mouth-covering gesture. ### Claim 10 (depends on 1) The intelligent electronic device according to claim 1, wherein the intelligent electronic device is configured to provide a visual feedback or an auditory feedback to prompt the user that the intelligent electronic device triggers to enter the voice input interaction. ### Claim 11 (depends on 1) The intelligent electronic device according to claim 1, wherein the intelligent electronic device is configured, in a case that the signal comprises a face image captured by a camera, to recognize one or more types of the mouth-covering gesture made by the user. ### Claim 12 (depends on 1) The intelligent electronic device according to claim 1, wherein features of the signal captured by the sensor system comprise one or more of a time domain feature, a frequency spectrum feature and a sound source position feature of a sound signal received by a single microphone. ### Claim 13 (depends on 1) The intelligent electronic device according to claim 1, wherein features of the signal captured by the sensor system comprise differences of features of sound signals received by a plurality of microphones. ### Claim 14 (depends on 13) The intelligent electronic device according to claim 13, wherein the intelligent electronic device is configured, in a case that a sensing device is a wireless headset, to recognize the mouth-covering gesture based on a difference between a signal of a left headphone and a signal of a right headphone. ### Claim 15 A voice interaction wakeup method performed by an intelligent electronic device, wherein the intelligent electronic device comprises a sensor system for capturing a signal of a user putting one hand on a mouth to make a mouth-covering gesture, and the voice interaction wakeup method performed by the intelligent electronic device comprises: processing the signal to determine that the user puts one hand on the mouth to make the mouth-covering gesture; in response to determining that the user keeps the mouth-covering gesture with a hand on the mouth, triggering to enter a voice input interaction, analyzing an interaction intention of the user based on the mouth-covering gesture, an interaction content of a current application on the intelligent device and information inputted in the voice input interaction by the user; based on the analyzed interaction intention, receiving and analyzing, by the intelligent device, information inputted by the user, and outputting, by the intelligent device, a corresponding content; in a case that the user performs interaction with the intelligent device based on the mouth-covering gesture, processing the signal to determine that the user stops making the mouth-covering gesture; and ending the voice input interaction in a case that the user stops making the mouth-covering gesture. ### Claim 16 (depends on 15) The interaction method according to claim 15, wherein the information inputted by the user comprises information of the mouth-covering gesture and other modal information of the user. ### Claim 17 (depends on 16) The interaction method according to claim 16, wherein the other modal information comprises voice information or eye contact information. ### Claim 18 (depends on 15) A non-transitory computer readable medium storing computer executable instructions, wherein the computer executable instructions, when executed by a computer, cause the computer to perform the voice interaction wakeup method according to claim 15.