# US 2020/0286484 A1 · Methods and systems for speech detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0286484 A1 - Document ID: US20200286484A1 - Status: Published application - Kind: A1 - Application number: 16/852,376 - Filed: 2020-04-17 - Published: 2020-09-10 - Earliest claimed: 2017-10-18 - Assignees: SoapBox Labs Ltd. - Applicants: SoapBox Labs Ltd. - Inventors: Patricia Scanlon - Web page: https://patentgrep.com/patent/US20200286484A1 ## CPC - G10L 15/24: Speech recognition using non-acoustical features - G10L 17/10: Multimodal systems, i.e. based on the integration of multiple recognition engines or fusion of expert systems - G06F 3/012: {Head tracking input arrangements} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 21/32: using biometric data, e.g. fingerprints, iris scans or voiceprints - G06F 21/6245: {Protecting personal data, e.g. for financial or medical purposes} - G06V 40/16: Human faces, e.g. facial parts, sketches or expressions - G06V 40/161: {Detection; Localisation; Normalisation} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Methods and systems for processing user input to a computing system are disclosed. The computing system has access to an audio input and a visual input such as a camera. Face detection is performed on an image from the visual input, and if a face is detected this triggers the recording of audio and making the audio available to a speech processing function. Further verification steps can be combined with the face detection step for a multi-factor verification of user intent to interact with the system. ## Claims ### Claim 1 A method of processing user input to a computing system having an audio input and a visual input, the method comprising: receiving, at the computing system, the audio input; determining whether the user has demonstrated an intent to interact with the computing system via the audio input, wherein determining whether the user has demonstrated the intent to interact with the computing system via the audio input comprises: performing a face detection method on an image received from the visual input after the audio input has been received; and determining whether said face detection method has detected a face within a predetermined time of the audio input having been received; and responsive to determining that the user has demonstrated the intent to interact with the computing system: recording an audio signal from said audio input; and making said audio signal available to a speech processing function. ### Claim 2 (depends on 1) The method of claim 1, further comprising performing one or more additional verification operations, wherein recording the audio signal and making said audio signal available are dependent on the outcome of said one or more further verification operations in addition to said face detection method detecting a face. ### Claim 3 (depends on 2) The method of claim 2, wherein said one or more additional verification operations comprise a gaze direction detection operation to verify that the user is looking in a predefined direction or range of directions. ### Claim 4 (depends on 2) The method of claim 2, wherein said one or more additional verification operations comprise a mouth movement detection operation to verify that the user's mouth is moving. ### Claim 5 (depends on 4) The method of claim 4, wherein said mouth movement detection operation further verifies that the mouth movement of the user corresponds to a movement pattern typical of speech. ### Claim 6 (depends on 2) The method of claim 2, wherein said one or more additional verification operations comprise an audio detection operation to verify that the audio input is receiving sound from the environment of the user. ### Claim 7 (depends on 6) The method of claim 6, wherein said audio detection operation further verifies that the characteristics of detected sound are consistent with speech. ### Claim 8 (depends on 6) The method of claim 6, wherein said audio detection operation further verifies that the direction from which sound is detected is consistent with the direction of the detected face. ### Claim 9 (depends on 6) The method of claim 6, wherein said audio detection operation further verifies that the characteristics of detected sound are consistent with a speech profile stored for a given user. ### Claim 10 (depends on 2) The method of claim 2, wherein at least two of said additional verification operations are performed. ### Claim 11 (depends on 2) The method of claim 2, wherein the determination of the face detection and/or the additional verification operations is a weighted determination indicative of the reliability of the determination in question, and wherein a positive determination is made when the weighted determination is above a threshold. ### Claim 12 (depends on 1) The method of claim 1, wherein said face detection operation verifies that the detected face is oriented in a predetermined direction or range of directions. ### Claim 13 (depends on 11) The method of claim 11, wherein the face detection operation verifies that the visual input is at or below the level of the user's eyes or nose. ### Claim 14 (depends on 1) The method of claim 1, further comprising performing said speech processing function on the recorded signal. ### Claim 15 (depends on 1) The method of claim 1, further comprising sending said recorded audio signal to a remote computing device for speech processing. ### Claim 16 (depends on 1) The method of claim 1, further comprising buffering said audio signal, and (i) responsive to determining that the user has demonstrated the intent to interact with the computing system, overwriting or discarding the buffered signal; and (ii) responsive to determining that the user has demonstrated the intent to interact with the computing system, retrieving said signal from the buffer. ### Claim 17 (depends on 15) The method of claim 15, wherein said buffer is of sufficient capacity to store an audio signal of a duration at least as long as the time required to determine the face detection and optionally the additional verification operations. ### Claim 18 A computing system for processing user input having an audio input and a visual input, the system comprising: a memory; and a processor, coupled to the memory, to: receive, at the computing system, the audio input; determine whether the user has demonstrated an intent to interact with the computing system via the audio input, wherein to determine whether the user has demonstrated the intent to interact with the computing system via the audio input, the processor is further to: perform a face detection method on an image received from the visual input after the audio input has been received; and determine whether said face detection method has detected a face within a predetermined time of the audio input having been received; and responsive to determining that the user has demonstrated the intent to interact with the computing system: record an audio signal from said audio input; and make said audio signal available to a speech processing function. ### Claim 19 A non-transitory computer readable medium comprising instructions, which when executed by a processor, cause the processor to perform a method of processing user input to a computing system having an audio input and a visual input, the method comprising: receiving, at the computing system, the audio input; determining whether the user has demonstrated an intent to interact with the computing system via the audio input, wherein determining whether the user has demonstrated the intent to interact with the computing system via the audio input comprises: performing a face detection method on an image received from the visual input after the audio input has been received; and determining whether said face detection method has detected a face within a predetermined time of the audio input having been received; and responsive to determining that the user has demonstrated the intent to interact with the computing system: recording an audio signal from said audio input; and making said audio signal available to a speech processing function. ### Claim 20 (depends on 19) The non-transitory computer readable medium of claim 19, wherein the method further comprises performing one or more additional verification operations, wherein recording the audio signal and making said audio signal available are dependent on the outcome of said one or more further verification operations in addition to said face detection method detecting a face.