# US 10,311,868 B2 · Method and apparatus for using image data to aid voice recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,868 B2 - Document ID: US10311868B2 - Status: Granted - Kind: B2 - Application number: 15/464,704 - Filed: 2017-03-21 - Published: 2017-07-06 - Granted: 2019-06-04 - Earliest claimed: 2013-05-24 - Assignees: Google Technology Holdings LLC - Applicants: Google Technology Holdings LLC - Inventors: Robert A. Zurek; Adrian M. Schuster; Fu-Lin Shau; Jincheng Wu - Web page: https://patentgrep.com/patent/US10311868B2 ## CPC - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions - G06V 40/166: {using acquisition arrangements} - G06V 40/19: Sensors therefor - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G10L 15/20: Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech (G10L21/02 takes precedence) - G10L 15/25: using position of the lips, movement of the lips or face analysis - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 21/0208: Noise filtering - G06V 40/18: Eye characteristics, e.g. of the iris - G10L 15/24: Speech recognition using non-acoustical features - G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/227: {of the speaker; Human-factor methodology} - G10L 2021/02166: {Microphone arrays; Beamforming} - H04R 2430/20: Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic (H04R2203/12 takes precedence) - H04R 2460/07: Use of position data from wide-area or local-area positioning systems in hearing devices, e.g. program or information selection - H04R 2499/11: Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract A device performs a method for using image data to aid voice recognition. The method includes the device capturing (302) image data of a vicinity of the device and adjusting (304), based on the image data, a set of parameters for voice recognition performed by the device (102). The set of parameters for the device performing voice recognition include, but are not limited to: a trigger threshold of a trigger for voice recognition; a set of beamforming parameters; a database for voice recognition; and/or an algorithm for voice recognition. The algorithm may include using noise suppression or using acoustic beamforming. ## Claims ### Claim 1 A computer-implemented method comprising: receiving, by a computing device, audio data corresponding to a user speaking in a vicinity of the computing device; obtaining, by the computing device, a first image that includes a first representation of the user and that was captured during receipt of a first portion of the audio data; obtaining, by the computing device, a second image that includes a second representation of the user and that was captured during receipt of a second portion of the audio data; determining by the computing device, a first feature of the first representation of the user by analyzing the first image; determining, by the computing device, a second feature of the second representation of the user by analyzing the second image; based on the first feature of the first representation of the user included in the first image, obtaining, by the computing device, a transcription of the first portion of the audio data; based on the second feature of the second representation of the user included in the second image, bypassing, by the computing device, obtaining a transcription of the second portion of the audio data; and providing, for output by the computing device, the transcription of a portion of the audio data. ### Claim 2 (depends on 1) The method of claim 1, wherein the first feature of the first representation of the user included in the first image includes both eyes, a nose, and a mouth of the user. ### Claim 3 (depends on 1) The method of claim 1, wherein the second feature of the second representation of the user included in the second image includes less than both eyes, a nose, and a mouth of the user. ### Claim 4 (depends on 1) The method of claim 1, wherein: the user is an authorized user of the computing device, and providing the transcription of the portion of the audio data based on the user being an authorized user of the computing device. ### Claim 5 (depends on 1) The method of claim 1, wherein the first feature of the first representation of the user included in the first image includes both pupils of the user. ### Claim 6 (depends on 1) The method of claim 1, wherein the computing device obtains the first image and the second image using an infrared camera. ### Claim 7 (depends on 1) The method of claim 1, wherein an accuracy of speech recognition performed on the first portion and the second portion of the audio data varies according to a number of other users who are in each of the first image and the second image. ### Claim 8 A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving, by a computing device, audio data corresponding to a user speaking in a vicinity of the computing device; obtaining, by the computing device, a first image that includes a first representation of the user and that was captured during receipt of a first portion of the audio data; obtaining, by the computing device, a second image that includes a second representation of the user and that was captured during receipt of a second portion of the audio data; determining, by the computing device, a first feature of the first representation of the user by analyzing the first image; determining, by the computing device, a second feature of the second representation of the user by analyzing the second image; based on the first feature of the first representation of the user included in the first image, obtaining, by the computing device, a transcription of the first portion of the audio data; based on the second feature of the second representation of the user included in the second image, bypassing, by the computing device, obtaining a transcription of the second portion of the audio data; and providing, for output by the computing device, the transcription of a portion of the audio data. ### Claim 9 (depends on 8) The system of claim 8, wherein the first feature of the first representation of the user included in the first image includes both eyes, a nose, and a mouth of the user. ### Claim 10 (depends on 8) The system of claim 8, wherein the second feature of the second representation of the user included in the second image includes less than both eyes, a nose, and a mouth of the user. ### Claim 11 (depends on 8) The system of claim 8, wherein: the user is an authorized user of the computing device, and providing the transcription of the portion of the audio data based on the user being an authorized user of the computing device. ### Claim 12 (depends on 8) The system of claim 8, wherein the first feature of the first representation of the user included in the first image includes both pupils of the user. ### Claim 13 (depends on 8) The system of claim 8, wherein the computing device obtains the first image and the second image using an infrared camera. ### Claim 14 (depends on 8) The system of claim 8, wherein an accuracy of speech recognition performed on the first portion and the second portion of the audio data varies according to a number of other users who are in each of the first image and the second image. ### Claim 15 A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising: receiving, by a computing device, audio data corresponding to a user speaking in a vicinity of the computing device; obtaining, by the computing device, a first image that includes a first representation of the user and that was captured during receipt of a first portion of the audio data; obtaining, by the computing device, a second image that includes a second representation of the user and that was captured during receipt of a second portion of the audio data; determining, by the computing device, a first feature of the first representation of the user by analyzing the first image; determining, by the computing device, a second feature of the second representation of the user by analyzing the second image; based on the first feature of the first representation of the user included in the first image, obtaining, by the computing device, a transcription of the first portion of the audio data; based on the second feature of the second representation of the user included in the second image, bypassing, by the computing device, obtaining a transcription of the second portion of the audio data; and providing, for output by the computing device, the transcription of a portion of the audio data. ### Claim 16 (depends on 15) The medium of claim 15, wherein the first feature of the first representation of the user included in the first image includes both eyes, a nose, and a mouth of the user. ### Claim 17 (depends on 15) The medium of claim 15, wherein the second feature of the second representation of the user included in the second image includes less than both eyes, a nose, and a mouth of the user. ### Claim 18 (depends on 15) The medium of claim 15, wherein: the user is an authorized user of the computing device, and providing the transcription of the portion of the audio data based on the user being an authorized user of the computing device. ### Claim 19 (depends on 15) The medium of claim 15, wherein the first feature of the first representation of the user included in the first image includes both pupils of the user. ### Claim 20 (depends on 15) The medium of claim 15, wherein the computing device obtains the first image and the second image using an infrared camera. ### Claim 21 (depends on 1) The method of claim 1, wherein: obtaining a transcription of the first portion of the audio data comprises performing speech recognition on the first portion of the audio data, and bypassing obtaining a transcription of the second portion of the audio data comprises bypassing performing speech recognition on the second portion of the audio data.