# US 12,190,900 B2 · Personalized, real-time audio processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,190,900 B2 - Document ID: US12190900B2 - Status: Granted - Kind: B2 - Application number: 18/206,600 - Filed: 2023-06-06 - Published: 2023-10-05 - Granted: 2025-01-07 - Earliest claimed: 2016-02-02 - Assignees: eBay Inc. - Applicants: eBay Inc. - Inventors: Dane Glasgow; David Ramadge; Matthew Bret MacLaurin; Justin Van Winkle; Corinne Elizabeth Sherman - Web page: https://patentgrep.com/patent/US12190900B2 ## CPC - G10L 21/028: using properties of sound source - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/022: {Viewing apparatus (G02B27/04, G02B27/06, G02B27/08 take precedence)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/16: Sound input; Sound output (speech processing G10L) - G06V 40/19: Sensors therefor - G10L 21/0364: for improving intelligibility - H04R 29/005: {Microphone arrays} - H04R 1/406: {microphones} - H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)} - H04R 2201/403: Linear arrays of transducers - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract An apparatus and method for real-time audio processing employs a gaze detection sensor to detect a direction of a user's gaze and output a gaze signal corresponding to the detected direction of the user's gaze. A digital signal processing unit responds to a plurality of signals corresponding to a plurality of sounds received at the apparatus, and the determined direction of gaze to identify a signal of interest from the plurality of signals using the gaze signal. The signal of interest is processed for output to the user. In embodiments, a microphone array provides the plurality of signals. An imaging sensor may work with either the microphone array or the gaze detection sensor to identify the signal of interest. ## Claims ### Claim 1 A computer-implemented method comprising: detecting, by a device associated with a user, a direction of a gaze of the user; based at least in part on determining that the direction of the gaze of the user is different from directions of a plurality of sound signals: receiving one or more image signals from an environment surrounding the user; and detecting, by the device based on the one or more image signals, a speaker speaking in a direction of the user; and performing, by the device, a noise cancelling algorithm on a first sound signal, of the plurality of sound signals, arriving from a direction different from a direction of the speaker. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, from the plurality of sound signals, a second sound signal that corresponds to speech of the speaker; and emphasizing the second sound signal. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein emphasizing the second sound signal comprises causing output of the second sound signal using one or more acoustical techniques to amplify the second sound signal. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein performing the noise cancelling algorithm on the first sound signal comprises generating and outputting an inverted frequency profile of the first sound signal. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein performing the noise cancelling algorithm on the first sound signal is based at least in part on determining that an audio source associated with the first sound signal is located a threshold number of degrees from a determined centerline between the speaker and the user. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein determining that the direction of the gaze of the user is different from the directions of the plurality of sound signals comprises: detecting, using one or more sensors, movement of: a position of eyes of the user, or one or more microphones placed on a body of the user; and comparing directions associated with the detected movement to the directions of the plurality of sound signals. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, based on comparing the direction of the gaze of the user to the directions of the plurality of sound signals, which of a first sensor or a second sensor to prioritize for use in determining which sound signal, of the plurality of sound signals, to emphasize. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, further comprising: receiving one or more gaze signals from a gaze detection sensor, wherein the one or more gaze signals are, by default, used to determine which sound signal, of the plurality of sound signals, to emphasize; receiving the one or more image signals from an image sensor; and based at least in part on determining that the direction of the gaze of the user is different from the directions of the plurality of sound signals, ignoring the one or more gaze signals and using the one or more image signals to determine which of the plurality of sound signals to emphasize. ### Claim 9 A device associated with a user comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the device to: detect a direction of a gaze of the user; based at least in part on determining that the direction of the gaze of the user is different from directions of a plurality of sound signals: receive one or more image signals from an environment surrounding the user; and detect, based on the one or more image signals, a speaker speaking in a direction of the user; and perform a noise cancelling algorithm on a first sound signal, of the plurality of sound signals, arriving from a direction different from a direction of the speaker. ### Claim 10 (depends on 9) The device of claim 9, wherein the instructions, when executed by the at least one processor, further cause the device to: determine, from the plurality of sound signals, a second sound signal that corresponds to speech of the speaker; and emphasize the second sound signal. ### Claim 11 (depends on 9) The device of claim 9, wherein, to perform the noise cancelling algorithm on the first sound signal, the instructions, when executed by the at least one processor, cause the device to generate and output an inverted frequency profile of the first sound signal. ### Claim 12 (depends on 9) The device of claim 9, wherein the instructions, when executed by the at least one processor, cause the device to perform the noise cancelling algorithm on the first sound signal based at least in part on a determination that an audio source associated with the first sound signal is located a threshold number of degrees from a determined centerline between the speaker and the user. ### Claim 13 (depends on 9) The device of claim 9, wherein, to determine that the direction of the gaze of the user is different from the directions of the plurality of sound signals, the instructions, when executed by the at least one processor, cause the device to: detect, using one or more sensors, movement of: a position of eyes of the user, or one or more microphones placed on a body of the user; and compare directions associated with the detected movement to the directions of the plurality of sound signals. ### Claim 14 (depends on 9) The device of claim 9, wherein the instructions, when executed by the at least one processor, further cause the device to: determine, based on comparing the direction of the gaze of the user to the directions of the plurality of sound signals, which of a first sensor or a second sensor to prioritize for use in determining which of the plurality of sound signals to emphasize. ### Claim 15 (depends on 9) The device of claim 9, wherein the instructions, when executed by the at least one processor, further cause the device to: receive one or more gaze signals from a gaze detection sensor, wherein the one or more gaze signals are, by default, used to determine which of the plurality of sound signals to emphasize; receive the one or more image signals from an image sensor; and based at least in part on determining that the direction of the gaze of the user is different from the directions of the plurality of sound signals, ignore the one or more gaze signals and using the one or more image signals to determine which of the plurality of sound signals to emphasize. ### Claim 16 A non-transitory, computer-readable medium storing instructions that, when executed by at least one processor of a device, cause the device to perform operations comprising: detecting a direction of a gaze of a user associated with the device; based at least in part on determining that the direction of the gaze of the user is different from directions of a plurality of sound signals: receiving one or more image signals from an environment surrounding the user; and detecting, based on the one or more image signals, a speaker speaking in a direction of the user; and performing a noise cancelling algorithm on a first sound signal, of the plurality of sound signals, arriving from a direction different from a direction of the speaker. ### Claim 17 (depends on 16) The non-transitory, computer-readable medium of claim 16, wherein the instructions, when executed by the at least one processor, further cause the device to perform operations comprising: determining, from the plurality of sound signals, a second sound signal that corresponds to speech of the speaker; and emphasizing the second sound signal. ### Claim 18 (depends on 16) The non-transitory, computer-readable medium of claim 16, wherein performing the noise cancelling algorithm on the first sound signal comprises generating and outputting an inverted frequency profile of the first sound signal. ### Claim 19 (depends on 16) The non-transitory, computer-readable medium of claim 16, wherein performing the noise cancelling algorithm on the first sound signal is based at least in part on determining that an audio source associated with the first sound signal is located a threshold number of degrees from a determined centerline between the speaker and the user. ### Claim 20 (depends on 16) The non-transitory, computer-readable medium of claim 16, wherein the instructions, when executed by the at least one processor, further cause the device to perform operations comprising: receiving one or more gaze signals from a gaze detection sensor, wherein the one or more gaze signals are, by default, used to determine which of the plurality of sound signals to emphasize; receiving the one or more image signals from an image sensor; and based at least in part on determining that the direction of the gaze of the user is different from the directions of the plurality of sound signals, ignoring the one or more gaze signals and using the one or more image signals to determine which of the plurality of sound signals to emphasize.