# US 11,122,373 B2 · Hearing device configured to utilize non-audio information to process audio signals > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,122,373 B2 - Document ID: US11122373B2 - Status: Granted - Kind: B2 - Application number: 16/557,120 - Filed: 2019-08-30 - Published: 2020-03-05 - Granted: 2021-09-14 - Earliest claimed: 2018-09-02 - Assignees: Oticon A/S - Applicants: Oticon A/S - Inventors: Jesper Bruun Jensen; Sigurdur Sigurdsson - Web page: https://patentgrep.com/patent/US11122373B2 ## CPC - H04R 25/505: {using digital signal processing} - H04R 25/55: {using an external connection, either wireless or wired} - G06V 40/168: {Feature extraction; Face representation} - G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) - H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils - H04R 25/558: {Remote control, e.g. of amplification, frequency} - H04R 25/604: {of acoustic or vibrational transducers} - G10L 21/0208: Noise filtering - G10L 25/48: specially adapted for particular use - H04R 25/407: {Circuits for combining signals of a plurality of transducers} - H04R 2225/43: Signal processing in hearing aids to enhance the speech intelligibility - H04R 2225/55: Communication between hearing aids and external devices via a network for data exchange - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A hearing device, e.g. a hearing aid, is configured to be worn by a user, e.g. fully or partially on the head of the user, comprises a) an input transducer for converting a sound comprising a target sound from a target talker and possible additional sound in an environment of the user, when the user wears the hearing device, to an electric sound signal representative of said sound, b) an auxiliary input unit configured to provide an auxiliary electric signal representative of said target signal or properties thereof, c) a processor connected to said input transducer and to said auxiliary input unit, and wherein said processor is configured to apply a processing algorithm to said electric sound signal, or a signal derived therefrom, to provide an enhanced signal by attenuating components of said additional sound relative to components of said target sound in said electric sound signal, or said signal derived therefrom. The auxiliary electric signal is derived from visual information, e.g. from a camera, containing information of current vibrations of a facial or throat region of said target talker, and the processing algorithm is configured to use the auxiliary electric signal or the signal derived therefrom to provide the enhanced signal. ## Claims ### Claim 1 A hearing device configured to be worn by a user or to be fully or partially implanted in the head of the user, the hearing device comprising: at least one input transducer for converting a sound comprising a target sound from a target talker and possible additional sound in an environment of the user, when the user wears the hearing device, to an electric sound signal representative of said sound, an auxiliary input unit configured to provide an auxiliary electric signal representative of said target signal or properties thereof, a processor connected to said input transducer and to said auxiliary input unit, said processor being configured to apply a processing algorithm to said electric sound signal, or a signal derived therefrom, to provide an enhanced signal by attenuating components of said additional sound relative to components of said target sound in said electric sound signal, or in said signal derived therefrom, wherein said auxiliary electric signal is derived from visual information containing information of current vibrations of a facial or throat region of said target talker, and wherein said processing algorithm is configured to use said auxiliary electric signal or said signal derived therefrom to provide said enhanced signal, said hearing device further comprises a filter bank for decomposing the electric sound signal in frequency sub-bands, at least providing a low-frequency part and a high-frequency part of the electric sound signal, and an estimate of a high-frequency part ŜHF of the enhanced signal is approximated by the unprocessed noisy high-frequency part xHF of the electric sound signal. ### Claim 2 (depends on 1) A hearing device according to claim 1 comprising a light sensitive sensor for providing said visual information. ### Claim 3 (depends on 1) A hearing device according to claim 1 comprising a camera for providing said visual information. ### Claim 4 (depends on 3) A hearing device according to claim 3 comprising a carrier whereon said camera is mounted. ### Claim 5 (depends on 4) A hearing device according to claim 4 wherein said carrier comprises a housing of the hearing device, a spectacle frame, or a boom of a headset, an article of clothing, or a clip. ### Claim 6 (depends on 1) A hearing device according to claim 1 comprising a user interface allowing a user to indicate a direction to or a location of a target talker of current interest to the user. ### Claim 7 (depends on 1) A hearing device according to claim 1 comprising an adaptive filter and a combination unit for estimating a low-frequency part ŜLF of the enhanced signal from said low frequency part xLF of the electric sound signal and said auxiliary electric signal es. ### Claim 8 (depends on 7) A hearing device according to claim 7 comprising a synthesis filter bank or a sum unit for providing said enhanced signal ŝ from said low-frequency part ŝLF and a high-frequency part ŝHF of the enhanced signal. ### Claim 9 (depends on 1) A hearing device according to claim 1 comprising a voice activity detector for providing a voice activity indicator representing an estimate of whether or not, or with what probability, an input signal comprises a voice signal at a given point in time, and wherein said voice activity indicator is determined in dependence of said auxiliary electric signal or said signal derived therefrom. ### Claim 10 (depends on 1) A hearing device according to claim 1 comprising a face tracking algorithm to extract features of the face region of a person in a field of view of the camera. ### Claim 11 (depends on 1) A hearing device according to claim 1 comprising an output unit for providing stimuli perceivable as sound to a user based on said enhanced signal ŝ. ### Claim 12 (depends on 1) A hearing device according to claim 1 configured to provide that the use of the auxiliary signal in the estimation of an enhanced (target) signal is only enabled, when vibrations in the facial or throat region are above a certain threshold taken to be due to the person having activated the vocal cords, and hence is talking. ### Claim 13 (depends on 1) A hearing device according to claim 1 being constituted by or comprising a hearing aid, a headset, an earphone, an ear protection device or a combination thereof. ### Claim 14 A method of operating a hearing device configured to be worn by a user or to be fully or partially implanted in the head of the user, the method comprising: converting a sound comprising a target sound from a target talker and possible additional sound in an environment of the user, when the user wears the hearing device, to an electric sound signal representative of said sound, providing an auxiliary electric signal representative of said target signal or properties thereof, applying a processing algorithm to said electric sound signal, or a signal derived therefrom, to provide an enhanced signal by attenuating components of said additional sound relative to components of said target sound in said electric sound signal, or in said signal derived therefrom, deriving said auxiliary electric signal from visual information containing information of current vibrations of a facial or throat region of said target talker, and using said auxiliary electric signal or said signal derived therefrom to provide said enhanced signal, wherein said method further comprises using a filter bank of the hearing device to decompose the electric sound signal in frequency sub-bands, at least providing a low-frequency part and a high-frequency part of the electric sound signal, and an estimate of a high-frequency part ŝHF of the enhanced signal is approximated by the unprocessed noisy high-frequency part xHF of the electric sound signal. ### Claim 15 (depends on 1) A hearing aid system, comprising a hearing device according to claim 1 and an auxiliary device, the hearing system being adapted to establish a communication link between the hearing device and the auxiliary device to provide that information can be exchanged or forwarded from one to the other. ### Claim 16 (depends on 15) A hearing aid system according to claim 15 wherein the auxiliary device is or comprises the light sensitive sensor. ### Claim 17 (depends on 15) A hearing aid system according to claim 15 wherein the auxiliary device is or comprises a remote control, a smartphone, or other portable or wearable electronic device. ### Claim 18 (depends on 15) A hearing aid system according to claim 15 wherein the auxiliary device is or comprises another hearing device and wherein the hearing system implements a binaural hearing system.