# US 12,604,148 B2 · Audio processing using ear-wearable device and wearable vision device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,604,148 B2 - Document ID: US12604148B2 - Status: Granted - Kind: B2 - Application number: 18/108,137 - Filed: 2023-02-10 - Published: 2023-09-14 - Granted: 2026-04-14 - Earliest claimed: 2022-02-11 - Assignees: Starkey Laboratories, Inc. - Applicants: Starkey Laboratories, Inc. - Inventors: William F. Austin; Achintya Kumar Bhowmik; David Alan Fabry; Amit Shahar; Majd Srour - Web page: https://patentgrep.com/patent/US12604148B2 ## CPC - H04R 25/554: {using a wireless connection, e.g. between microphone and amplifier or using Tcoils} - G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units} - H04R 1/08: Mouthpieces; {Microphones;} Attachments therefor - H04R 1/1041: {Mechanical or electronic switches, or control elements (switches in general H01H)} - H04R 25/407: {Circuits for combining signals of a plurality of transducers} - H04R 25/43: {Electronic input selection or mixing based on input signal analysis, e.g. mixing or selection between microphone and telecoil or between microphones with different directivity characteristics (H04R25/407 takes precedence)} - G02B 2027/0141: {characterised by the informative content of the display} - G09G 2354/00: Aspects of interface with display user - G09G 2380/08: Biomedical applications - H04R 1/1016: {Earpieces of the intra-aural type} - H04R 1/406: {microphones} - H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)} - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract A hearing assisting system is included having a first microphone and a second microphone. The system further includes a vision device having a display device, where the display device is configured to display visual information to the user when the user is wearing the vision device. The system is further configured to process audio signals to identify first audio content information and first audio direction information related to the first audio content information. The system further can transmit, to the vision device, first display information to cause the vision device to display a non-transcript content representation of the first audio content information and a direction representation of the first audio direction information using the display device. ## Claims ### Claim 1 A hearing assisting system comprising: a. a first ear-wearable device, the first ear-wearable device comprising a first microphone at a first microphone location, the first ear-wearable device further comprising: a hearing device housing, the hearing device housing comprising: a battery compartment comprising a battery; a flexible mother circuit comprising and electrically connected to: a digital signal processor; an audio output device electrically connected to the digital signal processor; a plurality of user switches electrically connected to the digital signal processor, wherein the user switches control volume and on/off; an inertial measurement unit (IMU) sensor; and a first memory storage device configured to store data from the IMU sensor; a first transducer electrically connected to the audio output device and configured to be positioned within an ear of a user; and a cable comprising one or more electrical conductors, the cable configured to provide electrical communication between the hearing device housing and the first transducer; wherein the first ear-wearable device is configured to: be worn by the user in contact with the ear of the user; direct sound from the first transducer toward the user's ear when the first ear-wearable device is worn by the user; and wirelessly transmit a first input audio signal received at the first microphone at the first microphone location; b. a vision device, the vision device comprising or defining a lens portion, wherein the vision device is configured to be worn by the user so that the user looks through the lens portion, the vision device further comprising a display device, wherein the display device is configured to display visual information to the user when the user is wearing the vision device, c. a second microphone, wherein the second microphone is configured to be positioned at a second microphone location; wherein the hearing assisting system is configured to store computer instructions for: receiving, from the first microphone, the first input audio signal; receiving, from the second microphone, a second input audio signal, wherein the first audio input signal and the second audio input signal both comprise first audio information from a first time period; processing the first input audio signal and the second input audio signal to identify first audio content information and first audio direction information related to the first audio content information; and wirelessly transmitting, to the vision device, first display information to cause the vision device to display a non-transcript content representation of the first audio content information and a direction representation of the first audio direction information using the display device; wherein the display information comprises at least one menu prompt to control the first ear-wearable device, and wherein the hearing assisting system is configured to receive user input to execute the at least one menu prompt displayed, and wherein the at least one menu prompt is executed by the user tapping on the first ear-wearable device. ### Claim 2 (depends on 1) The hearing assisting system of claim 1, wherein the first ear-wearable device further comprises: a first control circuit, the first transducer, wherein the first transducer is configured to generate sound and is in electrical communication with the first control circuit, the first microphone, wherein the first microphone is in electrical communication with the first control circuit, the first memory storage, and a first wireless communication device; wherein the vision device further comprises: a vision device control circuit, a vision device wireless communication device, a vision device camera, and a vision device memory, wherein the vision device wireless communication device is configured to receive the first display information. ### Claim 3 (depends on 1) The hearing assisting system of claim 1, further comprising a second ear-wearable device, the second ear-wearable device comprising: a. a second control circuit; b. a second electroacoustic transducer, wherein the second transducer is configured to generate sound and is in electrical communication with the second control circuit; c. the second microphone, wherein the second microphone is in electrical communication with the second control circuit; d. a second memory storage; and e. a second wireless communication device; and wherein the second ear-wearable device is configured to: be worn by the user in contact with a second ear of the user; direct sound from the first transducer toward the user's ear when the second ear-wearable device is worn by the user; and wirelessly transmit the second input audio signal received at the second microphone. ### Claim 4 (depends on 1) The hearing assisting system of claim 1, further comprising a gateway device, the gateway device comprising: a. a gateway device control circuit; b. a gateway device memory; and c. a gateway device wireless communication device, wherein the gateway device memory stores instructions for: wirelessly receiving, at the gateway device wireless communication device, the first input audio signal and the second input audio signal; processing the first input audio signal and the second input audio signal to identify the first audio content information and the first audio direction information; and wirelessly transmitting, to the vision device, the first display information to cause the vision device to display the non-transcript content representation of the first audio content information and the direction representation of the first audio direction information using the display device. ### Claim 5 (depends on 4) The hearing assisting system of claim 4, wherein the vision device further comprises a first vision device microphone positioned on the vision device and a second vision device microphone, wherein the system analyzes audio input from the first microphone, the second microphone, the first vision device microphone, and the second vision device microphone to process the first input audio signal and the second input audio signal to identify the first audio content information and the first audio direction information, and wherein first vision device microphone is spaced apart from the second vision device microphone by at least 2 centimeters. ### Claim 6 (depends on 4) The hearing assisting system of claim 4, wherein the gateway device memory stores instructions for: a. identifying a presence of a first emergency notification sound as the first audio content information within the first audio content information; and b. upon identifying the presence of the first emergency notification sound, the non-transcript content representation will identify the first emergency notification sound; wherein the first emergency notification sound is one of the group consisting of a siren, an emergency vehicle siren, a tornado siren, an emergency weather siren, a public safety siren, a horn, a train whistle, a train horn, a vehicle horn, a smoke alarm, a fire alarm, a carbon-monoxide alarm, and a law enforcement announcement. ### Claim 7 (depends on 6) The hearing assisting system of claim 6, configured to, upon displaying content representation, apply a tinting to the lens portion for an alert time period. ### Claim 8 (depends on 4) The hearing assisting system of claim 4, wherein the gateway device memory stores instructions for identifying a name call as the first audio content information within the first audio content information, wherein the instructions comprise a user name and user name variations. ### Claim 9 (depends on 4) The hearing assisting system of claim 4, wherein the first ear-wearable device further comprises a telecoil, wherein the hearing assisting system is configured to: a. detect a first electromagnetic signal at the telecoil, b. analyze the first electromagnetic signal to determine telecoil information related to a presence of an appliance that generated the first electromagnetic signal in an environment of the first ear-wearable device, and c. transmit, to the vision device, telecoil display information to cause the vision device to display a representation of telecoil information using the display device. ### Claim 10 (depends on 1) The hearing assisting system of claim 1, wherein the direction representation comprises text describing a direction of a sound within the first audio information. ### Claim 11 (depends on 1) The hearing assisting system of claim 1, wherein the direction representation comprises an arrow pointing toward a direction of a sound within the first audio information. ### Claim 12 (depends on 1) The hearing assisting system of claim 1, wherein the direction representation comprises tinting of a lens portion in a direction opposite from a direction of a sound within the first audio information. ### Claim 13 (depends on 1) The hearing assisting system of claim 1 wherein the vision device comprises a contact lens configured to be worn on an eyeball of a user. ### Claim 14 (depends on 13) The hearing assistance system of claim 13, wherein the contact lens comprises: a. an electronic component layer including a display device that is configured to provide visual information to the user; and b. outer encasing layers. ### Claim 15 (depends on 14) The hearing assistance system of claim 14, wherein the contact lens further comprise a contact lens microphone. ### Claim 16 A method comprising: receiving a first audio signal at a first microphone of an ear-wearable device of a hearing assisting system; receiving a second audio signal at a second microphone of the hearing assisting system, wherein the first audio signal and second audio signal comprise first audio information from a first time period; processing, by a processing system that includes one or more processors, the first input audio signal and the second input audio signal to identify first audio content information and first audio direction information related to the first audio content information; and wirelessly transmitting, to a wearable vision device, first display information to cause the vision device to display a non-transcript content representation of the first audio content information and a direction representation of the first audio direction information using a display device of the vision device, wherein the vision device further comprises or defines a lens portion, wherein the vision device is configured to be worn by a user so that the user looks through the lens portion, wherein the display device is configured to display visual information to the user when the user is wearing the vision device; wherein the ear-wearable device comprises a first transducer configured to be positioned within an ear of a user and a hearing device housing, the hearing device housing comprising: a flexible mother circuit comprising and electrically connected to: a digital signal processor; an audio output device electrically connected to the digital signal processor, a plurality of user switches electrically connected to the digital signal processor, wherein the user switches control volume and on/off; an inertial measurement unit (IMU) sensor; and a first memory storage device configured to store data from the IMU sensor; wherein the first display information comprises at least one menu prompt to control the ear-wearable device, and wherein the hearing assisting system is configured to receive user input to execute the at least one menu prompt displayed, and wherein the at least one menu prompt is executed by the user tapping on the ear-wearable device. ### Claim 17 (depends on 16) The method of claim 16, further comprising: a. identifying, by a processing system that includes one or more processors, a presence of a first emergency notification sound as the first audio content information within the first audio content information; and b. upon identifying the presence of the first emergency notification sound, the non-transcript content representation will identify the first emergency notification sound; wherein the first emergency notification sound is one of the group consisting of a siren, an emergency vehicle siren, a tornado siren, an emergency weather siren, a public safety siren, a horn, a train whistle, a train horn, a vehicle horn, a smoke alarm, a fire alarm, a carbon-monoxide alarm, and a law enforcement announcement. ### Claim 18 (depends on 16) The method of claim 16, further comprising identifying, by the processing system, a name call as the first audio content information within the first audio content information, wherein a memory system comprising one or more memory devices stores instructions comprising a user name and user name variations. ### Claim 19 (depends on 16) The method of claim 16, wherein the first ear-wearable device further comprises a telecoil, further comprising: a. detecting a first electromagnetic signal at the telecoil, b. analyzing, using the processing system, the first electromagnetic signal to determine telecoil information related to a presence of an appliance that generated the first electromagnetic signal in an environment of the ear-wearable device, and c. transmitting, to the vision device, telecoil display information to cause the vision device to display a representation of telecoil information using the display device. ### Claim 20 A method comprising: obtaining, by a processing system that includes one or more processors implemented in circuitry, a signal from a telecoil of an ear-wearable device, wherein the ear-wearable device is configured to be worn by a user in contact with an ear of the user and direct sound from a first electroacoustic transducer toward the user's ear when the first ear-wearable device is worn by the user; determining, by the processing system, telecoil information related to a presence of an electromagnetic signal in an environment of the ear-wearable device, based at least in part on the signal from the telecoil; wirelessly transmitting, to a wearable vision device, first display information to cause the vision device to display a representation of the telecoil information using a display device of the vision device, wherein the vision device further comprises or defines a lens portion, wherein the vision device is configured to be worn by the user so that the user looks through the lens portion, wherein the display device is configured to display visual information to the user when the user is wearing the vision device; wirelessly receiving, by the telecoil, audio information from a magnetic field produced by a listening system; and generating sound, by the first electroacoustic transducer, containing the audio information from the magnetic field; wherein the ear-wearable device comprises the first electroacoustic transducer configured to be positioned within an ear of a user and a hearing device housing, the hearing device housing comprising: a flexible mother circuit comprising and electrically connected to: a digital signal processor; an audio output device electrically connected to the digital signal processor, a plurality of user switches electrically connected to the digital signal processor, wherein the user switches control volume and on/off; an inertial measurement unit (IMU) sensor; and a first memory storage device configured to store data from the IMU sensor; wherein the first display information comprises at least one menu prompt to control the ear-wearable device, and wherein the hearing assisting system is configured to receive user input to execute the at least one menu prompt displayed, and wherein the at least one menu prompt is executed by the user tapping on the ear-wearable device.