# US 2025/0157481 A1 · Visualizing Auditory Content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0157481 A1
- Document ID: US20250157481A1
- Status: Published application
- Kind: A1
- Application number: 19/023,598
- Filed: 2025-01-16
- Published: 2025-05-15
- Earliest claimed: 2016-07-16
- Inventors: Ron Zass
- Web page: https://patentgrep.com/patent/US20250157481A1

## CPC

- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 21/0364: for improving intelligibility
- G06F 40/10: Text processing (natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40)
- G10L 17/00: Speaker identification or verification techniques
- H04R 1/406: {microphones}
- H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)}
- H04R 25/407: {Circuits for combining signals of a plurality of transducers}
- G02B 27/017: {Head mounted}
- G10L 2015/221: {Announcement of recognition results}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

Systems, methods and non-transitory computer readable media for processing audio and visually presenting information are provided. Audio data may be obtained. The audio data may be analyzed to obtain textual information. The audio data may be analyzed to associate different portions of the textual information with different speakers. Each portion of the textual information may be presented in a presentation region associated with the speaker associated with the portion of the textual information. The audio data may be analyzed to identify a nonverbal sound. A textual description of the nonverbal sound may be generated.

## Claims

### Claim 1

A non-transitory computer readable medium storing data and computer implementable instructions that when executed by at least one processor cause the at least one processor to perform a method for processing audio and visually presenting information, the method comprising:
obtaining audio data;
analyzing the audio data to obtain textual information;
analyzing the audio data to associate different portions of the textual information with different speakers;
presenting each portion of the textual information in a presentation region associated with the speaker associated with the portion of the textual information;
analyzing the audio data to identify a nonverbal sound; and
generating a textual description of the nonverbal sound.

### Claim 2 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the method further comprises:
analyzing the audio data to identify different voice properties in different parts of the audio data;
associating different parts of the textual information with different voice properties;
determining information based on particular voice properties; and
presenting the information determined based on the particular voice properties along the part of the textual information associated with the particular voice properties.

### Claim 3 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the method further comprises:
analyzing the audio data to identify a melody; and
generating textual description of the melody.

### Claim 4 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the presentation is a presentation using a head mounted display system.

### Claim 5 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the presentation is a presentation using an augmented reality display system, and the association of the presentation regions with the speakers is configured to overlay the portions of the textual information associated with a speaker over the speaker in the augmented reality display system.

### Claim 6 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the method further comprises: analyzing the audio data to identify an item in the audio data; representing the item using a graphical symbol; and displaying the graphical symbol in conjunction with the textual information.

### Claim 7 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the method further comprises displaying the different portions of the textual information using different sets of visual display parameters.

### Claim 8 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the method further comprises: associating the different parts of the textual information with different textual formats based on the association of the different parts of the textual information with the different voice properties.

### Claim 9 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the method further comprises associating the different portions of the textual information with different textual formats based on the association of the different portions of the textual information with the different speakers.

### Claim 10 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the method further comprises: determining that a portion of a speech is said in a specific linguistic tone; selecting a visual display parameter for a part of the textual information associated with the portion of the speech based on the specific linguistic tone; and displaying the part of the textual information using the selected visual display parameter.

### Claim 11 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the method further comprises visually presenting information associated with a speaker in conjunction with a portion of the textual information associated with the speaker.

### Claim 12 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the association of the presentation regions with the speakers is based on spatial orientation of the speakers.

### Claim 13 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the association of the presentation regions with the speakers is based on positions of the speakers.

### Claim 14 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the method further comprises: associating a part of the textual information with speech produced by a user; and avoiding presenting the part of the textual information associated with the speech produced by the user.

### Claim 15 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the method further comprises: determining that a part of the textual information is associated with speech that do not involve the user; and avoiding presenting the part of the textual information associated.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the speech that do not involve the user is a conversation that do not involve the user.

### Claim 17 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the speech that do not involve the user is a speech not directed at the user.

### Claim 18 (depends on 1)

The non-transitory computer readable medium of claim 1, wherein the audio data is audio data captured by the one or more audio sensors from an environment of a wearer of a wearable apparatus.

### Claim 19

A system for processing audio and visually presenting information, the system comprising:
at least one processing unit configured to:
obtain audio data;
analyze the audio data to obtain textual information;
analyze the audio data to associate different portions of the textual information with different speakers;
present each portion of the textual information in a presentation region associated with the speaker associated with the portion of the textual information;
analyze the audio data to identify a nonverbal sound; and
generate a textual description of the nonverbal sound.

### Claim 20

A method for processing audio and visually presenting information, the method comprising:
obtaining audio data;
analyzing the audio data to obtain textual information;
analyzing the audio data to associate different portions of the textual information with different speakers;
presenting each portion of the textual information in a presentation region associated with the speaker associated with the portion of the textual information;
analyzing the audio data to identify a nonverbal sound; and
generating a textual description of the nonverbal sound.
