# US 2023/0068099 A1 · Method and system for enhancing the intelligibility of information for a user

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0068099 A1
- Document ID: US20230068099A1
- Status: Published application
- Kind: A1
- Application number: 17/886,785
- Filed: 2022-08-12
- Published: 2023-03-02
- Earliest claimed: 2021-08-13
- Assignees: NeoSensory, Inc.
- Applicants: NeoSensory, Inc.
- Inventors: Oleksii Abramenko; Kaan Donbekci; Michael V. Perrotta; Scott Novich; Kathleen W. McMahon; David M. Eagleman
- Web page: https://patentgrep.com/patent/US20230068099A1

## CPC

- G10L 15/04: Segmentation; Word boundary detection
- G10L 15/16: using artificial neural networks
- G10L 15/063: {Training}
- G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
- G10L 21/0364: for improving intelligibility
- G10L 21/16: Transforming into a non-visible representation (devices or methods enabling ear patients to replace direct auditory perception by another kind of perception A61F11/04)
- G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A system for providing information to a user includes and/or interfaces with a set of models and/or algorithms. Additionally or alternatively, the system can include and/or interface with any or all of: a processing subsystem; a sensory output device; a user device; an audio input device; and/or any other components. A method for providing information to a user includes and/or interfaces with: receiving a set of inputs; processing the set of inputs to determine a set of sensory outputs; and providing the set of sensory outputs.

## Claims

### Claim 1

A method for enhancing an intelligibility of audio information for a user, the method comprising:
collecting the audio information;
processing the audio information to determine a set of features of the audio information with a set of trained models, the set of features comprising a set of high frequency phonemes and the set of trained models comprising a set of neural networks;
calculating a set of parameters associated with the set of features;
determining a set of adjustments for the set of parameters;
adjusting the set of parameters based on the set of adjustments, wherein adjusting the set of parameters is operating to enhance the intelligibility of the set of features;
determining an altered audio stream comprising the adjusted set of parameters in temporal relation with occurrence of the set of features; and
providing the altered audio stream to the user.

### Claim 2 (depends on 1)

The method of claim 1, wherein the altered audio stream comprises at least a portion of unaltered audio from the audio information.

### Claim 3 (depends on 2)

The method of claim 2, further comprising pre-processing the audio information prior to processing the audio information, wherein pre-processing the audio information comprises:
classifying the audio information with a set of classifiers, the set of classifiers comprising at least one of:
an accent classifier configured to classify an accent of a second user, the second user speaking the audio information; and
a sex classifier configured to classify a sex of the second user.

### Claim 4 (depends on 3)

The method of claim 3, wherein a particular trained model from the set of trained models is selected and used to process the audio information based on a set of outputs produced with the set of classifiers.

### Claim 5 (depends on 3)

The method of claim 3, wherein a set of adjustment values defining the set of adjustments is determined at least in part based on a set of outputs produced with the set of classifiers.

### Claim 6 (depends on 5)

The method of claim 5, wherein the set of adjustment values are further determined with an enhancement specific to the user, the enhancement determined based on an audiogram defining the user's hearing loss.

### Claim 7 (depends on 1)

The method of claim 1, wherein the set of trained models is trained to implement a latency-constrained loss function, the latency-constrained loss function operable to produce an output when processing the audio information with the set of trained models within a predetermined time threshold, thereby increasing the intelligibility of a conversation between the user and a second user, the conversation comprising the audio information.

### Claim 8 (depends on 7)

The method of claim 7, wherein the latency-constrained loss function enables the altered audio stream to be provided to the user with a delay of less than 100 milliseconds relative to a time at which the audio information was collected.

### Claim 9 (depends on 1)

The method of claim 1, wherein the set of parameters comprises at least one of a volume and a pitch.

### Claim 10 (depends on 9)

The method of claim 9, wherein the set of parameters is adjusted at least in part based on a severity parameter defining a hearing loss condition of the user.

### Claim 11 (depends on 1)

The method of claim 1, wherein the audio information is collected at a mobile user device during a phone call between the user and a second user.

### Claim 12

A method for enhancing an intelligibility of audio information for a user, the method comprising:
training a set of trained models, comprising:
producing a set of augmented audio data for use in training the set of trained models, wherein producing the set of augmented audio data comprises:
simulating the audio effects of a set of multiple room configurations to determine a set of room impulse response filters;
for a set of audio inputs:
altering the set of audio inputs based on the set of room impulse response filters; and
augmenting the set of audio inputs with a library of noise features;
with the set of augmented audio data, training the set of trained models, comprising:
implementing a constrained loss function, the constrained loss function operable to produce an output with the set of trained models within a predetermined time threshold, thereby increasing the interpretability of the audio information;
receiving the audio information and processing the audio information with the set of trained models to detect a set of high frequency phonemes present in the audio information;
producing a set of sensory outputs to be conveyed to the user based on the detected set of phonemes and a set of temporal features associated with the set of detected phonemes;
providing the set of sensory outputs to the user, thereby enhancing the intelligibility of the audio information; and
updating the set of trained models based on the audio information.

### Claim 13 (depends on 12)

The method of claim 12, wherein updating the set of trained models comprises fine-tuning parameters associated with the set of trained models based on the audio information.

### Claim 14 (depends on 13)

The method of claim 13, further comprising receiving a second instance of audio information and processing the second instance of audio information with the updated set of trained models.

### Claim 15 (depends on 12)

The method of claim 12, wherein training the set of trained models further comprises a curriculum learning process.

### Claim 16 (depends on 12)

The method of claim 12, wherein the set of trained models comprises a set of neural networks, the set of neural networks comprising a recurrent neural network and a convolutional neural network.

### Claim 17 (depends on 12)

The method of claim 12, wherein the constrained loss function is a latency-constrained Connectionist Temporal Classification (CTC) loss function, wherein the latency-constrained loss function enables the audio information and the sensory outputs to be provided to the user with a delay of less than 100 milliseconds relative to a time at which the audio information was collected.

### Claim 18 (depends on 12)

The method of claim 12, further comprising providing the audio information to the user, wherein the sensory outputs are provided in an overlaid fashion with the audio information.

### Claim 19 (depends on 18)

The method of claim 18, wherein the audio information comprises speech from a second user, wherein the audio information is collected at a mobile user device during a phone call between the user and the second user.

### Claim 20 (depends on 12)

The method of claim 12, further comprising converting the set of trained models into a representation configured for substantially real-time use on a set of resource-constrained embedded devices.
