# US 2022/0036878 A1 · Speech assessment using data from ear-wearable devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036878 A1
- Document ID: US20220036878A1
- Status: Published application
- Kind: A1
- Application number: 17/443,756
- Filed: 2021-07-27
- Published: 2022-02-03
- Earliest claimed: 2020-07-31
- Assignees: Starkey Laboratories, Inc.
- Applicants: Starkey Laboratories, Inc.
- Inventors: Nicole Cyr; Karrie Recker; Dean G. Meyer; Jeffery Lee Crukley
- Web page: https://patentgrep.com/patent/US20220036878A1

## CPC

- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 1/1684: {Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675}
- G06F 1/1688: {the I/O peripheral being integrated loudspeakers}
- G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06N 20/00: Machine learning
- G09B 19/04: Speaking (with audible presentation of the material to be studied G09B5/04)
- G10L 25/27: characterised by the analysis technique
- G10L 25/48: specially adapted for particular use
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

A computing system may store user profile information of a user of an ear-wearable device, where the user profile information includes parameters that control operation of the ear-wearable device. The computing system may also obtain audio data from one or more sensors that are included in the ear-wearable device and determine whether to generate speech assessment data based on the user profile information of the user and audio data. In some examples, the computing system may compare one or more acoustic parameters determined based on the audio data with an acoustic criterion determined based on the user profile information of the user. If one or more acoustic parameters satisfy the acoustic criterion, the computing system may generate speech assessment data based on the determination.

## Claims

### Claim 1

A method comprising:
storing user profile information of a user of an ear-wearable device, wherein the user profile information comprises parameters that control operation of the ear-wearable device;
obtaining audio data from one or more sensors that are included in the ear-wearable device;
determining whether to generate speech assessment data based on the user profile information of the user and the audio data, wherein the speech assessment data provides information regarding speech of the user; and
generating the speech assessment data based on the determination to generate the speech assessment data.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining whether to generate the speech assessment data based on the user profile information of the user and the audio data further comprises:
determining whether to generate speech assessment data based on sensor data or location data.

### Claim 3 (depends on 1)

The method of claim 1, wherein determining whether to generate the speech assessment data based on the user profile information of the user and the audio data comprises:
determining one or more acoustic parameters based on the audio data;
determining an acoustic criterion based on the user profile information of the user;
comparing the one or more acoustic parameters to the acoustic criterion; and
determining to generate the speech assessment data in response to determining the one or more acoustic parameters satisfy the acoustic criterion.

### Claim 4 (depends on 1)

The method of claim 1, wherein generating the speech assessment data comprises:
determining whether the user has an abnormal speech pattern based on the audio data; and
in response to determining the user has the abnormal speech pattern, providing audible feedback, visual feedback, or vibrotactile feedback to the user.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating the speech assessment data comprises:
determining whether the user has abnormal speech patterns based on the audio data and the user profile information of the user;
in response to determining the user has the abnormal speech patterns, determining a potential type of abnormal speech patterns based on the audio data; and
generating a recommendation based on the potential type of abnormal speech patterns.

### Claim 6 (depends on 5)

The method of claim 5, wherein the type of abnormal speech patterns is determined based on speech and language attributes, wherein the speech and language attributes include voice attributes, speech quality attributes, language attributes, sociability attributes, or repetition attributes.

### Claim 7 (depends on 6)

The method of claim 6,
wherein the voice attributes include at least one of: frequency, amount of glottal fry, breathiness measurement, prosody measurement, or voice level (dB),
wherein the speech quality attributes include at least one of: mean length utterance (MLU), words per unit of time, amount of disfluencies, amount of filler words, amount of sound substitutions, amount of sound omissions, accuracy of vowel sounds, or articulation accuracy,
wherein the language attributes include at least one of: amount of grammar errors, amount of incorrect use of words, grade level of speech, or age level of speech,
wherein the sociability attributes include at least one of: amount of turn-taking, number of communication partners, duration of conversations, mediums of conversations, or repetition frequency, and
wherein repetition attributes include amount of repetition asked in a quiet environment, or amount of repetition asked in a noisy environment.

### Claim 8 (depends on 1)

The method of claim 1, wherein generating the speech assessment data comprises:
receiving a plurality of speech and language skill scores from a computing device;
generating a machine learning model based on the plurality of speech and language skill scores; and
generating the speech assessment data based on the audio data using the machine learning model.

### Claim 9 (depends on 8)

The method of claim 8, wherein the plurality of speech and language skill scores are provided by one or more human raters via the computing device, wherein the plurality of speech and language skill scores serve as inputs to the machine learning model, wherein the speech assessment data generated based on the audio data using the machine learning model comprises one or more machine-generated speech and language skill scores.

### Claim 10 (depends on 1)

The method of claim 1, wherein generating the speech assessment data comprises:
extracting speech features from the audio data;
generating the speech assessment data at least based on the extracted speech features and a normative speech profile; and
outputting the speech assessment data.

### Claim 11 (depends on 10)

The method of claim 10, wherein generating the speech assessment data at least based on the extracted speech features and the normative speech profile comprises:
selecting the normative speech profile from a plurality of normative speech profiles, wherein at least a portion of the normative speech profile matches the user profile information of the user;
generating one or more speech scores based on the extracted speech features and the selected normative speech profile; and
generating the speech assessment data based on the speech score.

### Claim 12 (depends on 1)

The method of claim 1, wherein the user profile information of the user further comprises at least one of: demographic information, an acoustic profile of own voice of the user, data indicating presence, status or settings of one or more pieces of hardware on the ear-wearable device, data indicating when a snapshot or the speech assessment data should be generated, data indicating which analyses should be performed on the audio data, data indicating which results should be displayed or sent to a companion computing device.

### Claim 13 (depends on 12)

The method of claim 12,
wherein the demographic information comprises one or more of: age, gender, geographic location, place of origin, native language, language that is being learned, education level, hearing status, socio-economic status, health condition, fitness level, speech or language diagnosis, speech or language goal, treatment type, or treatment duration of the user,
wherein the acoustic profile of own voice of the user comprises one or more of: the fundamental frequency of the user or one or more frequency relationships of sounds spoken by the user, wherein the one or more frequency relationships comprises formants and formant transitions,
wherein the settings of the one or more pieces of hardware on the ear-wearable device comprise one or more of: a setting of the one or more sensors, a setting of microphones, a setting of receivers, a setting of telecoils, a setting of wireless transmitters, a setting of wireless receivers, or a setting of batteries of the ear-wearable device, and
wherein the data indicating when the snapshot or the speech assessment data should be generated comprises one or more of: a specified time or a time interval, whether a sound class or an acoustic characteristic is identified, whether a specific activity is detected, whether a certain communication medium is detected, whether a certain biometric threshold has been passed, whether a specific geographic location is entered.

### Claim 14 (depends on 12)

The method of claim 12, further comprising:
receiving an instruction provided by the user or a third party; and
generating the speech assessment data based on the instruction, wherein the instruction comprises one or more of:
an on instruction configured to turn on the analyses;
an off instruction configured to turn off the analyses; and
an edit instruction configured to edit the analyses.

### Claim 15

A computing system comprising:
a data storage system configured to store data related to an ear-wearable device; and
one or more processing circuits configured to:
store user profile information of a user of the ear-wearable device,
wherein the user profile information comprises parameters that control operation of the ear-wearable device;
obtain audio data from one or more sensors that are included in the ear-wearable device;
determine whether to generate speech assessment data based on the user profile information of the user and the audio data, wherein the speech assessment data provides information regarding speech of the user; and
generate the speech assessment data based on the determination.

### Claim 16 (depends on 15)

The computing system of claim 15, wherein the one or more processing circuits are configured to:
determine whether the user has abnormal speech patterns based on the audio data and the user profile information of the user;
in response to determining the user has the abnormal speech patterns, determine a potential type of abnormal speech patterns based on the audio data; and
generate a recommendation based on the potential type of abnormal speech patterns.

### Claim 17 (depends on 15)

The computing system of claim 15, wherein the one or more processing circuits are configured to, as part of generating the speech assessment data:
receive a plurality of speech and language skill scores from a computing device;
generate a machine learning model based on the plurality of speech and language skill scores; and
generate the speech assessment data based on the audio data using the machine learning model.

### Claim 18 (depends on 15)

The computing system of claim 15, wherein the one or more processing circuits are configured to, as part of generating the speech assessment data:
extract speech features from the audio data;
generate the speech assessment data at least based on the extracted speech features and a normative speech profile; and
output the speech assessment data.

### Claim 19 (depends on 15)

The computing system of claim 15, wherein the user profile information of the user further comprises at least one of: demographic information, an acoustic profile of own voice of the user, data indicating presence, status or settings of one or more pieces of hardware on the ear-wearable device, data indicating when a snapshot or the speech assessment data should be generated, data indicating which analyses should be performed on the audio data, data indicating which results should be displayed or sent to a companion computing device.

### Claim 20

An ear-wearable device comprising:
one or more processors configured to:
store user profile information of a user of the ear-wearable device,
wherein the user profile information comprises parameters that control operation of the ear-wearable device;
obtain audio data from one or more sensors that are included in the ear-wearable device;
determine whether to generate speech assessment data based on the user profile information of the user and the audio data, wherein the speech assessment data provides information regarding speech of the user; and
generate the speech assessment data based on the determination.

### Claim 21 (depends on 20)

The ear-wearable device of claim 20, wherein the one or more processing circuits are configured to:
determine whether the user has abnormal speech patterns based on the audio data and the user profile information of the user;
in response to determining the user has the abnormal speech patterns, determine a potential type of abnormal speech patterns based on the audio data; and
generate a recommendation based on the potential type of abnormal speech patterns.

### Claim 22 (depends on 20)

The ear-wearable device of claim 20, wherein the one or more processing circuits are configured to, as part of generating the speech assessment data:
receive a plurality of speech and language skill scores from a computing device;
generate a machine learning model based on the plurality of speech and language skill scores; and
generate the speech assessment data based on the audio data using the machine learning model.

### Claim 23 (depends on 20)

The ear-wearable device of claim 20, wherein the one or more processing circuits are configured to, as part of generating the speech assessment data:
extract speech features from the audio data;
generate the speech assessment data at least based on the extracted speech features and a normative speech profile; and
output the speech assessment data.
