# US 10,311,858 B1 · Method and system for building an integrated user profile

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,311,858 B1
- Document ID: US10311858B1
- Status: Granted
- Kind: B1
- Application number: 15/385,493
- Filed: 2016-12-20
- Granted: 2019-06-04
- Earliest claimed: 2014-05-12
- Assignees: SoundHound, Inc.
- Applicants: SoundHound, Inc.
- Inventors: Bernard Mont-Reynaud; Jun Huang; Kiran Garaga Lokeswarappa; Joel Gedalius
- Web page: https://patentgrep.com/patent/US10311858B1

## CPC

- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G06F 40/205: Parsing
- G06F 40/211: Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
- G06F 40/253: Grammatical analysis; Style critique
- G06N 20/00: Machine learning
- G06Q 30/0251: Targeted advertisements
- G06Q 30/0276: {Advertisement creation}
- G06Q 30/0277: {Online advertisement}
- G10L 15/063: {Training}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 25/90: Pitch determination of speech signals
- H04L 67/306: User profiles
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 25/51: for comparison or discrimination
- G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A system and method are provided for adding user characterization information to a user profile by analyzing user's speech. User properties such as age, gender, accent, and English proficiency may be inferred by extracting and deriving features from user speech, without the user having to configure such information manually. A feature extraction module that receives audio signals as input extracts acoustic, phonetic, textual, linguistic, and semantic features. The module may be a system component independent of any particular vertical application or may be embedded in an application that accepts voice input and performs natural language understanding. A profile generation module receives the features extracted by the feature extraction module and uses classifiers to determine user property values based on the extracted and derived features and store these values in a user profile. The resulting profile variables may be globally available to other applications.

## Claims

### Claim 1

A non-transitory computer-readable medium storing instructions, which when executed by a processor of a car navigation system, cause the processor to:
extract an acoustic feature from user speech;
derive a second acoustic feature from the extracted acoustic feature;
use an accent classifier on the derived second acoustic feature to determine a value of an accent property;
assign the value to the accent property in a user profile; and
generate, by the car navigation system, voice instruction addressed to the user, the voice instruction bearing an accent that corresponds to the value of the accent property.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by a processor of a car navigation system, cause the car navigation system to:
extract a linguistic feature from user speech;
determine a value of a region property in the user profile based on the extracted linguistic feature; and
generate, by the car navigation system, voice instructions addressed to the user, the voice instructions comprising words from a vocabulary that corresponds to the value of the region property.

### Claim 3 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the classifier is trained using machine learning.

### Claim 4 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the classifier uses pitch statistics.

### Claim 5 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the extracted acoustic feature comprises at least one of: a cepstrum and a spectrogram.

### Claim 6 (depends on 1)

The non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by a processor of a car navigation system, cause the car navigation system to:
compare the value of the accent property to location context information to determine compatibility,
wherein the assigning of the value to the accent property in the user profile is performed responsive to determining that the value of the accent property is compatible with the location context information.

### Claim 7

A method of advertisement selection in an online ad system, the method comprising:
extracting an acoustic feature from user speech;
deriving a second acoustic feature from the extracted acoustic feature;
using an accent classifier on the derived second acoustic feature to determine a value of an accent property;
assigning the value to the accent property in a user profile; and
selecting, by the online ad system, an ad for display, the ad being aimed at users of a specific demographic group indicated by the value of the accent property.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
extracting a linguistic feature from user speech;
determining a value of a region property in the user profile based on the extracted linguistic feature; and
selecting, by the online ad system, an ad for display, the ad comprising words from a vocabulary that corresponds to the value of the region property.

### Claim 9 (depends on 7)

The method of claim 7, wherein the classifier is trained using machine learning.

### Claim 10 (depends on 7)

The method of claim 7, wherein the classifier uses pitch statistics.

### Claim 11 (depends on 7)

The method of claim 7, wherein the extracted acoustic feature comprises at least one of: a cepstrum and a spectrogram.

### Claim 12 (depends on 7)

The method of claim 7, further comprising:
comparing the value of the accent property to location context information to determine compatibility,
wherein the assigning of the value to the accent property in the user profile is responsive to determining that the value of the accent property is compatible with the location context information.

### Claim 13 (depends on 7)

The method of claim 7, wherein the ad is a music ad.

### Claim 14

A method of advertisement selection in an online ad system, the method comprising:
extracting an acoustic feature from user speech;
deriving a second acoustic feature from the extracted acoustic feature;
using an age classifier on the derived second acoustic feature to determine a value of an age range property;
assigning the value to the age range property in a user profile; and
selecting, by the online ad system, an ad for display, the ad being aimed at users having an age within the age range assigned to the age range property.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
extracting a linguistic feature from user speech;
determining the value of the age range property in the user profile based on the extracted linguistic feature; and
selecting, by the online ad system, an ad for display, the ad comprising words from a vocabulary that corresponds to the value of the age range property.

### Claim 16 (depends on 14)

The method of claim 14, wherein the classifier is trained using machine learning.

### Claim 17 (depends on 14)

The method of claim 14, wherein the classifier uses pitch statistics.

### Claim 18 (depends on 14)

The method of claim 14, wherein the extracted acoustic feature comprises at least one of: a cepstrum and a spectrogram.

### Claim 19 (depends on 14)

The method of claim 14, wherein the ad is a music ad.

### Claim 20

A non-transitory computer-readable medium storing instructions, which when executed by a processor of an online add system, cause the processor to:
extract an acoustic feature from user speech;
derive a second acoustic feature from the extracted acoustic feature;
use an age classifier on the derived second acoustic feature to determine a value of an age range property;
assign the value to the age range property in a user profile; and
select, by the online ad system, an ad for display, the ad being aimed at users having an age within the age range assigned to the age range property.

### Claim 21 (depends on 20)

The car navigation system of claim 20, further comprising instructions that, when executed by the one or more processors, cause the car navigation system implement actions comprising:
extracting a linguistic feature from user speech;
determining a value of a region property in the user profile based on the extracted linguistic feature; and
generating, by the car navigation system, voice instructions addressed to the user, the voice instructions comprising words from a vocabulary that corresponds to the value of the region property.

### Claim 22 (depends on 20)

The car navigation system of claim 20, wherein the classifier is trained using machine learning.

### Claim 23 (depends on 20)

The car navigation system of claim 20, wherein the classifier uses pitch statistics.

### Claim 24 (depends on 20)

The car navigation system of claim 20, wherein the extracted acoustic feature comprises at least one of: a cepstrum and a spectrogram.

### Claim 25 (depends on 20)

The car navigation system of claim 20, further comprising instructions that, when executed by the one or more processors, cause the car navigation system implement actions comprising:
comparing the value of the accent property to location context information to determine compatibility, wherein the assigning of the value to the accent property in the user profile is performed responsive to determining that the value of the accent property is compatible with the location context information.

### Claim 26

A non-transitory computer-readable medium storing instructions for advertisement selection, which when executed by a processor of an online ad system, cause the processor to implement a method comprising:
extracting an acoustic feature from user speech;
deriving a second acoustic feature from the extracted acoustic feature;
using an accent classifier on the derived second acoustic feature to determine a value of an accent property;
assigning the value to the accent property in a user profile; and
selecting, by the online ad system, an ad for display, the ad being aimed at users of a specific demographic group indicated by the value of the accent property.

### Claim 27 (depends on 26)

The non-transitory computer-readable medium of claim 26, wherein the method implemented by the processor further comprises:
extracting a linguistic feature from user speech;
determining a value of a region property in the user profile based on the extracted linguistic feature; and
selecting, by the online ad system, an ad for display, the ad comprising words from a vocabulary that corresponds to the value of the region property.

### Claim 28 (depends on 26)

The non-transitory computer-readable medium of claim 26, wherein the classifier is trained using machine learning.

### Claim 29 (depends on 26)

The non-transitory computer-readable medium of claim 26, wherein the classifier uses pitch statistics.

### Claim 30 (depends on 26)

The non-transitory computer-readable medium of claim 26, wherein the extracted acoustic feature comprises at least one of: a cepstrum and a spectrogram.

### Claim 31 (depends on 26)

The non-transitory computer-readable medium of claim 26, the method further comprising:
comparing the value of the accent property to location context information to determine compatibility, wherein the assigning of the value to the accent property in the user profile is responsive to determining that the value of the accent property is compatible with the location context information.

### Claim 32 (depends on 26)

The non-transitory computer-readable medium of claim 26, wherein the ad is a music ad.