# US 2019/0172462 A1 · Audio analysis learning with video data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0172462 A1
- Document ID: US20190172462A1
- Status: Published application
- Kind: A1
- Application number: 16/272,054
- Filed: 2019-02-11
- Published: 2019-06-06
- Earliest claimed: 2010-06-07
- Assignees: Affectiva, Inc.
- Applicants: Affectiva, Inc.
- Inventors: Taniya Mishra; Rana el Kaliouby
- Web page: https://patentgrep.com/patent/US20190172462A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- B60R 16/0373: {Voice control (in general G10L)}
- B60W 50/10: Interpretation of driver requests or demands
- G06V 40/172: {Classification, e.g. identification}
- G06V 40/176: {Dynamic expression}
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/1807: {using prosody or stress}
- G10L 15/25: using position of the lips, movement of the lips or face analysis
- G10L 21/055: for synchronising with other signals, e.g. video signals
- G10L 25/51: for comparison or discrimination
- G10L 25/63: for estimating an emotional state
- G10L 25/90: Pitch determination of speech signals
- B60W 2540/21: Voice
- B60W 2540/215: Selection or confirmation of options
- G10L 21/0356: for synchronising with other signals, e.g. video signals
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Audio analysis learning is performed using video data. Video data is obtained, on a first computing device, wherein the video data includes images of one or more people. Audio data is obtained, on a second computing device, which corresponds to the video data. A face within the video data is identified. A first voice, from the audio data, is associated with the face within the video data. The face within the video data is analyzed for cognitive content. Audio features corresponding to the cognitive content of the video data are extracted. The audio data is segmented to correspond to an analyzed cognitive state. An audio classifier is learned, on a third computing device, based on the analyzing of the face within the video data. Further audio data is analyzed using the audio classifier.

## Claims

### Claim 1

A computer-implemented method for audio analysis comprising:
obtaining video data, on a first computing device, wherein the video data includes images of one or more people;
obtaining audio data, on a second computing device, corresponding to the video data;
identifying a face within the video data;
associating a first voice, from the audio data, with the face within the video data;
analyzing the face within the video data for cognitive content;
learning an audio classifier, on a third computing device, based on the analyzing of the face within the video data, wherein the learning the audio classifier is based on analyzing a plurality of faces within the video data; and
analyzing further audio data using the audio classifier.

### Claim 2 (depends on 1)

The method of claim 1 further comprising extracting audio features corresponding to the cognitive content of the video data.

### Claim 3 (depends on 1)

The method of claim 1 further comprising segmenting the audio data to correspond to an analyzed cognitive state.

### Claim 4 (depends on 3)

The method of claim 3 wherein the segmenting the audio data is for a human sensorially detectable unit of time.

### Claim 5 (depends on 4)

The method of claim 4 wherein the segmenting the audio data includes noticeable differences in intensity, duration, or pitch.

### Claim 6 (depends on 3)

The method of claim 3 wherein the segmenting the audio data is for less than thirty seconds.

### Claim 7 (depends on 1)

The method of claim 1 further comprising synchronizing the audio data with the video data.

### Claim 8 (depends on 1)

The method of claim 1 further comprising analyzing a first voice for features.

### Claim 9 (depends on 8)

The method of claim 8 wherein the analyzing the first voice for features includes evaluation of timbre.

### Claim 10 (depends on 8)

The method of claim 8 wherein the analyzing the first voice for features includes evaluation of prosody.

### Claim 11 (depends on 8)

The method of claim 8 wherein the analyzing the first voice for features includes analysis of vocal register and vocal resonance, pitch, speech volumes, or speech rate.

### Claim 12 (depends on 8)

The method of claim 8 wherein the analyzing the first voice for features includes language analysis.

### Claim 13 (depends on 12)

The method of claim 12 wherein the language analysis is dependent on language content.

### Claim 14 (depends on 1)

The method of claim 1 wherein the learning is independent of language content.

### Claim 15 (depends on 1)

The method of claim 1 wherein the learning further encompasses learning a second audio classifier.

### Claim 16 (depends on 1)

The method of claim 1 wherein the cognitive content includes detection of one or more of sadness, stress, happiness, anger, frustration, confusion, disappointment, hesitation, cognitive overload, focusing, engagement, attention, boredom, exploration, confidence, trust, delight, disgust, skepticism, doubt, satisfaction, excitement, laughter, calmness, curiosity, humor, depression, envy, sympathy, embarrassment, poignancy, fatigue, drowsiness, or mirth.

### Claim 17 (depends on 1)

The method of claim 1 wherein the identifying of the face includes detection of facial expressions.

### Claim 18 (depends on 1)

The method of claim 1 further comprising determining a temporal audio signature for use with the further audio data.

### Claim 19 (depends on 1)

The method of claim 1 further comprising identifying and separating a second voice from the obtained audio data corresponding to the video data but not associated with the face associated with a first voice, wherein the second voice is included in the learning and wherein the second voice corresponds to a second person.

### Claim 20 (depends on 19)

The method of claim 19 further comprising associating the second voice, from the second person, with a second face within the video data.

### Claim 21 (depends on 1)

The method of claim 1 further comprising manipulating a vehicle based on the analyzing of the further audio data.

### Claim 22 (depends on 21)

The method of claim 21 wherein the manipulating the vehicle includes transferring into autonomous mode; transferring out of autonomous mode; locking out operation; recommending a break for an occupant; recommending a different route; recommending how far to drive; responding to traffic; adjusting seats, mirrors, climate control, lighting, music, audio stimuli, or interior temperature; activating brakes; and assuming steering control.

### Claim 23 (depends on 1)

The method of claim 1 wherein the cognitive content includes emotional content or mental state content.

### Claim 24 (depends on 1)

The method of claim 1 wherein the first computing device and the second computing device, or the second computing device and the third computing device, are a common device.

### Claim 25 (depends on 1)

The method of claim 1 wherein the learning further comprises:
synchronizing the audio data and the video data;
extracting an audio feature associated with the cognitive content that was analyzed from the face; and
abstracting an audio classifier based on the extracted audio feature.

### Claim 26

A computer program product embodied in a non-transitory computer readable medium for audio analysis, the computer program product comprising code which causes one or more processors to perform operations of:
obtaining video data wherein the video data includes images of one or more people;
obtaining audio data corresponding to the video data;
identifying a face within the video data;
associating a first voice, from the audio data, with the face within the video data;
analyzing the face within the video data for cognitive content;
learning an audio classifier, on a third computing device, based on the analyzing of the face within the video data, wherein the learning the audio classifier is based on analyzing a plurality of faces within the video data; and
analyzing further audio data using the audio classifier.

### Claim 27

A computer system for audio analysis comprising:
a memory which stores instructions;
one or more processors attached to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to:
obtain video data, on a first computing device, wherein the video data includes images of one or more people;
obtain audio data, on a second computing device, corresponding to the video data;
identify a face within the video data;
associate a first voice, from the audio data, with the face within the video data;
analyze the face within the video data for cognitive content;
learn an audio classifier, on a third computing device, based on the analyzing of the face within the video data, wherein the learning the audio classifier is based on analyzing a plurality of faces within the video data; and
analyze further audio data using the audio classifier.
