# US 2024/0005583 A1 · Systems and methods for ai driven generation of content attuned to a user

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005583 A1
- Document ID: US20240005583A1
- Status: Published application
- Kind: A1
- Application number: 18/369,061
- Filed: 2023-09-15
- Published: 2024-01-04
- Earliest claimed: 2021-05-18
- Assignees: Attune Media Labs, PBC
- Applicants: Attune Media Labs, PBC
- Inventors: Robert E. Bosnak; David E. Bosnak; Albert Rizzo
- Web page: https://patentgrep.com/patent/US20240005583A1

## CPC

- G06F 40/30: Semantic analysis
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 13/205: {driven by audio data}
- G06V 20/20: in augmented reality scenes
- G06V 40/174: {Facial expression recognition}
- G06V 40/176: {Dynamic expression}
- G06V 40/193: {Preprocessing; Feature extraction}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 25/63: for estimating an emotional state
- G06F 40/279: Recognition of textual entities
- G10L 13/033: Voice editing, e.g. manipulating the voice of the synthesiser
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Systems and methods enabling rendering an avatar attuned to a user. The systems and methods include receiving audio-visual data of user communications of a user. Using the audio-visual data, the systems and methods may determine vocal characteristics of the user, facial action units representative of facial features of the user, and speech of the user based on a speech recognition model and/or natural language understanding model. Based on the vocal characteristics, an acoustic emotion metric can be determined. Based on the speech recognition data, a speech emotion metric may be determined. Based on the facial action units, a facial emotion metric may be determined. An emotional complex signature may be determined to represent an emotional state of the user for rendering the avatar attuned to the emotional state based on a combination of the acoustic emotion metric, the speech emotion metric and the facial emotion metric.

## Claims

### Claim 1

A method comprising:
receiving, by a processor via an audio-visual input device, audio-visual input data of user communications during a period of time;
utilizing, by the processor, at least one speech recognition model to recognize speech data of the audio-visual input data;
inputting, by the processor, the speech data into at least one natural language understanding model to produce speech recognition data indicative of meaning, intent and sentiment;
determining, by the processor, at least one current emotional complex signature associated with user reactions during a current emotional state of the user during the period of time based at least in part on:
the speech recognition data and at least one of:
at least one time-varying speech emotion metric or
at least one time-varying facial emotion metric;
wherein the at least one emotion metric is determined by:
determining, by the processor, at least one time-varying speech emotion metric throughout the period of time based at least in part on the speech recognition data; and
wherein the at least one time-varying facial emotion metric is determined by:
utilizing, by the processor, at least one facial emotion recognition model to produce facial action units representative of recognized facial features represented in the audio-visual input data;
determining, by the processor, the at least one time-varying facial emotion metric throughout the period of time based at least in part on the speech recognition data, the facial action units and a facial action coding system;
logging, by the processor, at least one current emotional complex signature in a memory;
tagging, by the processor, a high amplitude-high confidence interaction to indicate at least one changed emotional state where a magnitude of the at least one current emotional complex signature exceeds a predetermined threshold; and
presenting, via at least one output device, by the processor, a virtual representation of a responder to the user in response to the at least one changed emotional state.

### Claim 2 (depends on 1)

The method as recited in claim 1, further comprising:
determining, by the processor, at least one vocal characteristic of acoustic data of the audio-visual input data based at least in part on at least one of wavelengths, frequencies or amplitudes of the acoustic data; and
determining, by the processor, at least one time-varying acoustic emotion metric based at least in part on the vocal characteristics.

### Claim 3 (depends on 2)

The method as recited in claim 2, wherein the vocal characteristics include at least one of pitch, loudness, shimmer, jitter, speech rate, harmonics or prosody characteristics.

### Claim 4 (depends on 1)

The method as recited in claim 1, wherein the representation of the responder comprises at least one of:
an interactive attuned vocal agent, or
an interactive attuned discrete avatar.

### Claim 5 (depends on 1)

The method as recited in claim 1, further comprising:
determining, by the processor, attuned facial action units attuned to the at least one changed emotional state;
generating, by the processor, a computer-generated face based at least in part on the attuned facial action units; and
rendering, via the at least one output device, by the processor, the virtual representation of the responder using the computer-generated face.

### Claim 6 (depends on 5)

The method as recited in claim 5, further comprising:
determining, by the processor, computer-generated speech based at least in part on the at least one changed emotional state;
determining, by the processor, attuned vocal qualities based at least in part on vocal characteristics of acoustic data of the audio-visual data;
determining, by the processor, a synchronization of the computer-generated face and the computer-generated speech based at least in part on the attuned vocal characteristics; and
rendering, via the at least one output device, by the processor, the virtual representation of the responder comprising an interactive attuned discrete avatar using the computer-generated face, the computer-generated speech and the synchronization of the computer-generated face and the computer-generated speech in response to the audio-visual input data.

### Claim 7 (depends on 1)

The method as recited in claim 1, wherein the facial action coding system comprises Paul Ekman's Facial Action Coding System.

### Claim 8 (depends on 1)

The method of claim 1, wherein the at least one facial recognition model comprises:
a gaze recognition and recording model to recognize and record eye gaze of the user;
a turn taking model to recognize a communication turn indicative of a turn to communicate; and
a pupil dilation model to determine pupil dilation of the user.

### Claim 9

A system comprising:
a processor in communication with a non-transitory computer readable medium having software instructions stored thereon, wherein the processor, upon execution of the software instructions, is further configured to:
receive, via an audio-visual input device, audio-visual input data of user communications during a period of time;
utilize at least one speech recognition model to recognize speech data of the audio-visual input data;
input the speech data into at least one natural language understanding model to produce speech recognition data indicative of meaning, intent and sentiment;
determine at least one current emotional complex signature associated with user reactions during a current emotional state of the user during the period of time based at least in part on:
the speech recognition data and at least one of:
at least one time-varying speech emotion metric or
at least one time-varying facial emotion metric;
wherein the at least one emotion metric is determined by:
determining at least one time-varying speech emotion metric throughout the period of time based at least in part on the speech recognition data; and
wherein the at least one time-varying facial emotion metric is determined by:
utilizing at least one facial emotion recognition model to produce facial action units representative of recognized facial features represented in the audio-visual input data;
determining the at least one time-varying facial emotion metric throughout the period of time based at least in part on the speech recognition data, the facial action units and a facial action coding system;
log at least one current emotional complex signature in a memory;
tag a high amplitude-high confidence interaction to indicate at least one changed emotional state where a magnitude of the at least one current emotional complex signature exceeds a predetermined threshold; and
present, via at least one output device a virtual representation of a responder to the user in response to the at least one changed emotional state.

### Claim 10 (depends on 9)

The system as recited in claim 9, wherein the processor, upon execution of the software instructions, is further configured to:
determine at least one vocal characteristic of acoustic data of the audio-visual input data based at least in part on at least one of wavelengths, frequencies or amplitudes of the acoustic data; and
determine at least one time-varying acoustic emotion metric based at least in part on the vocal characteristics.

### Claim 11 (depends on 10)

The system as recited in claim 10, wherein the vocal characteristics include at least one of pitch, loudness, shimmer, jitter, speech rate, harmonics or prosody characteristics.

### Claim 12 (depends on 9)

The system as recited in claim 9, wherein the representation of the responder comprises at least one of:
an interactive attuned vocal agent, or
an interactive attuned discrete avatar.

### Claim 13 (depends on 9)

The system as recited in claim 9, wherein the processor, upon execution of the software instructions, is further configured to:
determine attuned facial action units attuned to the at least one changed emotional state;
generate a computer-generated face based at least in part on the attuned facial action units; and
render, via the at least one output device, the virtual representation of the responder using the computer-generated face.

### Claim 14 (depends on 13)

The system as recited in claim 13, wherein the processor, upon execution of the software instructions, is further configured to:
determine computer-generated speech based at least in part on the at least one changed emotional state;
determine attuned vocal qualities based at least in part on vocal characteristics of acoustic data of the audio-visual data;
determine a synchronization of the computer-generated face and the computer-generated speech based at least in part on the attuned vocal characteristics; and
render, via the at least one output device the virtual representation of the responder comprising an interactive attuned discrete avatar using the computer-generated face, the computer-generated speech and the synchronization of the computer-generated face and the computer-generated speech in response to the audio-visual input data.

### Claim 15 (depends on 9)

The system as recited in claim 9, wherein the facial action coding system comprises Paul Ekman's Facial Action Coding System.

### Claim 16 (depends on 9)

The system of claim 9, wherein the at least one facial recognition model comprises:
a gaze recognition and recording model to recognize and record eye gaze of the user;
a turn taking model to recognize a communication turn indicative of a turn to communicate; and
a pupil dilation model to determine pupil dilation of the user.

### Claim 17

A non-transitory computer-readable medium comprising software instructions configured to cause at least one processor to perform steps comprising:
at least one processor in communication with at least one non-transitory computer-readable medium having software instructions stored thereon, wherein the software instructions are configured, upon execution, to cause the at least one processor to perform steps comprising:
receive, via an audio-visual input device, audio-visual input data of user communications during a period of time;
utilize at least one speech recognition model to recognize speech data of the audio-visual input data;
input the speech data into at least one natural language understanding model to produce speech recognition data indicative of meaning, intent and sentiment;
determine at least one current emotional complex signature associated with user reactions during a current emotional state of the user during the period of time based at least in part on:
the speech recognition data and at least one of:
at least one time-varying speech emotion metric or
at least one time-varying facial emotion metric;
wherein the at least one emotion metric is determined by:
determining at least one time-varying speech emotion metric throughout the period of time based at least in part on the speech recognition data; and
wherein the at least one time-varying facial emotion metric is determined by:
utilizing at least one facial emotion recognition model to produce facial action units representative of recognized facial features represented in the audio-visual input data;
determining the at least one time-varying facial emotion metric throughout the period of time based at least in part on the speech recognition data, the facial action units and a facial action coding system;
log at least one current emotional complex signature in a memory;
tag a high amplitude-high confidence interaction to indicate at least one changed emotional state where a magnitude of the at least one current emotional complex signature exceeds a predetermined threshold; and
present, via at least one output device a virtual representation of a responder to the user in response to the at least one changed emotional state.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium as recited in claim 17, wherein the software instructions are configured, upon execution, to cause the at least one processor to perform steps further comprising:
determine at least one vocal characteristic of acoustic data of the audio-visual input data based at least in part on at least one of wavelengths, frequencies or amplitudes of the acoustic data; and
determine at least one time-varying acoustic emotion metric based at least in part on the vocal characteristics.

### Claim 19 (depends on 17)

The non-transitory computer-readable medium as recited in claim 17, wherein the representation of the responder comprises at least one of:
an interactive attuned vocal agent, or
an interactive attuned discrete avatar.

### Claim 20 (depends on 17)

The non-transitory computer-readable medium as recited in claim 17, wherein the software instructions are configured, upon execution, to cause the at least one processor to perform steps further comprising:
determine computer-generated speech based at least in part on the at least one changed emotional state;
determine attuned vocal qualities based at least in part on vocal characteristics of acoustic data of the audio-visual data;
determine a synchronization of a computer-generated face and the computer-generated speech based at least in part on the attuned vocal characteristics; and
render, via the at least one output device, the virtual representation of the responder comprising an interactive attuned discrete avatar using the computer-generated face, the computer-generated speech and the synchronization of the computer-generated face and the computer-generated speech in response to the audio-visual input data.
