# US 2024/0203279 A1 · Content-driven virtual agent facilitator for online group activity

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203279 A1
- Document ID: US20240203279A1
- Status: Published application
- Kind: A1
- Application number: 18/287,036
- Filed: 2022-05-03
- Published: 2024-06-20
- Earliest claimed: 2021-05-04
- Assignees: Ramot at Tel-Aviv University Ltd.
- Applicants: Ramot at Tel-Aviv University Ltd.
- Inventors: Goren GORDON; Elinor MIZRAHI
- Web page: https://patentgrep.com/patent/US20240203279A1

## CPC

- G06V 40/16: Human faces, e.g. facial parts, sketches or expressions
- G09B 5/06: with both visual and audible presentation of the material to be studied
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
- G06N 20/00: Machine learning
- G06V 40/174: {Facial expression recognition}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 25/63: for estimating an emotional state
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

The disclosure is of a virtual agent (web based or robotic) for controlling group activities, such as learning and discussion. The aim is enhancing the effectivity of the activity by maintaining an effective, positive atmosphere and equitable participation. The agent applies multimodal input, comprising video and sound to monitor certain aspects of the group dynamics and their facets, pertaining to the users. Some aspects and facets thereof may trigger facilitation patterns, which may be performed using multimodal output. The agent uses a set of preset behavior, emotional patterns and facilitation patterns. The agent is trained using a dataset of recorded group activities and machine learning algorithms, and may apply online learning for optimization and personalization of the facilitation patterns and the associated enticers.

## Claims

### Claim 1

A computer implemented method of implementing and optimizing a virtual agent for conducting a group activity, the group comprises a plurality of users, the virtual agent comprising a conversational output platform, and the computer implemented method comprising:
using a conversational input platform for acquiring at least one behavioral or emotional aspect of the plurality of users;
analyzing the at least one aspect, using a machine learning model; and
when at least one facet of the at least one aspect, pertaining to at least one user, is matched with an enticer of at least one facilitation pattern, performing the at least one facilitation pattern, addressing the at least one user, using the conversational output platform.

### Claim 2 (depends on 1)

The computer implemented method of claim 1, further comprising updating the model according to rules corresponding to at least one following behavior or emotional aspect of the plurality of users.

### Claim 3 (depends on 2)

The computer implemented method of claim 2, further comprising personalizing the at least one facilitation pattern addressing the at least one user, using the facet pertaining to the at least one user of the at least one following behavior or emotional aspect of the plurality of users.

### Claim 4 (depends on 1)

The computer implemented method of claim 1, wherein at least one behavior patterns is whether a phrase complies with one or more elements of training material.

### Claim 5 (depends on 1)

The computer implemented method of claim 1, further comprising collecting at least one following behavior aspect of a user in response to the at least one facilitation pattern, for use in further training of the model.

### Claim 6 (depends on 1)

The computer implemented method of claim 1, further comprising comparing the facets of at least one behavior aspect, pertaining to the plurality of users, and applying an additional facilitation pattern chosen on accordance to the comparing.

### Claim 7 (depends on 6)

The computer implemented method of claim 6, wherein the facets is related to the participation time as a speaker in the group activity and the facilitation pattern is intended to promote evenness of participation.

### Claim 8 (depends on 1)

The computer implemented method of claim 1, wherein the at least one facet is related to expression of an emotion, and the at least one facilitation pattern is intended to help a user manage the emotion.

### Claim 9 (depends on 1)

The computer implemented method of claim 1, wherein the conversational input platform is multimodal sensor, comprising audio and video sensors for interacting with the group.

### Claim 10 (depends on 1)

The computer implemented method of claim 1, wherein the at least one aspect is a preset behavior pattern from a set including little interaction, excessive interaction, and negative response.

### Claim 11 (depends on 1)

The computer implemented method of claim 1, wherein at least one facilitation pattern is selected from a group consisting of comprises verbally addressing the least-participating user, verbally suggesting another point of view for a discussion facilitated for the group, displaying a face resembling feature gazing at a specific user.

### Claim 12 (depends on 1)

The computer implemented method of claim 1, wherein the machine learning model was trained using a preset knowledge base, comprising preset behavior and emotional patterns and at least one facilitation pattern.

### Claim 13 (depends on 1)

The computer implemented method of claim 1, further comprising updating the model by recommendations based on processing at least one behavior aspect of at least one of plurality of users by time-series clustering.

### Claim 14

A computer implemented method of training a virtual agent for conducting a group activity, the group comprises a plurality of users, the virtual agent comprising a conversational output platform, and the computer implemented method comprising:
feeding a dataset comprising data from a conversational input platform, and a set of facilitation patterns;
using a machine learning model for determining a goal function based on at least one aspect of the group activity, observed through the conversational input platform;
evaluating the goal function using evaluation of at least one aspect from at least one segment of the data from the conversational input platform; and
updating parameters of the machine learning model, the parameters controlling at least one facilitation pattern from the set of facilitation patterns.

### Claim 15 (depends on 14)

The computer implemented method of claim 14, wherein the goal function comprises a function of the emotional content of the group.

### Claim 16 (depends on 14)

The computer implemented method of claim 14, wherein the goal function comprises a function of the plurality of users' participation in a discussion facilitated for the group.

### Claim 17

A system for implementing and optimizing a virtual agent for conducting a group activity, the group comprises a plurality of users, the virtual agent comprising a conversational output platform, and the system is configured for:
using a conversational input platform for acquiring at least one behavioral or emotional aspect of the plurality of users;
analyzing the at least one aspect, using a machine learning model; and
when at least one facet of the at least one aspect, pertaining to at least one user, is matched with an enticer of at least one facilitation pattern, performing the at least one facilitation pattern, addressing the at least one user, using the conversational output platform.

### Claim 18 (depends on 17)

The system of claim 17, further comprising updating the model according to rules corresponding to at least one following behavior or emotional aspect of the plurality of users.

### Claim 19 (depends on 17)

The system of claim 17, further comprising comparing the facets of at least one behavior aspect, pertaining to the plurality of users, and applying an additional facilitation pattern chosen on accordance to the comparing.

### Claim 20

A computer readable medium having instructions stored thereon, which, when executed by a computer system, cause the computer system comprising a conversational output platform to carry out a computer-implemented method comprising:
using a conversational input platform for acquiring at least one behavioral or emotional aspect of a plurality of users;
analyzing the at least one aspect, using a machine learning model; and
when at least one facet of the at least one aspect, pertaining to at least one user, is matched with an enticer of at least one facilitation pattern, performing the at least one facilitation pattern, addressing the at least one user, using the conversational output platform.
