# US 2025/0165744 A1 · Method and system for integrated multimodal input processing for virtual agents

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0165744 A1
- Document ID: US20250165744A1
- Status: Published application
- Kind: A1
- Application number: 18/511,796
- Filed: 2023-11-16
- Published: 2025-05-22
- Earliest claimed: 2023-11-16
- Assignees: Quantiphi, Inc
- Applicants: Quantiphi, Inc.
- Inventors: Dagnachew Birru; Saisubramaniam Gopalakrishnan; Siva Prasad Sompalli; Varun V; Vishal Vaddina
- Web page: https://patentgrep.com/patent/US20250165744A1

## CPC

- G06N 20/00: Machine learning
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/08: Learning methods
- G06N 5/041: {Abduction}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method and system for multimodal input processing for a virtual agent is provided herein. The method comprises obtaining a multimodal input by the virtual agent from a user. The method further comprises identifying a plurality of principal entities within the multimodal input. The method further comprises extracting information about each entity of the plurality of principal entities. Further, the method comprises generating a response based on the extracted information.

## Claims

### Claim 1

A computer-implemented method for multimodal input processing for a virtual agent comprising:
obtaining a multimodal input by the virtual agent from a user, wherein the virtual agent employs an Artificial Intelligence (AI) model;
identifying a plurality of principal entities within the multimodal input;
extracting information about each entity of the plurality of principal entities; and
generating a response based on the extracted information.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the AI model is a Generative AI model.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising storing the extracted information within an associated database in each cycle of input processing.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the multimodal input comprises data from modalities comprising sensors, ensembled data, speech, text, and vision.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising dynamically adapting user's accustomed communication style based on historical interactions of the virtual agent with the user.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the AI model employs a role-based approach following user-provided instructions and prompts.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the AI model is trained to understand and respond to user emotions conveyed through the multimodal input.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising continuously monitoring user engagement and satisfaction during interactions.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the principal entities are selected from a group of a name, a date, a time, a numeric value, an address, a location, a sentiment, an emotional cue, a facial feature, a visual cue, a gesture, a body language, a parameter, an object, a command, and a keyword.

### Claim 10

A computer system multimodal input processing for a virtual agent comprising, the computer system comprising: one or more computer processors, one or more computer readable memories, one or more computer readable storage devices, and program instructions stored on the one or more computer readable storage devices for execution by the one or more computer processors via the one or more computer readable memories, the program instructions comprising:
obtaining a multimodal input by the virtual agent from a user, wherein the virtual agent employs an Artificial Intelligence (AI) model;
identifying a plurality of principal entities within the multimodal input;
extracting information about each entity of the plurality of principal entities; and
generating a response based on the extracted information.

### Claim 11 (depends on 10)

The system of claim 10, wherein the AI model is a Generative AI model.

### Claim 12 (depends on 10)

The system of claim 10, wherein the program instructions further comprising storing the extracted information within an associated database in each cycle of input processing.

### Claim 13 (depends on 10)

The system of claim 10, wherein the multimodal input comprises data from modalities comprising sensors, ensembled data, speech, text, and vision.

### Claim 14 (depends on 10)

The system of claim 10, wherein the program instructions further comprising dynamically adapting user's accustomed communication style based on historical interactions of the virtual agent with the user.

### Claim 15 (depends on 10)

The system of claim 10, wherein the AI model employs a role-based approach following user-provided instructions and prompts.

### Claim 16 (depends on 10)

The system of claim 10, wherein the AI model is trained to understand and respond to user emotions conveyed through the multimodal input.

### Claim 17 (depends on 10)

The system of claim 10, wherein the program instructions further comprising continuously monitoring user engagement and satisfaction during interactions.

### Claim 18 (depends on 10)

The system of claim 10, wherein the principal entities are selected from a group of a name, a date, a time, a numeric value, an address, a location, a sentiment, an emotional cue, a facial feature, a visual cue, a gesture, a body language, a parameter, an object, a command, and a keyword.

### Claim 19

A non-transitory computer-readable storage medium having stored thereon computer executable instruction which when executed by one or more processors, cause the one or more processors to carry out operations for multimodal input processing for a virtual agent, the operations comprising perform the operations comprising:
obtaining a multimodal input by the virtual agent from a user, wherein the virtual agent employs an Artificial Intelligence (AI) model;
identifying a plurality of principal entities within the multimodal input;
extracting information about each entity of the plurality of principal entities; and
generating a response based on the extracted information.
