# US 11,908,476 B1 · System and method of facilitating human interactions with products and services over a network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,476 B1
- Document ID: US11908476B1
- Status: Granted
- Kind: B1
- Application number: 18/471,932
- Filed: 2023-09-21
- Granted: 2024-02-20
- Earliest claimed: 2023-09-21
- Assignees: Rabbit Inc.
- Applicants: Cyber Manufacture Co.
- Inventors: Cheng Lyu; Peiyuan Liao; Zhuoheng Yang
- Web page: https://patentgrep.com/patent/US11908476B1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 40/30: Semantic analysis
- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L 15/04: Segmentation; Word boundary detection
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An artificial intelligence enabled system is disclosed. The system includes a core component for enabling AI-powered interactions between the system and its users and one or more agents that understand user intent and automatically interact with products and services on the web and/or in the physical world through imitation of a human user.

## Claims

### Claim 1

A system comprising:
a core component configured to perform the following steps in reconciliation loop in a streaming fashion where one or more of the steps may be performed in parallel:
receiving a user input from a user;
segmenting the user input;
performing streaming automatic speech recognition on the user input;
performing streaming large language model (LLM) or natural language processing on the speech recognized user input to generate a first output;
communicating the first output to one or more agents;
receiving a second output form the one or more agents; and
generating a result from the second output from the one or more agents; and
the one or more agents, each configured to imitate user interaction with a corresponding application over a network;
wherein each of the one or more agents comprises:
an intention model configured to translate a user intention into one or more actions without receiving explicit step by step execution instructions from the user;
an interface model in communication with the intention model and configured to translate one or more requests from the intention model into a sequence of user interface (UI) actions on one or more target applications; and
an action executor in communication with the interface model and configured to execute the sequence of UI actions on the one or more target applications.

### Claim 2 (depends on 1)

The system of claim 1, wherein the user input is an audio input and the core component is configured to perform speech to text conversion on the user input.

### Claim 3 (depends on 2)

The system of claim 2, wherein the result is in text format and the core component is configured to perform text to speech conversion on the result.

### Claim 4 (depends on 1)

The system of claim 1, wherein generating the result from the second output from the one or more agents comprises the core component constantly receiving updates from one or more of other core component replicas, other Artificial Intelligence (AI) services, and state changes from external events.

### Claim 5 (depends on 1)

The system of claim 1, wherein the core component further comprises a multi-modal memory system based on vector databases configured to improve the recognition quality of speech to text conversion.

### Claim 6 (depends on 5)

The system of claim 5, wherein the multi-modal memory is persistent and comprises one or more of raw captured sounds, transcripts, and external tool and agent invocations.

### Claim 7 (depends on 1)

The system of claim 1, wherein the intention model is configured to have access to a memory system based on vector databases, and a customized re-ranker, to infer additional context requests.

### Claim 8 (depends on 1)

The system of claim 1, wherein the intention model is configured to:
receive the first output form the core component;
decompose the first output to one or more tasks;
assign each of the one or more tasks to a corresponding language-model-powered algorithm for processing;
synchronize results from the processing of the one or more tasks;
finalizing and presenting to interface model intermediate information from the synchronized results.

### Claim 9 (depends on 1)

The system of claim 1, wherein the interface model comprises:
a static model configured to understand UI components of an application; and
a dynamic model configured to understand interactions among the UI components and operations that occur when a UI action is completed.

### Claim 10 (depends on 9)

The system of claim 9, wherein the static model comprises:
an input layer configured to handle user inputs; and
a view layer configured to handle the UI components and actions associated with the UI components.

### Claim 11 (depends on 9)

The system of claim 9, wherein the dynamic model comprises:
an entity layer configured to recognize entities being manipulated by an application, a relationship between entities, and attributes of the entities;
a transition layer configured to extract all available operations on the recognized entities; and
a semantics layer configured to assign physical semantics to operations and entities.

### Claim 12 (depends on 9)

The system of claim 9, wherein the interface model further comprises a demonstration model configured to train the interface model using a video of operating the application or a browser session comprising the document object model change stream.

### Claim 13 (depends on 12)

The system of claim 12, wherein the demonstration model uses an auto-formalization process of creating a formalized system that approximates observed UI interactions.

### Claim 14 (depends on 1)

The system of claim 1, wherein the action executor comprises:
a container orchestrator; and
one or more agent hosts;
wherein the container orchestrator is configured to spawn, manage, and interact with one or more agent hosts; and
wherein each of the one or more agent hosts comprises containerized environment for running an application.

### Claim 15 (depends on 14)

The system of claim 14, wherein the container orchestrator is configured to allocate computing resources to the one or more agent hosts according to nature of the apps.

### Claim 16 (depends on 14)

The system of claim 14, wherein the container orchestrator is implemented based on Kubernetes clusters and alternative workload orchestrators.

### Claim 17 (depends on 16)

The system of claim 16, wherein each of the one or more agent hosts is configured to send UI actions to the application and collect UI signals from the application.

### Claim 18 (depends on 17)

The system of claim 17, wherein the UI signals comprise one or more of video, audio, DOM change stream from the application.

### Claim 19 (depends on 17)

The system of claim 17, where the UI actions are received from either the interface model or the user.