# US 2023/0409615 A1 · Systems and Methods for Providing User Experiences on Smart Assistant Systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0409615 A1
- Document ID: US20230409615A1
- Status: Published application
- Kind: A1
- Application number: 18/334,235
- Filed: 2023-06-13
- Published: 2023-12-21
- Earliest claimed: 2022-06-16
- Assignees: Meta Platforms, Inc.
- Applicants: Meta Platforms, Inc.
- Inventors: Piyush Khemka; Brandon Ramos; Ryan Wolff; Stephen Chee-Ching Wu; Ashley Gustafson; Gabrielle Catherine Moskey; Hyundong Cho; Andrea Madotto; Zhaojiang Lin; Satwik Kottur; Chinnadhurai Sankar; Ashish Vishwanath Shenoy; Jiangning Chen; Rahim Manji; Bing Liu; Xin Liu; Ziyun Zhang
- Web page: https://patentgrep.com/patent/US20230409615A1

## CPC

- G06F 8/38: for implementing user interfaces
- G06F 16/3329: Natural language query formulation
- G06F 9/451: Execution arrangements for user interfaces
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06V 20/68: Food, e.g. fruit or vegetables
- G10L 15/1822: {Parsing for meaning understanding}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

In one embodiment, a system includes an automatic speech recognition (ASR) module, a natural-language understanding (NLU) module, a dialog manager, one or more agents, an arbitrator, a delivery system, one or more processors, and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to receive a user input, process the user input using the ASR module, the NLU module, the dialog manager, one or more of the agents, the arbitrator, and the delivery system, and provide a response to the user input.

## Claims

### Claim 1

A method comprising, by one or more computing systems:
receiving, from a first client system associated with a user, a user input;
executing a task corresponding to the user input;
generating a user interface for delivering executing results of the task based on a meta design system, wherein the meta design system is based on at least an attention system and an assistant layer; and
sending, to the first client system and one or more second client systems associated with the user, instructions for presenting the user interface.

### Claim 2

A method comprising, by a client system:
capturing, by one or more cameras associated with the client system, visual signals associated with a field of view of a user;
identifying one or more food items based on the visual signals;
determining calorie and nutrient information associated with the one or more food items;
selecting one or more of the food items based on (a) the calorie and nutrient information and (b) knowledge about the user; and
presenting, at the client system, recommendations for the selected food items.

### Claim 3

A method comprising, by one or more computing systems:
receiving, from a client system associated with a first user, a first user utterance at a first turn associated with a first dialog session in a first domain;
determining a first dialog state associated with the first user utterance based on one or more slots associated with first user utterance;
transforming, based on the first dialog state, the one or more slots in the first domain to one or more questions, respectively;
retrieving, from a database based on the one or more questions, one or more example dialogs; and
generating, based on the one or more questions and the one or more example dialogs, a dialog-state-tracking model.

### Claim 4

A method comprising, by a client system:
receiving, at the client system, an incoming message for a first user;
determining, based on the incoming message by one or more machine-learning models, one or more candidate responses for the first user, wherein the one or more machine-learning models are trained based on a plurality of prior user-typed keystrokes in response to a plurality of prior incoming messages; and
presenting, at the client system, the one or more candidate responses.

### Claim 5

A method comprising, by one or more computing systems:
receiving, from a client system associated with a user, a user utterance comprising an entity mention;
accessing a knowledge graph comprising a plurality of entity pairs, wherein at least one of the entity pairs comprises a resolved entity name and a prior entity mention failed to be resolved, and wherein the at least one entity pair is generated based on one or more of cosine similarity or lexical string similarity between the resolved entity name and the prior entity mention failed to be resolved;
resolving an entity name to the entity mention in the user utterance;
executing one or more tasks associated with the entity name resolved to the entity mention in the user utterance; and
sending, to the client system responsive to the user utterance, instructions for presenting a response generated based on execution results of one or more of the tasks.
