# US 2026/0030249 A1 · Device-agnostic framework to measure reliability during user interactions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0030249 A1
- Document ID: US20260030249A1
- Status: Published application
- Kind: A1
- Application number: 19/348,532
- Filed: 2025-10-02
- Published: 2026-01-29
- Earliest claimed: 2024-04-22
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Shashwat Lal Das; Prithwish Mukherjee; Xiaoyi Sheng; Zilong Huang; Prashant Gupta; Alexander Sparber
- Web page: https://patentgrep.com/patent/US20260030249A1

## CPC

- G06F 16/24573: {using data annotations, e.g. user-defined metadata}
- G06F 16/285: {Clustering or classification}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Implementations relate to retrieving and processing metadata associated with a user query directed to an interactive assistant application. Implementations further relate to classifying the user query using labels assigned to invocation stage, input-receiving stage, response-receiving stage, and/or response-rendering stage of the user query that are determined based on processing the metadata associated with the user query. Whether the user query can be applied to evaluate a performance (e.g., surface reliability) of the interactive assistant application can be determined based on the classification of the user query.

## Claims

### Claim 1

A method implemented using one or more processors, the method comprising:
identifying a plurality of user queries directed to an interactive assistant application;
for each of the plurality of user queries:
identifying metadata associated with a respective user query from the plurality of user queries, and
processing the metadata associated with the respective user query to determine a classification category to which the respective user query belongs, wherein processing the metadata associated with the respective user query includes:
determining a respective label, from a plurality of predefined labels, for each of one or more stages of the interactive assistant application handling the respective user query, wherein the one or more stages belong to a plurality of predefined stages of the interactive assistant application, and
determining the classification category based on the respective label for each of the one or more stages of the interactive assistant application handling the respective user query, and
determining a surface reliability of the interactive assistant application based on the classification categories determined for a subset of the plurality of user queries that were received via a particular surface.

### Claim 2 (depends on 1)

The method of claim 1, wherein the plurality of predefined stages of the interactive assistant application include an invocation stage, an input-receiving stage, a response-receiving stage, and a response-rendering stage, of the interactive assistant application, handling the interactive assistant application.

### Claim 3 (depends on 1)

The method of claim 1, wherein the plurality of predefined labels include a first label indicating a respective stage of the interactive assistant application handling the user query was completed within a corresponding threshold of time, and a second label indicating the respective stage of the interactive assistant application handling the user query was not completed or was completed but beyond the corresponding threshold of time.

### Claim 4 (depends on 1)

The method of claim 1, wherein determining the classification category based on the respective label for each of the one or more stages of the interactive assistant application handling the respective user query comprises:
determining a first classification category for the respective user query based at least on each label for each of the plurality of predefined stages being the first label, the first classification category indicating a satisfactory overall surface performance of the interactive assistant application handling the user query, and
determining a second classification category for the respective user query based on at least one second label being determined for at least one of the plurality of predefined stages, the second classification category indicating an unsatisfactory overall surface performance of the interactive assistant application handling the user query.

### Claim 5 (depends on 4)

The method of claim 4, wherein determining the surface reliability of the interactive assistant application based on the classification categories determined for the subset of the plurality of user queries that were received via the particular surface comprises:
determining a ratio between a first quantity of user queries from the subset that each corresponds to the first classification category and a total quantity of user queries from the subset.

### Claim 6 (depends on 5)

The method of claim 5, wherein the plurality of predefined labels further include a third label indicating that the respective stage of the interactive assistant application handling the user query renders the user query ineligible, and/or a fourth label indicating that the metadata associated with the user query misses information to classify the respective stage of the interactive assistant application for the user query.

### Claim 7 (depends on 6)

The method of claim 6, wherein the subset of user queries include no user query for which a third or fourth label has determined or be associated with.

### Claim 8

A system comprising one or more processors and memory storing instructions that, when executed, cause the one or more processors to:
identify a plurality of user queries directed to an interactive assistant application;
for each of the plurality of user queries:
identify metadata associated with a respective user query from the plurality of user queries, and
process the metadata associated with the respective user query to determine a classification category to which the respective user query belongs, wherein processing the metadata associated with the respective user query includes:
determine a respective label, from a plurality of predefined labels, for each of one or more stages of the interactive assistant application handling the respective user query, wherein the one or more stages belong to a plurality of predefined stages of the interactive assistant application,
determine the classification category based on the respective label for each of the one or more stages of the interactive assistant application handling the respective user query, and
determine a surface reliability of the interactive assistant application based on the classification categories determined for a subset of the plurality of user queries that were received via a particular surface.

### Claim 9 (depends on 8)

The system of claim 8, wherein the plurality of predefined stages of the interactive assistant application include an invocation stage, an input-receiving stage, a response-receiving stage, and a response-rendering stage, of the interactive assistant application, handling the interactive assistant application.

### Claim 10 (depends on 8)

The system of claim 8, wherein the plurality of predefined labels include a first label indicating a respective stage of the interactive assistant application handling the user query was completed within a corresponding threshold of time, and a second label indicating the respective stage of the interactive assistant application handling the user query was not completed or was completed but beyond the corresponding threshold of time.

### Claim 11 (depends on 8)

The system of claim 8, wherein the instructions to determine the classification category based on the respective label for each of the one or more stages of the interactive assistant application handling the respective user query comprise instructions to:
determine a first classification category for the respective user query based at least on each label for each of the plurality of predefined stages being the first label, the first classification category indicating a satisfactory overall surface performance of the interactive assistant application handling the user query, and
determine a second classification category for the respective user query based on at least one second label being determined for at least one of the plurality of predefined stages, the second classification category indicating an unsatisfactory overall surface performance of the interactive assistant application handling the user query.

### Claim 12 (depends on 11)

The system of claim 11, wherein the instructions to determine the surface reliability of the interactive assistant application based on the classification categories determined for the subset of the plurality of user queries that were received via the particular surface comprise instructions to:
determine a ratio between a first quantity of user queries from the subset that each corresponds to the first classification category and a total quantity of user queries from the subset.

### Claim 13 (depends on 12)

The system of claim 12, wherein the plurality of predefined labels further include a third label indicating that the respective stage of the interactive assistant application handling the user query renders the user query ineligible, and/or a fourth label indicating that the metadata associated with the user query misses information to classify the respective stage of the interactive assistant application for the user query.

### Claim 14 (depends on 13)

The system of claim 13, wherein the subset of user queries include no user query for which a third or fourth label has determined or be associated with.

### Claim 15

A system comprising one or more processors and memory storing instructions that, when executed, cause the one or more processors to:
identify a plurality of user queries directed to an interactive assistant application;
for each of the plurality of user queries:
identify metadata associated with a respective user query from the plurality of user queries, and
process the metadata associated with the respective user query to determine a classification category to which the respective user query belongs, wherein processing the metadata associated with the respective user query includes:
determine a respective label, from a plurality of predefined labels, for each of one or more stages of the interactive assistant application handling the respective user query, wherein the one or more stages belong to a plurality of predefined stages of the interactive assistant application, and
determine the classification category based on the respective label for each of the one or more stages of the interactive assistant application handling the respective user query; and
determine a surface reliability of the interactive assistant application based on the classification categories determined for a subset of the plurality of user queries received via a particular surface.
