US 12,586,575 B1Granted

Updating an artificial intelligence system

AssigneeAmazon Technologies, Inc.

InventorsMohammad Kachuee, Vaibhav Kumar, Xiang Li, Puyang Xu, Xiaohu Liu, Xing Fan

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques for improving query generation (or generation of an input to another component) by a language model are described. In some embodiments, the generated query is used to retrieve API calls relevant for responding to a user input. The language model can be finetuned using language model generated queries. Results retrieved using the generated query are evaluated against ground truth data to determine performance metrics data, which may be based on a ranking of the ground truth API call in the retrieved results. Based on the performance metrics data, training data including the generated query can be used to update/finetune the language model may be determined. Updating of the language model may be initiated based on feedback corresponding to the generated query.

Claims20

Dependent claims

A computer-implemented method comprising:

receiving a natural language user input requesting performance of a first action;
determining a natural language prompt including the natural language user input and a first request to determine at least one Application Programming Interface (API) call to cause performance of the first action;
processing, using a language model, the natural language prompt to generate a natural language query for an API storage component, the natural language query requesting at least one API call to cause performance of the first action;
in response to the natural language query, receiving, from the API storage component, a ranked list of API calls including at least a first API call;
determining, from first data representing associations between user requested actions and API calls, a second API call capable of performing the first action;
processing the ranked list of API calls and the second API call to determine metric data corresponding to performance of the language model with respect to generation of the natural language query to cause retrieval of the second API call from the API storage component;
based on the metric data satisfying a condition, determining second data including the natural language prompt and the natural language query;
and
training the language model using the second data.

6 more elements

A computer-implemented method comprising:

determining a first prompt including a first request to determine first data corresponding to a first action;
processing, using a language model, the first prompt to generate a first query for a storage component, the first query requesting the first data corresponding to the first action;
in response to the first query, receiving, from the storage component, at least first Application Programming Interface (API) data;
determining second API data corresponding to the first action, the second API data being ground truth data;
determining first performance metrics data based on a first correspondence between the first API data and the second API data;
and
based on the first performance metrics data, determining second data including the first prompt and the first query, the second data to be used for training the language model.

4 more elements

A system comprising:

at least one processor;
and
at least one memory including instructions that, when executed by the at least one processor, cause the system to:
determine a first prompt including a first request to determine first data corresponding to a first action;
process, using a language model, the first prompt to generate a first query for a storage component, the first query requesting the first data corresponding to the first action;
in response to the first query, receive, from the storage component, at least first (Application Programming Interface) API data;
determine second API data corresponding to the first action, the second API data being ground truth data;
determine first performance metrics data based on a first correspondence between the first API data and the second API data;
and
based on the first performance metrics data, determine second data including the first prompt and the first query, the second data to be used for training the language model.

7 more elements

Dependencies are inferred from the claim text.