# US 2024/0202458 A1 · Generating prompt recommendations for natural language processing tasks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202458 A1
- Document ID: US20240202458A1
- Status: Published application
- Kind: A1
- Application number: 18/067,680
- Filed: 2022-12-16
- Published: 2024-06-20
- Earliest claimed: 2022-12-16
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Sheng Zha; Miguel Ballesteros Martinez; Yassine Benajiba; Cole Hawkins; Aditya Rawal; Dhananjay Ram; Min Rong Samson Tan; Abhinav Goyal; Brant Swidler
- Web page: https://patentgrep.com/patent/US20240202458A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F 40/279: Recognition of textual entities
- G06F 40/30: Semantic analysis
- G06F 40/205: Parsing
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Prompt discovery is performed for identifying prompts to natural language processing machine learning models. A request to determine a prompt for a natural language processing task performed by a pre-trained natural language processing machine learning model may be received. A task classification for the natural language processing task may be determined and candidate prompts for the natural language processing prompt task collection selected. Respective prompt results for the candidate prompts are evaluated to generate a prompt recommendation for the natural language processing task.

## Claims

### Claim 1

A system, comprising:
at least one processor; and
a memory, storing program instructions that when executed by the at least one processor, cause the at least one processor to implement a prompt development system for natural language processing (NLP) machine learning (ML) models, the prompt development system configured to:
receive, via an interface of the prompt development system, a request to determine a prompt for an NLP task performed by a pre-trained NLP ML model;
determine a task classification for the NLP task based, at least in part, on the request;
access a prompt task collection maintained by the prompt development system that corresponds to the task classification and select one or more candidate prompts for the NLP task;
cause the one or more candidate prompts to be provided as input to the pre-trained NLP ML model to generate respective prompt results produced by the pre-trained NLP model;
evaluate the respective prompt results produced by the one or more candidate prompts using the pre-trained NLP ML model to generate a prompt recommendation of the NLP task; and
return, via the interface of the prompt development system, the prompt recommendation for the NLP task.

### Claim 2 (depends on 1)

The system of claim 1, wherein to determine the task classification for the NLP task, the prompt development system is configured to evaluate a sample input and sample output specified in the request.

### Claim 3 (depends on 1)

The system of claim 1, wherein the prompt development system is further configured to:
receive the one or more candidate prompts as prompt submissions to be maintained by the prompt development system; and
add the one or more candidate prompts to the prompt task collection.

### Claim 4 (depends on 1)

The system of claim 1, wherein the prompt development system is implemented as part of a machine learning service offered by a provider network, wherein the prompt task collection is one of a plurality of different prompt task collections maintained by the machine learning service that are collections of prompts submitted to the machine learning service via an interface of the machine learning service.

### Claim 5

A method, comprising:
receiving, at a prompt development system for natural language processing (NLP) machine learning (ML) models, a request to determine a prompt for an NLP task performed by a pre-trained NLP ML model;
selecting, by the prompt development system, one or more candidate prompts for the NLP task from a prompt task collection maintained by the prompt development system;
evaluating, by the prompt development system, respective prompt results produced by the one or more candidate prompts using the pre-trained NLP ML model; and
returning, by the prompt development system, a prompt recommendation for the NLP task based, at least in part, on the evaluating of the respective prompt results.

### Claim 6 (depends on 5)

The method of claim 5, further comprising determining, by the prompt development system, a task classification for the NLP task based, at least in part, on the request, wherein the prompt task classification corresponds to the task classification.

### Claim 7 (depends on 6)

The method of claim 6, wherein determining the task classification for the NLP task comprises identifying the task classification as specified in the request.

### Claim 8 (depends on 6)

The method of claim 6, wherein determining the task classification for the NLP task comprises evaluating a sample input and sample output specified in the request.

### Claim 9 (depends on 5)

The method of claim 5, wherein the prompt recommendation comprises two or more of the candidate prompts with respective samples outputs for comparison.

### Claim 10 (depends on 5)

The method of claim 5, further comprising selecting, by the prompt development system, one or more candidate pre-trained NLP ML models that corresponds to the task classification.

### Claim 11 (depends on 5)

The method of claim 5, wherein the prompt recommendation comprises an instruction and a stylistic feature.

### Claim 12 (depends on 5)

The method of claim 5, wherein the prompt recommendation comprises computational performance for one of the candidate prompts included in the prompt recommendation.

### Claim 13 (depends on 5)

The method of claim 5, wherein the prompt recommendation is further generated based on an evaluation of the one or more candidate prompts with respect to performance criteria specified in the request.

### Claim 14 (depends on 5)

The method of claim 5, further comprising:
receiving, by the prompt development system, the one or more candidate prompts as prompt submissions to be maintained by the prompt development system; and
adding, by the prompt development system, the one or more candidate prompts to the prompt task collection.

### Claim 15

One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:
receiving, at a prompt development system for natural language processing (NLP) machine learning (ML) models, a request to determine a prompt for an NLP task performed by a pre-trained NLP ML model;
selecting, by the prompt development system, one or more candidate prompts for the NLP task from a prompt task collection maintained by the prompt development system that corresponds to the task classification;
causing, by the prompt development system, an evaluation of respective prompt results produced by the one or more candidate prompts using the pre-trained NLP ML model; and
returning, by the prompt development system, a prompt recommendation for the NLP task based, at least in part, on the evaluating of the respective prompt results.

### Claim 16 (depends on 15)

The one or more non-transitory, computer-readable storage media of claim 15, wherein a task classification is specified in the request, wherein the prompt task collection corresponds to the task classification.

### Claim 17 (depends on 15)

The one or more non-transitory, computer-readable storage media of claim 15, storing further program instructions that when executed on or across the one or more computing devices to further implement evaluating a sample input and sample output specified in the request to determine a task classification, wherein the prompt task collection corresponds to the task classification.

### Claim 18 (depends on 14)

The one or more non-transitory, computer-readable storage media of claim 14, wherein the request does not specify the pre-trained NLP ML model, and wherein the prompt recommendation comprises an identification of the pre-trained NLP ML model.

### Claim 19 (depends on 14)

The one or more non-transitory, computer-readable storage media of claim 14, storing further program instructions that when executed on or across the one or more computing devices cause the one or more computing devices to further implement:
receiving, by the prompt development system, the one or more candidate prompts as prompt submissions to be maintained by the prompt development system; and
adding, by the prompt development system, the one or more candidate prompts to the prompt task collection.

### Claim 20 (depends on 14)

The one or more non-transitory, computer-readable storage media of claim 14, wherein the prompt development system is implemented as part of a machine learning service offered by a provider network, wherein the prompt task collection is one of a plurality of different prompt task collections maintained by the machine learning service that are collections of prompts submitted to the machine learning service via an interface of the machine learning service.
