US 2023/0418566 A1Published application

Programmatically generating evaluation data sets for code generation models

AssigneeAmazon Technologies, Inc.

InventorsPraphruetpong Athiwaratkun, Zixuan Lin, Ramana Keerthi, Zijian Wang, Yuchen Tian, Hantian Ding, Sri Ranga Akhilesh Bontala, Matthew Lee, Yanitsa Donchev, Ramesh M Nallapati, Parminder Bhatia, Andrew Oliver Arnold, Bing Xiang, Sudipta Sengupta, Rama Krishna Sandeep Pokkunuri, Srinivas Iragavarapu, Atul Deo, Ankur Deepak Desai

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Evaluation data sets may be programmatically generated for code generation models. An evaluation data set is obtained that includes items that correspond to different evaluation tests for a code generation system. The individual items of the evaluation data set maybe converted, including the conversion of a function signature for the items, the test statements for the items and using a code generation system to generate the body of the function.

Claims20

Dependent claims

A system, comprising:

at least one processor;
and
a memory, storing program instructions that when executed by the at least one processor, cause the at least one processor to implement a programming language conversion system, configured to:
receive, via an interface of the programming language conversion system, a request to convert an evaluation data set specified in a first programming language, wherein different items of the evaluation data set correspond to different respective evaluation tests for a code generation system;
convert individual ones of the different items of the data set into a second programming language:
convert a function signature of the item in the first programming language to the second programming language;
convert one or more test statements of the item in the first programming language to the second programming language;
send a request to the code generation system to generate a body of the converted function signature in the second programming language according to a prompt in the item;
and
receive the body of the function signature in the second programming language from the code generation system;
and
store the converted individual ones of the different items of the evaluation data set as part of a new evaluation data set.

9 more elements

A method, comprising:

receiving, via an interface of a programming language conversion system, an evaluation data set specified in a first programming language, wherein different items of the evaluation data set correspond to different respective evaluation tests for a code generation system;
converting individual ones of the different items of the data set into a second programming language:
converting, by the programming language conversion system, a function signature of the item in the first programming language to the second programming language;
converting, by the programming language conversion system, one or more test statements of the item in the first programming language to the second programming language;
and
causing, by the programming language conversion system, a body of the converted function signature to be generated in the second programming language according to a prompt in the item used as input to a machine learning model trained to generate code in the second programming language;
and
storing the converted individual ones of the different items of the evaluation data set as part of a new evaluation data set.

5 more elements

One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:

receiving, via an interface of a programming language conversion system, an evaluation data set specified in a first programming language, wherein different items of the evaluation data set correspond to different respective evaluation tests for a code generation system;
converting individual ones of the different items of the data set into a second programming language:
converting, by the programming language conversion system, a function signature of the item in the first programming language to the second programming language;
converting, by the programming language conversion system, one or more test statements of the item in the first programming language to the second programming language;
and
causing, by the programming language conversion system, a body of the converted function signature to be generated in the second programming language according to a prompt in the item used as input to a machine learning model trained to generate code in the second programming language;
and
storing the converted individual ones of the different items of the evaluation data set as part of a new evaluation data set.

5 more elements

Dependencies are inferred from the claim text.