Systems and methods for a testing pipeline of code generation artificial intelligence agents
AssigneeSalesforce, Inc.
InventorsJiacheng Xu, Bo Pang, Jin Qu, Hiroaki Hayashi, Yingbo Zhou, Caiming Xiong
Life of this patent
Estimating the term…
Loading patent…
AssigneeSalesforce, Inc.
InventorsJiacheng Xu, Bo Pang, Jin Qu, Hiroaki Hayashi, Yingbo Zhou, Caiming Xiong
Estimating the term…
Embodiments described herein provide a testing pipelines for LLM-based software engineering agents. For example, the testing pipeline comprise a variety of tasks to evaluate a software engineering agent's abilities in crafting test cases under specific requirements, ranging from relatively simple ones, such as completing assertion statements, to more complex challenges, like writing test cases to cover specific code blocks across up to ten files. The testing pipeline comprises a data collection module that collects test programs, and a testing environment construction module that run the agent-generated programs on different repositories.
A computer-implemented method of operating a structured testing pipeline for validating a code generation artificial intelligence (AI) agent, the method comprising:
A system for operating a structured testing pipeline for validating a code generation artificial intelligence (AI) agent, the system comprising:
A non-statutory processor-readable medium storing a plurality of processor-executable instructions for operating a structured testing pipeline for validating a code generation artificial intelligence (AI) agent, the instructions being executed by one or more processors to perform operations comprising:
Dependencies are inferred from the claim text.