Large language model response optimization for data pipeline generation

A computer-implemented method for generating a data pipeline using a large language model, the method comprising:

• receiving, via a user interface, a natural language query and a selection of a plurality of data sets for generating a data pipeline;
• generating a prompt comprising at least:
• the natural language query, indications of the plurality of data sets, an indication of a format of a first computer language, and an indication of available data transformations;
• transmitting the prompt to a large language model (“LLM”);
• receiving, from the LLM, a response to the prompt in the format of the first computer language;
• parsing the response in the first computer language to identify at least an indication of one or more recommended data transformations to be performed in the data pipeline;
• and
• generating, based on the indication of the one or more recommended data transformations, the data pipeline using a second computer language.