Systems and methods for code understanding and generation

A system for programming language (PL) generation and understanding using a neural network model, the system comprising:

• a memory storing the neural network model comprising an encoder and a decoder, and a plurality of processor-executable instructions;
• and
• a processor executing the plurality of processor-executable instructions to perform operations comprising:
• generating, by the neural network model, a first predicted output in response to a bimodal input sequence comprising a PL segment and a natural language (NL) segment according to a first pre-training task, wherein the first predicted output is generated by:
• masking a plurality of identifiers in the PL segment of the bimodal input sequence to generate a masked bimodal input sequence, wherein a designated mask token is used for a specific identifier of the plurality of identifiers,
• encoding, by an encoder of the neural network model, the masked bimodal input sequence into a first representation, and
• generating, by a decoder of the neural network model, a target sequence comprising the masked plurality of identifiers and corresponding designated mask tokens, from the first representation to generate the first predicted output,
• wherein the PL segment and the NL segment belong to an unlabeled code sample;
• computing a first training objective based on the first predicted output according to the first pre-training task;
• generating, by the neural network model, a second predicted output in response to the bimodal input sequence according to a second pre-training task;
• computing a second training objective based on the second predicted output according to the second pre-training task;
• and
• alternately or jointly updating the neural network model based on the first training objective and the second training objective.

A non-transitory machine-readable medium storing a plurality of processor-executable instructions for programming language (PL) generation and understanding using an encoder-decoder model, the plurality of processor-executable instructions being executed by one or more hardware processors to perform operations comprising:

• generating, by the neural network model, a first predicted output in response to a bimodal input sequence comprising a PL segment and a natural language (NL) segment according to a first pre-training task, wherein the first predicted output is generated by:
• masking a plurality of identifiers in the PL segment of the bimodal input sequence to generate a masked bimodal input sequence, wherein a designated mask token is used for a specific identifier of the plurality of identifiers,
• encoding, by an encoder of the neural network model, the masked bimodal input sequence into a first representation, and
• generating, by a decoder of the neural network model, a target sequence comprising the masked plurality of identifiers and corresponding designated mask tokens, from the first representation to generate the first predicted output,
• wherein the PL segment and the NL segment belong to an unlabeled code sample;
• computing a first training objective based on the first predicted output according to the first pre-training task;
• generating, by the neural network model, a second predicted output in response to the bimodal input sequence according to a second pre-training task;
• computing a second training objective based on the second predicted output according to the second pre-training task;
• and
• alternately or jointly updating the neural network model based on the first training objective and the second training objective.