# US 12,217,033 B2 · Systems and methods for code understanding and generation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,033 B2 - Document ID: US12217033B2 - Status: Granted - Kind: B2 - Application number: 18/475,103 - Filed: 2023-09-26 - Published: 2024-01-18 - Granted: 2025-02-04 - Earliest claimed: 2021-05-18 - Assignees: Salesforce, Inc. - Applicants: Salesforce, Inc. - Inventors: Yue Wang; Weishi Wang; Shafiq Rayhan Joty; Chu Hong Hoi - Web page: https://patentgrep.com/patent/US12217033B2 ## CPC - G06F 8/30: Creation or generation of source code - G06F 8/427: {Parsing} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30) - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/047: Probabilistic or stochastic networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 3/096: Transfer learning - G06F 40/216: using statistical methods - G06F 40/30: Semantic analysis - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Embodiments described herein a code generation and understanding model that builds on a Transformer-based encoder-decoder framework. The code generation and understanding model is configured to derive generic representations for programming language (PL) and natural language (NL) in code domain via pre-training on unlabeled code corpus, and then to benefit many code-related downstream tasks with fine-tuning. Apart from the denoising sequence-to-sequence objectives widely adopted for pre-training on natural language, identifier tagging and prediction pre-training objective is adopted to enable the model to better leverage the crucial token type information from PL, which specifically are the identifiers assigned by developers. ## Claims ### Claim 1 A method for programming language (PL) generation and understanding using an encoder-decoder model, the method comprising: generating, by a neural network model, a first predicted output in response to a bimodal input sequence comprising a PL segment and a natural language (NL) segment according to a first pre-training task, wherein the first predicted output is generated by: masking a plurality of identifiers in the PL segment of the bimodal input sequence to generate a masked bimodal input sequence, wherein a designated mask token is used for a specific identifier of the plurality of identifiers, encoding, by an encoder of the neural network model, the masked bimodal input sequence into a first representation, and generating, by a decoder of the neural network model, a target sequence comprising the masked plurality of identifiers and corresponding designated mask tokens, from the first representation to generate the first predicted output wherein the PL segment and the NL segment belong to an unlabeled code sample; computing a first training objective based on the first predicted output according to the first pre-training task; generating, by the neural network model, a second predicted output in response to the bimodal input sequence according to a second pre-training task; computing a second training objective based on the second predicted output according to the second pre-training task; and alternately or jointly updating the neural network model based on the first training objective and the second training objective. ### Claim 2 (depends on 1) The method of claim 1, wherein the first predicted output is generated by: randomly masking a plurality of spans in the bimodal input sequence; and generating, by the decoder, a predicted masked sequence conditioned on the masked bimodal input sequence and a set of previously predicted masked sequences up to a current time instance. ### Claim 3 (depends on 2) The method of claim 2, wherein the first training objective is computed based on a conditional probability distribution of the masked bimodal input sequence and the set of previously predicted masked sequences up to the current time instance. ### Claim 4 (depends on 1) The method of claim 1 wherein the first training objective is computed based on a conditional probability of each token in the target sequence conditioned on the masked bimodal input sequence and tokens previously generated for the target sequence up to a respective token. ### Claim 5 (depends on 1) The method of claim 1, wherein the first predicted output is generated by: converting the PL segment into an Abstract Syntax Tree (AST); constructing, based on the AST, a sequence of binary labels, each binary label representing whether a respective code token in the PL segment is an identifier; and mapping final hidden states corresponding to the PL segment in the encoder into a sequence of probabilities. ### Claim 6 (depends on 5) The method of claim 5, wherein the first training objective is computed based on a binary cross-entropy loss between the sequence of probabilities and the sequence of binary labels. ### Claim 7 (depends on 1) The method of claim 1, wherein the second pre-training task is different from the first pre-training task. ### Claim 8 (depends on 1) The method of claim 1, further comprising: receiving one or more training datasets for refining the neural network model on multiple tasks at a time; prepending an input training sequence with a task control code indicating a type of a task corresponding to a specific training dataset; and generating, by the neural network model, a predicted output in response to the input training sequence according to the task control code. ### Claim 9 (depends on 8) The method of claim 8, wherein the input training sequence is sampled from the one or more training datasets according to a multinomial distribution computed based on respective sizes of the one or more training datasets. ### Claim 10 A system for programming language (PL) generation and understanding using a neural network model, the system comprising: a memory storing the neural network model comprising an encoder and a decoder, and a plurality of processor-executable instructions; and a processor executing the plurality of processor-executable instructions to perform operations comprising: generating, by the neural network model, a first predicted output in response to a bimodal input sequence comprising a PL segment and a natural language (NL) segment according to a first pre-training task, wherein the first predicted output is generated by: masking a plurality of identifiers in the PL segment of the bimodal input sequence to generate a masked bimodal input sequence, wherein a designated mask token is used for a specific identifier of the plurality of identifiers, encoding, by an encoder of the neural network model, the masked bimodal input sequence into a first representation, and generating, by a decoder of the neural network model, a target sequence comprising the masked plurality of identifiers and corresponding designated mask tokens, from the first representation to generate the first predicted output, wherein the PL segment and the NL segment belong to an unlabeled code sample; computing a first training objective based on the first predicted output according to the first pre-training task; generating, by the neural network model, a second predicted output in response to the bimodal input sequence according to a second pre-training task; computing a second training objective based on the second predicted output according to the second pre-training task; and alternately or jointly updating the neural network model based on the first training objective and the second training objective. ### Claim 11 (depends on 10) The system of claim 10, wherein the first predicted output is generated by: randomly masking a plurality of spans in the bimodal input sequence; and generating, by the decoder, a predicted masked sequence conditioned on the masked bimodal input sequence and a set of previously predicted masked sequences up to a current time instance. ### Claim 12 (depends on 11) The system of claim 11, wherein the first training objective is computed based on a conditional probability distribution of the masked bimodal input sequence and the set of previously predicted masked sequences up to the current time instance. ### Claim 13 (depends on 10) The system of claim 10, wherein the first training objective is computed based on a conditional probability of each token in the target sequence conditioned on the masked bimodal input sequence and tokens previously generated for the target sequence up to a respective token. ### Claim 14 (depends on 10) The system of claim 10, wherein the first predicted output is generated by: converting the PL segment into an Abstract Syntax Tree (AST); constructing, based on the AST, a sequence of binary labels, each binary label representing whether a respective code token in the PL segment is an identifier; and mapping final hidden states corresponding to the PL segment in the encoder into a sequence of probabilities. ### Claim 15 (depends on 14) The system of claim 14, wherein the first training objective is computed based on a binary cross-entropy loss between the sequence of probabilities and the sequence of binary labels. ### Claim 16 (depends on 10) The system of claim 10, wherein the second pre-training task is different from the first pre-training task. ### Claim 17 (depends on 10) The system of claim 10, wherein the operations further comprise: receiving one or more training datasets for refining the neural network model on multiple tasks at a time; prepending an input training sequence with a task control code indicating a type of a task corresponding to a specific training dataset; and generating, by the neural network model, a predicted output in response to the input training sequence according to the task control code. ### Claim 18 A non-transitory machine-readable medium storing a plurality of processor-executable instructions for programming language (PL) generation and understanding using an encoder-decoder model, the plurality of processor-executable instructions being executed by one or more hardware processors to perform operations comprising: generating, by the neural network model, a first predicted output in response to a bimodal input sequence comprising a PL segment and a natural language (NL) segment according to a first pre-training task, wherein the first predicted output is generated by: masking a plurality of identifiers in the PL segment of the bimodal input sequence to generate a masked bimodal input sequence, wherein a designated mask token is used for a specific identifier of the plurality of identifiers, encoding, by an encoder of the neural network model, the masked bimodal input sequence into a first representation, and generating, by a decoder of the neural network model, a target sequence comprising the masked plurality of identifiers and corresponding designated mask tokens, from the first representation to generate the first predicted output, wherein the PL segment and the NL segment belong to an unlabeled code sample; computing a first training objective based on the first predicted output according to the first pre-training task; generating, by the neural network model, a second predicted output in response to the bimodal input sequence according to a second pre-training task; computing a second training objective based on the second predicted output according to the second pre-training task; and alternately or jointly updating the neural network model based on the first training objective and the second training objective.