# US 12,008,341 B2 · Systems and methods for generating natural language using language models trained on computer code > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,341 B2 - Document ID: US12008341B2 - Status: Granted - Kind: B2 - Application number: 18/321,921 - Filed: 2023-05-23 - Published: 2024-01-18 - Granted: 2024-06-11 - Earliest claimed: 2022-07-14 - Assignees: OpenAI OpCo, LLC - Applicants: OpenAI Opco, LLC - Inventors: Mark Chen; Jaroslaw Tworek; Ilya Sutskever; Wojciech Zaremba; Hee Woo Jun; Henrique Ponde De Oliveira Pinto - Web page: https://patentgrep.com/patent/US12008341B2 ## CPC - G06F 8/30: Creation or generation of source code - G06F 8/33: Intelligent editors - G06F 8/73: Program documentation - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/30: Semantic analysis - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06F 11/3684: {for test design, e.g. generating new test cases} - G06F 40/186: Templates - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Disclosed herein are methods, systems, and computer-readable media for generating natural language based on computer code input. In an embodiment, a method may comprise one or more of: accessing a docstring generation model configured to generate docstrings from computer code; receiving one or more computer code samples; generating, using the docstring generation model and based on the received one or more computer code samples, one or more candidate docstrings representing natural language text, each of the one or more candidate docstrings being associated with at least a portion of the one or more computer code samples; identifying at least one of the one or more candidate docstrings that provides an intent of the at least a portion of the one or more computer code samples; and/or outputting, via a user interface, the at least one identified docstring with the at least a portion of the one or more computer code samples. ## Claims ### Claim 1 A computer-implemented method, comprising: training a machine learning model to generate natural language docstrings from computer code; receiving one or more computer code samples at the trained machine learning model; generating, via the trained machine learning model and based on the received one or more computer code samples, one or more candidate natural language docstrings representing natural language text, each of the one or more candidate natural language docstrings being associated with at least a portion of the one or more computer code samples; identifying at least one of the one or more candidate natural language docstrings that provides an intent of the at least a portion of the one or more computer code samples; outputting from the trained machine learning model the at least one identified natural language docstring with the at least a portion of the one or more computer code samples; and receiving, at the machine learning model, a selection of the one or more computer code samples, wherein the machine learning model provides an automatic description of the selection and generates a template for building an additional machine learning model. ### Claim 2 (depends on 1) The method of claim 1, wherein the machine learning model further generates a similarity between the intent and an additional natural language text. ### Claim 3 (depends on 2) The method of claim 2, wherein the machine learning model is further trained using the outputted at least one identified natural language docstring in association with the at least a portion of the one or more computer code samples. ### Claim 4 (depends on 1) The method of claim 1, wherein the machine learning model is trained using concatenated strings, each concatenated string comprising at least two of a function signature, a reference solution, or a docstring. ### Claim 5 (depends on 1) The method of claim 1, wherein identifying at least one of the one or more candidate natural language docstrings is based on a correctness score computed for at least one natural language candidate docstring, the correctness score indicating a proportion of correctly classified instances compared to a total number of candidate natural language docstrings generated. ### Claim 6 (depends on 1) The method of claim 1, further comprising verifying each of the one or more natural language candidate docstrings, wherein verifying includes determining a correctness score for each of the one or more candidate natural language docstrings, wherein the identifying at least one of the one or more candidate natural language docstrings is based on the determined correctness score. ### Claim 7 (depends on 6) The method of claim 6, wherein the machine learning model is fine-tuned based on verified candidate natural language docstrings. ### Claim 8 (depends on 6) The method of claim 6, further comprising ranking the one or more candidate natural language docstrings based on the determined correctness score, wherein identifying one of the one or more candidate natural language docstrings is based on selecting a top-k candidate natural language docstring. ### Claim 9 (depends on 1) The method of claim 1, wherein the at least one identified natural language docstring and the at least a portion of the one or more computer code samples are output via an application programming interface (API). ### Claim 10 (depends on 1) The method of claim 1, wherein the trained machine learning model has between 10 billion and 14 billion parameters. ### Claim 11 (depends on 1) The method of claim 1, wherein the trained machine learning model comprises a plurality of layers, at least one of the layers having a transformer decoder architecture. ### Claim 12 (depends on 1) The method of claim 1, wherein the machine learning model further suggests a change to improve existing code within the received one or more computer code samples. ### Claim 13 (depends on 1) The method of claim 1, wherein the machine learning model is fine-tuned based on at least one public web source or software repository. ### Claim 14 (depends on 13) The method of claim 13, wherein the machine learning model is fine-tuned based on a set of training data constructed from examples within the at least one public web source or software repository. ### Claim 15 (depends on 1) The method of claim 1, wherein identifying at least one of the one or more candidate natural language docstrings is further based on a mean-log probability. ### Claim 16 (depends on 1) The method of claim 1, wherein the machine learning model is developed by applying training data comprising annotated computer code to a precursor model comprising a machine learning model trained on natural language prom pts. ### Claim 17 (depends on 1) The method of claim 1, further comprising training a machine learning model used for generating computer code based on natural language input using training data comprising the outputted at least one identified natural language docstring in association with the at least a portion of the one or more computer code samples. ### Claim 18 A system comprising: at least one memory storing instructions; at least one processor configured to execute the instructions to perform operations comprising: training a machine learning model to generate natural language docstrings from computer code; receiving one or more computer code samples at the trained machine learning model; generating, via the trained machine learning model and based on the received one or more computer code samples, one or more candidate natural language docstrings representing natural language text, each of the one or more candidate natural language docstrings being associated with at least a portion of the one or more computer code samples; identifying at least one of the one or more candidate natural language docstrings that provides an intent of the at least a portion of the one or more computer code samples; outputting from the trained machine learning model the at least one identified natural language docstring with the at least a portion of the one or more computer code samples; and receiving, at the machine learning model, a selection of the one or more computer code samples, wherein the machine learning model provides an automatic description of the selection and generates a template for building an additional machine learning model. ### Claim 19 A non-transitory computer-readable medium including instructions that are executable by one or more processors to perform operations comprising: training a machine learning model to generate natural language docstrings from computer code; receiving one or more computer code samples at the trained machine learning model; generating, via the trained machine learning model and based on the received one or more computer code samples, one or more candidate natural language docstrings representing natural language text, each of the one or more candidate natural language docstrings being associated with at least a portion of the one or more computer code samples; identifying at least one of the one or more candidate natural language docstrings that provides an intent of the at least a portion of the one or more computer code samples; outputting from the trained machine learning model the at least one identified natural language docstring with the at least a portion of the one or more computer code samples; and receiving, at the machine learning model, a selection of the one or more computer code samples, wherein the machine learning model provides an automatic description of the selection and generates a template for building an additional machine learning model.