# US 2021/0240453 A1 · Generating and using joint representations of source code

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0240453 A1
- Document ID: US20210240453A1
- Status: Published application
- Kind: A1
- Application number: 16/781,344
- Filed: 2020-02-04
- Published: 2021-08-05
- Earliest claimed: 2020-02-04
- Assignees: X Development LLC
- Applicants: X Development LLC
- Inventors: Rohan Badlani; Owen Lewis; Georgios Evangelopoulos; Olivia Hatalsky; Bin Ni
- Web page: https://patentgrep.com/patent/US20210240453A1

## CPC

- G06F 8/36: Software reuse
- G06F 8/42: {Syntactic analysis}
- G06F 8/38: for implementing user interfaces
- G06F 8/73: Program documentation
- G06F 8/75: Structural analysis for program understanding
- G06F 8/77: Software metrics
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Implementations are described herein for generating embeddings of source code using both the language and graph domains, and leveraging combinations of these semantically-rich and structurally-informative embeddings for various purposes. In various implementations, tokens of a source code snippet may be applied as input across a sequence-processing machine learning model to generate a plurality of token embeddings. A graph may also be generated based on the source code snippet. A joint representation may be generated based on the graph and the incorporated token embeddings. The joint representation generated from the source code snippet may be compared to one or more other joint representations generated from one or more other source code snippets to make a determination about the source code snippet.

## Claims

### Claim 1

A method implemented using one or more processors, comprising:
applying tokens of a source code snippet as input across a sequence-processing machine learning model to generate a plurality of token embeddings;
generating a graph based on the source code snippet;
augmenting the graph by infusing one or more nodes of the graph that correspond to one or more of the tokens of the source code snippet with one or more corresponding token embeddings;
processing the augmented graph using a graph neural network to generate a joint representation based on the augmented graph infused with the token embeddings; and
calculating one or more distances in latent space between the joint representation generated from the source code snippet and one or more other joint representations generated from one or more other source code snippets to make a determination about the source code snippet.

### Claim at position 2

2-4. (canceled)

### Claim 5 (depends on 1)

The method of claim 1, wherein the joint representation comprises a mean over an aggregate embedding generated from the graph.

### Claim 6 (depends on 1)

The method of claim 1, wherein the calculating is performed as part of a search for other source code similar to the source code snippet.

### Claim 7 (depends on 1)

The method of claim 1, wherein the calculating is performed to automatically generate a comment associated with the source code snippet.

### Claim 8 (depends on 1)

The method of claim 1, wherein the determination about the source code snippet comprises a source code quality score.

### Claim 9 (depends on 1)

The method of claim 1, wherein the graph comprises an abstract syntax tree.

### Claim 10 (depends on 1)

The method of claim 1, wherein the sequence-processing machine learning model comprises a neural network that includes a self attention mechanism.

### Claim 11 (depends on 10)

The method of claim 10, wherein the neural network comprises a transformer network.

### Claim 12

A system comprising one or more processors and memory storing instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to:
apply tokens of a source code snippet as input across a sequence-processing machine learning model to generate a plurality of token embeddings;
generate a graph based on the source code snippet;
augment the graph by infusing one or more nodes of the graph that correspond to one or more of the tokens of the source code snippet with one or more corresponding token embeddings;
processing the augmented graph using a graph neural network to generate a joint representation based on the augmented graph infused with the token embeddings; and
calculate one or more distances in latent space between the joint representation generated from the source code snippet and one or more other joint representations generated from one or more other source code snippets to make a determination about the source code snippet.

### Claim at position 11

13-15. (canceled)

### Claim 16 (depends on 12)

The system of claim 12, wherein the determination comprises a similarity between the source code snippet and other source code.

### Claim 17 (depends on 12)

The system of claim 12, wherein the determination comprises an automatically-generated comment associated with the source code snippet.

### Claim 18 (depends on 12)

The system of claim 12, wherein the determination about the source code snippet comprises a source code quality score.

### Claim 19 (depends on 12)

The system of claim 12, wherein the sequence-processing machine learning model comprises a transformer network.

### Claim 20

At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to:
apply tokens of a source code snippet as input across a sequence-processing machine learning model to generate a plurality of token embeddings;
generate a graph based on the source code snippet;
augment the graph by infusing one or more nodes of the graph that correspond to one or more of the tokens of the source code snippet with one or more corresponding token embeddings;
processing the augmented graph using a graph neural network to generate a joint representation based on the augmented graph infused with the token embeddings; and
calculate one or more distances in latent space between the joint representation generated from the source code snippet and one or more other joint representations generated from one or more other source code snippets to make a determination about the source code snippet.
