# US 2023/0410938 A1 · Predicting protein structures using protein graphs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410938 A1
- Document ID: US20230410938A1
- Status: Published application
- Kind: A1
- Application number: 18/034,989
- Filed: 2021-11-23
- Published: 2023-12-21
- Earliest claimed: 2020-11-28
- Assignees: DeepMind Technologies Limited
- Applicants: DeepMind Technologies Limited
- Inventors: Alexander Pritzel; Mikhail Figurnov; John Jumper
- Web page: https://patentgrep.com/patent/US20230410938A1

## CPC

- G16B 15/20: Protein or domain folding
- G06N 3/045: Combinations of networks
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G16B 15/30: Drug targeting using structural data; Docking or binding prediction
- G16B 30/10: Sequence alignment; Homology search
- G16B 40/20: Supervised data analysis
- G06N 3/02: Neural networks
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G16B 30/00: ICT specially adapted for sequence analysis involving nucleotides or amino acids
- G16B: BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining a predicted structure of a protein. According to one aspect, there is provided a method comprising maintaining graph data representing a graph of the protein; obtaining a respective pair embedding for each edge in the graph; processing the pair embeddings using a sequence of update blocks, wherein each update block performs operations comprising, for each edge in the graph: generating a respective representation of each of a plurality of cycles in the graph that include the edge by, for each cycle, processing embeddings for edges in the cycle in accordance with the values of the update block parameters of the update block to generate the representation of the cycle; and updating the pair embedding for the edge using the representations of the cycles in the graph that include the edge.

## Claims

### Claim 1

A method performed by one or more data processing apparatus for determining a predicted structure of a protein, the method comprising:
maintaining graph data representing a graph of the protein, wherein the graph comprises a set of nodes and a set of edges, wherein the set of nodes comprises a plurality of amino acid nodes that each represent a respective amino acid in the protein, and wherein the set of edges comprises a respective edge connecting each pair of amino acid nodes in graph;
obtaining a respective pair embedding for each edge in the graph that connects a pair of amino acid nodes;
processing an input comprising the pair embeddings using an embedding neural network, wherein the embedding neural network comprises a sequence of update blocks and uses the update blocks to repeatedly update the pair embeddings,
wherein each update block has a plurality of update block parameters and performs operations comprising:
receiving the pair embeddings;
updating the pair embeddings in accordance with values of the update block parameters of the update block, comprising, for each edge in the graph that connects a pair of amino acid nodes:
generating a respective representation of each of a plurality of cycles in the graph that include the edge by, for each cycle, processing embeddings for edges in the cycle in accordance with the values of the update block parameters of the update block to generate the representation of the cycle; and
updating the pair embedding for the edge using the representations of the cycles in the graph that include the edge; and
after processing the pair embeddings using the embedding neural network, determining the predicted structure of the protein based on the pair embeddings.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating a respective representation of each of a plurality of cycles in the graph that include the edge comprises generating a respective representation of every cycle in the graph that includes the edge and that has a predefined length.

### Claim 3 (depends on 2)

The method of claim 2, wherein the predefined length is three.

### Claim 4 (depends on 1)

The method of claim 1, wherein updating the pair embedding for the edge using the representations of the cycles in the graph that include the edge comprises:
processing the pair embedding for the edge and the representations of the cycles in the graph that include the edge, in accordance with the values of the update block parameters of the update block, to generate a residual embedding; and
adding the residual embedding to the pair embedding for the edge.

### Claim 5 (depends on 4)

The method of claim 4, wherein processing the pair embedding for the edge and the representations of the cycles in the graph that include the edge, in accordance with the values of the update block parameters of the update block, to generate a residual embedding comprises:
summing the representations of the cycles in the graph that include the edge; and
processing the pair embedding for the edge and the sum of the representations of the cycles in the graph that include the edge using one or more neural network layers to generate the residual embedding.

### Claim 6 (depends on 1)

The method of claim 1, wherein each update block receives a multiple sequence alignment (MSA) representation for the protein that represents a respective MSA corresponding to each amino acid chain in the protein; and
wherein the pair embedding for each edge is updated based at least in part on the MSA representation.

### Claim 7 (depends on 6)

The method of claim 6, wherein set of nodes of the graph comprises a plurality of multiple sequence alignment (MSA) sequence nodes that each represent a respective MSA sequence corresponding to an amino acid chain in the protein,
wherein the set of edges of the graph comprises a respective edge connecting each MSA sequence node in the graph to each amino acid node in the graph, and
wherein the MSA representation for the protein comprises a respective embedding for each edge in the graph that connects a MSA sequence node to an amino acid node.

### Claim 8 (depends on 7)

The method of claim 7, wherein for each edge in the graph that connects a pair of amino acid nodes, generating the respective representation of each of the plurality of cycles in the graph that include the edge comprises generating a respective representation of each of one or more cycles in the graph that include an edge that connects a MSA sequence node to an amino acid node.

### Claim 9 (depends on 6)

The method of claim 6, wherein each update block further performs operations comprising:
applying a transformation operation to the MSA representation; and
updating the pair embeddings by adding a result of the transformation operation to the pair embeddings.

### Claim 10 (depends on 9)

The method of claim 9, wherein the transformation operation comprises an outer product mean operation.

### Claim 11 (depends on 6)

The method of claim 6, wherein each update block further performs operations comprising:
updating the MSA representation based on the pair embeddings.

### Claim 12 (depends on 11)

The method of claim 11, wherein updating the MSA representation based on the pair embeddings comprises:
updating the MSA representation using attention over embeddings in the MSA representation, wherein the attention is conditioned on the pair embeddings.

### Claim 13 (depends on 12)

The method of claim 12, wherein updating the MSA representation using attention over the embeddings in the MSA representation comprises:
generating, based on the MSA representation, a plurality of attention weights;
generating, based on the pair embeddings, a respective attention bias corresponding to each of the attention weights;
generating a plurality of biased attention weights based on the attention weights and the attention biases; and
updating the embeddings in the MSA representation using attention over the embeddings in the MSA representation based on the biased attention weights.

### Claim 14 (depends on 13)

The method of claim 13, wherein updating the embeddings in the MSA representation using attention based on the biased attention weights comprises, for each embedding in the MSA representation:
updating the embedding, based on the biased attention weights, using attention over only embeddings in the MSA representation that are located in a same row as the embedding in an arrangement of the embeddings in the MSA representation into a two-dimensional array.

### Claim 15

A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for determining a predicted structure of a protein, the operations comprising:
maintaining graph data representing a graph of the protein, wherein the graph comprises a set of nodes and a set of edges, wherein the set of nodes comprises a plurality of amino acid nodes that each represent a respective amino acid in the protein, and wherein the set of edges comprises a respective edge connecting each pair of amino acid nodes in graph;
obtaining a respective pair embedding for each edge in the graph that connects a pair of amino acid nodes;
processing an input comprising the pair embeddings using an embedding neural network, wherein the embedding neural network comprises a sequence of update blocks and uses the update blocks to repeatedly update the pair embeddings,
wherein each update block has a plurality of update block parameters and performs operations comprising:
receiving the pair embeddings;
updating the pair embeddings in accordance with values of the update block parameters of the update block, comprising, for each edge in the graph that connects a pair of amino acid nodes:
generating a respective representation of each of a plurality of cycles in the graph that include the edge by, for each cycle, processing embeddings for edges in the cycle in accordance with the values of the update block parameters of the update block to generate the representation of the cycle; and
updating the pair embedding for the edge using the representations of the cycles in the graph that include the edge; and
after processing the pair embeddings using the embedding neural network, determining the predicted structure of the protein based on the pair embeddings.

### Claim 16

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for determining a predicted structure of a protein, the operations comprising:
maintaining graph data representing a graph of the protein, wherein the graph comprises a set of nodes and a set of edges, wherein the set of nodes comprises a plurality of amino acid nodes that each represent a respective amino acid in the protein, and wherein the set of edges comprises a respective edge connecting each pair of amino acid nodes in graph;
obtaining a respective pair embedding for each edge in the graph that connects a pair of amino acid nodes;
processing an input comprising the pair embeddings using an embedding neural network, wherein the embedding neural network comprises a sequence of update blocks and uses the update blocks to repeatedly update the pair embeddings,
wherein each update block has a plurality of update block parameters and performs operations comprising:
receiving the pair embeddings;
updating the pair embeddings in accordance with values of the update block parameters of the update block, comprising, for each edge in the graph that connects a pair of amino acid nodes:
generating a respective representation of each of a plurality of cycles in the graph that include the edge by, for each cycle, processing embeddings for edges in the cycle in accordance with the values of the update block parameters of the update block to generate the representation of the cycle; and
updating the pair embedding for the edge using the representations of the cycles in the graph that include the edge; and
after processing the pair embeddings using the embedding neural network, determining the predicted structure of the protein based on the pair embeddings.

### Claim 17

(canceled)

### Claim 18

(canceled)

### Claim 19

(canceled)

### Claim 20

(canceled)

### Claim 21

(canceled)

### Claim 22

(canceled)

### Claim 23

(canceled)

### Claim 24

(canceled)

### Claim 25

(canceled)

### Claim 26

(canceled)

### Claim 27

(canceled)

### Claim 28 (depends on 16)

The non-transitory computer storage media of claim 16, wherein generating a respective representation of each of a plurality of cycles in the graph that include the edge comprises generating a respective representation of every cycle in the graph that includes the edge and that has a predefined length.

### Claim 29 (depends on 28)

The non-transitory computer storage media of claim 28, wherein the predefined length is three.

### Claim 30 (depends on 16)

The non-transitory computer storage media of claim 16, wherein updating the pair embedding for the edge using the representations of the cycles in the graph that include the edge comprises:
processing the pair embedding for the edge and the representations of the cycles in the graph that include the edge, in accordance with the values of the update block parameters of the update block, to generate a residual embedding; and
adding the residual embedding to the pair embedding for the edge.

### Claim 31 (depends on 30)

The non-transitory computer storage media of claim 30, wherein processing the pair embedding for the edge and the representations of the cycles in the graph that include the edge, in accordance with the values of the update block parameters of the update block, to generate a residual embedding comprises:
summing the representations of the cycles in the graph that include the edge; and
processing the pair embedding for the edge and the sum of the representations of the cycles in the graph that include the edge using one or more neural network layers to generate the residual embedding.
