# US 2026/0086511 A1 · Fast attention mechanisms for physical systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0086511 A1
- Document ID: US20260086511A1
- Status: Published application
- Kind: A1
- Application number: 18/897,283
- Filed: 2024-09-26
- Published: 2026-03-26
- Earliest claimed: 2024-09-26
- Applicants: DeepMind Technologies Limited
- Inventors: Oliver Thorsten Unke; Jan Thorben Frank
- Web page: https://patentgrep.com/patent/US20260086511A1

## CPC

- G05B 13/027: {using neural networks only}
- G06N 3/045: Combinations of networks
- G05B 13/026: {using a predictor}
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G16C 20/50: Molecular design, e.g. of drugs
- G16C 20/70: Machine learning, data mining or chemometrics
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16C: COMPUTATIONAL CHEMISTRY; CHEMOINFORMATICS; COMPUTATIONAL MATERIALS SCIENCE

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a prediction characterizing a physical system. In one aspect, a method comprises: generating, for each of the plurality of objects in the physical system, a feature embedding for the object; generating, for each of the plurality of objects, a spatial encoding for the object representing the spatial location of the object, wherein: the spatial encoding for each object comprises a representation of a complex number characterizing a spatial relationship between the position vector for the object and a shared reference vector; and generating an embedding of the physical system by combining, for each of the plurality of objects, the feature embedding for the object with the spatial encoding for the object; and processing the embedding of the physical system to generate a prediction characterizing the physical system.

## Claims

### Claim 1

A method performed by one or more computers, comprising:
obtaining data characterizing a physical system, the data specifying, for each of a plurality objects in the physical system, a position vector representing a spatial location of the object and one or more physical properties of the object;
processing the data characterizing the physical system to generate an embedding of the physical system, comprising:
processing the data characterizing the physical system using an embedding neural network to generate, for each of the plurality of objects in the physical system, a feature embedding for the object representing the one or more physical properties of the object;
processing the data characterizing the physical system to determine, for each of the plurality of objects in the physical system, a spatial encoding for the object representing the spatial location of the object, wherein:
the spatial encoding for each object comprises a representation of a complex number characterizing a spatial relationship between the position vector for the object and a shared reference vector; and
generating the embedding of the physical system by combining, for each of the plurality of objects, the feature embedding for the object with the spatial encoding for the object;
processing the embedding of the physical system using an attention neural network to generate a network output characterizing a prediction for the physical system; and
providing the network output characterizing the prediction for the physical system.

### Claim 2 (depends on 1)

The method of claim 1, wherein the spatial encoding for each object comprises a representation of a complex number characterizing a dot product between the position vector for the object and a shared reference vector.

### Claim 3 (depends on 1)

The method of claim 1, wherein, for each pair of objects from the plurality of objects, an inner product of the spatial encodings for each of the pair of objects characterizes a distance between the pair of objects.

### Claim 4 (depends on 1)

The method of claim 1, wherein the spatial encoding for each object comprises a matrix representation of the complex number characterizing the spatial relationship between the position vector for the object and the shared reference vector.

### Claim 5 (depends on 1)

The method of claim 1, wherein the attention neural network is configured to process the embedding of the physical system by computing a respective attention operation for each of the plurality of objects within the physical system

### Claim 6 (depends on 5)

The method of claim 5, wherein, for each of the plurality of objects within the physical system, computing the respective attention operation for the object comprises:
generating an updated feature embedding for the object as a linear combination of value feature vectors for each of a plurality of other objects of the plurality of objects within the physical system associated with the attention operation for the object, wherein the value feature vector for each other object associated with the attention operation for the object:
(i) depends on the feature embedding for the other object; and
(ii) is scaled by an attention weight for the other object that depends on the spatial encoding for the object and the spatial encoding for the other object.

### Claim 7 (depends on 6)

The method of claim 6, wherein generating the updated feature embedding for the object as a linear combination of value feature vectors for each of the plurality of other objects within the physical system associated with the attention operation for the object comprises:
determining a combined key-value matrix for the plurality of objects within the physical system, wherein the combined key-value matrix represents a sum of outer products, comprising respective outer products of key feature vectors with the value feature vectors for each object within the physical system, wherein each key feature vector depends on a feature embedding for a corresponding object within the physical system; and
generating the updated feature embedding for the object by computing a product between a query feature vector for the object and the combined key-value matrix for the plurality of objects, wherein the query feature vector for the object depends on the feature embedding the object.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
processing the data characterizing the physical system to generate an embedding of the physical system, further comprises:
generating a plurality of shared reference vectors; and
for each of the plurality of shared reference vectors:
determining spatial encodings for each of the plurality of objects for the shared reference vector; and
processing the embedding of the physical system using the attention neural network to generate the network output characterizing the prediction for the physical system comprises:
for each of the plurality of objects:
for each of the plurality of shared reference vectors, generating a respective updated feature embedding for the object and for the shared reference vector that depends on the spatial encodings determined for the shared reference vector; and
generating an updated feature embedding for the object as a linear combination of the updated feature embeddings for the object for each of the plurality of shared reference vectors; and
generating the network output characterizing the prediction for the physical system by processing the updated feature embeddings for each of the plurality of objects.

### Claim 9 (depends on 8)

The method of claim 8, wherein generating the plurality of shared reference vectors comprises randomly sampling the plurality of shared reference vectors from a distribution of shared reference vectors.

### Claim 10 (depends on 8)

The method of claim 8, wherein generating the updated feature embedding for the object as a linear combination of the updated feature embeddings for the object for each of the plurality of shared reference vectors comprises:
generating the updated feature embedding for the object as a linear combination of the updated feature embeddings for the object for each of the plurality of shared reference vectors determined in accordance with a numerical integration with respect to the shared reference vectors.

### Claim 11 (depends on 10)

The method of claim 10, wherein generating the plurality of shared reference vectors comprises generating the plurality of shared reference vectors in accordance with the numerical integration with respect to the shared reference vectors.

### Claim 12 (depends on 10)

The method of claim 10, wherein the numerical integration comprises a Lebedev quadrature with respect to the shared reference vectors.

### Claim 13 (depends on 10)

The method of claim 10, wherein the linear combination of the updated feature embeddings for the object for each of the plurality of shared reference vectors comprises, for each of the plurality of shared reference vectors, the updated feature embeddings for the object and for the shared reference vector determined by a tensor product of the feature embeddings for the object and the value of one or more basis functions determined using the shared reference vector.

### Claim 14 (depends on 13)

The method of claim 13, wherein the basis function comprises a spherical harmonic basis function.

### Claim 15 (depends on 1)

The method of claim 1, wherein the data characterizing the physical system comprises data specifying, for each of the plurality objects in the physical system, a three-dimensional position vector of the object.

### Claim 16 (depends on 15)

The method of claim 15, wherein the physical system comprises a chemical system.

### Claim 17 (depends on 16)

The method of claim 16, wherein processing the embedding of the physical system using the attention neural network to generate the network output characterizing the prediction for the physical system comprises:
processing the embedding of the physical system using the attention neural network to generate a network output characterizing predicted energies for the plurality of objects in the physical system.

### Claim 18 (depends on 17)

The method of claim 17, wherein processing the embedding of the physical system using the attention neural network to generate the network output characterizing the prediction for the physical system comprises:
processing the embedding of the physical system using the attention neural network to generate a network output characterizing predicted inter-atomic forces for the physical system.

### Claim 19

A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
obtaining data characterizing a physical system, comprising data specifying, for each a plurality objects in the physical system, a position vector representing a spatial location of the object and one or more physical properties of the object;
processing the data characterizing the physical system to generate an embedding of the physical system, comprising:
processing the data characterizing the physical system using an embedding neural network to generate, for each of the plurality of objects in the physical system, a feature embedding for the object representing the one or more physical properties of the object;
processing the data characterizing the physical system to determine, for each of the plurality of objects in the physical system, a spatial encoding for the object representing the spatial location of the object, wherein:
the spatial encoding for each object comprises a representation of a complex number characterizing a spatial relationship between the position vector for the object and a shared reference vector; and
generating the embedding of the physical system by combining, for each of the plurality of objects, the feature embedding for the object with the spatial encoding for the object;
processing the embedding of the physical system using an attention neural network to generate a network output characterizing a prediction for the physical system; and
providing the network output characterizing the prediction for the physical system.

### Claim 20

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
obtaining data characterizing a physical system, comprising data specifying, for each a plurality objects in the physical system, a position vector representing a spatial location of the object and one or more physical properties of the object;
processing the data characterizing the physical system to generate an embedding of the physical system, comprising:
processing the data characterizing the physical system using an embedding neural network to generate, for each of the plurality of objects in the physical system, a feature embedding for the object representing the one or more physical properties of the object;
processing the data characterizing the physical system to determine, for each of the plurality of objects in the physical system, a spatial encoding for the object representing the spatial location of the object, wherein:
the spatial encoding for each object comprises a representation of a complex number characterizing a spatial relationship between the position vector for the object and a shared reference vector; and
generating the embedding of the physical system by combining, for each of the plurality of objects, the feature embedding for the object with the spatial encoding for the object;
processing the embedding of the physical system using an attention neural network to generate a network output characterizing a prediction for the physical system; and
providing the network output characterizing the prediction for the physical system.
