# US 2026/0073143 A1 · Multi-modal hierarchical tokenization deep neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0073143 A1
- Document ID: US20260073143A1
- Status: Published application
- Kind: A1
- Application number: 19/392,684
- Filed: 2025-11-18
- Published: 2026-03-12
- Earliest claimed: 2024-08-23
- Assignees: D. E. Shaw Research, LLC
- Applicants: D. E. Shaw Research, LLC
- Inventors: Paul Maragakis; Andreas Kraemer; James P. Roney; Peter Skopp
- Web page: https://patentgrep.com/patent/US20260073143A1

## CPC

- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G16C 20/70: Machine learning, data mining or chemometrics
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G16C 20/30: Prediction of properties of chemical compounds, compositions or mixtures
- G16C 20/50: Molecular design, e.g. of drugs
- G16B 15/30: Drug targeting using structural data; Docking or binding prediction
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G16C: COMPUTATIONAL CHEMISTRY; CHEMOINFORMATICS; COMPUTATIONAL MATERIALS SCIENCE
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16B: BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY

## Abstract

A system is disclosed for encoding a data string of a first modality into a hierarchical tokenized representation for processing by a text-based deep neural network (DNN) trained on a second modality. The data string comprises multiple units, each having one or more attributes. Each attribute is represented in the tokenized string as a sequence of hierarchical tokens, with a first hierarchical token encoding one or more most significant bits and a subsequent hierarchical token encoding one or more less significant bits. The DNN processes the data string bidirectionally, across the sequence of units and within the token hierarchy, to select tokens that capture attribute information. The selected hierarchical tokens output by the DNN from a representation of the original data string that preserves attribute detail while enabling cross-modal processing using models trained on text.

## Claims

### Claim 1

A computer-implemented method, comprising:
encoding a data string into a tokenized string, wherein the data string comprises a plurality of units, wherein each unit is encoded in the tokenized string as a plurality of hierarchical tokens representing one or more attributes of the unit, wherein a first hierarchical token in a hierarchy represents one or more most significant bits (MSBs) of an attribute of the unit and a second hierarchical token represents one or more less significant bits (LSBs) of the attribute instantiating a deep neural network (DNN);
applying the DNN to the data string to select the hierarchical tokens in the tokenized string, wherein selecting the hierarchical tokens is performed according to both a first direction of the data string and a second direction of the hierarchy of the hierarchical tokens from the MSBs to the LSBs; and
outputting, by the DNN, selected hierarchical tokens as a representation of the data string, wherein the selected hierarchical tokens contain information of the attributes of the plurality of units in the data string.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the DNN is trained using text in a first modality as training data and then re-trained using a plurality of data strings in a second modality.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the DNN is trained using a plurality of data strings, and wherein training of the DNN comprises:
determining, in forward propagation, a plurality of tokenized strings, the plurality of tokenized strings representing different configurations corresponding to the data string;
determining first differences in attribute values among the plurality of tokenized strings, wherein the first differences in the attribute values are determined through the forward propagation of the DNN;
comparing the first differences with second differences obtained from ground truth; and
adjusting parameters of the DNN based on comparing the first differences with the second differences.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the first differences in the attribute values are determined at least partially through aggregating values obtained from a plurality of attention blocks in the DNN.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the attribute of the unit with the MSBs and the LSBs is a coordinate of the unit.

### Claim 6

A computer-implemented method comprising:
accessing, by a computing system, a set of physical structures corresponding to one or more molecules, each physical structure representative of a configuration of the one or more molecules;
encoding, by the computing system, the accessed physical structures to produce a set of encoded physical structures by encoding, for each accessed physical structure, a position of each atom of the one or more molecules within the accessed physical structure;
accessing, from a data store, a machine-learned model; and
retraining the machine-learned model by iteratively:
accessing two or more candidate physical structures for the one or more molecules;
determining, using the machine-learned model, a first energy difference between the two or more candidate physical structures for the one or more molecules, the first energy difference corresponding to a ranking of likelihood among the two or more candidate physical structures;
obtaining a second energy difference between the two or more candidate physical structures using an energy function; and
retraining the machine-learned model using the set of candidate physical structures, the first energy difference, and the second energy difference.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein encoding an accessed physical structure comprises:
converting one or more structural formulas of the one or more molecules into a sequence string representation of the one or more molecules;
tokenizing the sequence string representation of the one or more molecules to produce a tokenized structural formula;
tokenizing, for each atom in the one or more molecules, coordinates for the atom within the accessed physical structure to produce tokenized coordinates for the atom; and
combining the tokenized structural formula and the tokenized coordinates for each atom to produce a tokenized physical structure corresponding to the accessed physical structure.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein tokenizing coordinates for the atom within the accessed physical structure comprises:
pixelating a rendered sphere to produce a set of pixels each corresponding to a location on a surface of the rendered sphere;
tokenizing, for a first atom in the one or more molecules, coordinates at a center of the sphere; and
tokenizing, for each additional atom in the one or more molecules, coordinates corresponding to a pixel selected from the set of pixels based on a location of the additional atom relative to the center of the sphere.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein two or more additional atoms define a plane relative to the center of the sphere.

### Claim 10 (depends on 7)

The computer-implemented method of claim 7, wherein tokenizing coordinates for the atom within the accessed physical structure comprises using a Cartesian coordinate system, an xyz coordinate system, an octree coordinate system, a polar coordinate system, a cylindrical coordinate system, or a barycentric coordinate system to generate the coordinates.

### Claim 11 (depends on 7)

The computer-implemented method of claim 7, wherein the sequence string representation of the one or more molecules includes an ordered set of atoms, and wherein the coordinates for an atom comprise coordinates relative to a center of a coordinate sphere.

### Claim 12 (depends on 6)

The computer-implemented method of claim 6, wherein the machine-learned model is iteratively retrained until one or more retraining criteria is satisfied.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein the retraining criteria is satisfied when an average difference between the first energy difference and the second energy difference for a threshold number of consecutive iterations is less than a threshold difference.

### Claim 14 (depends on 12)

The computer-implemented method of claim 12, wherein the retraining criteria is satisfied when a distribution of candidate physical structures generated over a threshold number of consecutive iterations is within a threshold difference from a Boltzmann distribution.

### Claim 15 (depends on 12)

The computer-implemented method of claim 12, wherein the retraining criteria is satisfied when performance measurement of a holdout set starts to decrease.

### Claim 16 (depends on 6)

The computer-implemented method of claim 6, wherein retraining the machine-learned model comprises modifying weights of one or more layers of the machine-learned model to minimize a difference between the first energy difference and the second energy difference over subsequent iterations.

### Claim 17 (depends on 6)

The computer-implemented method of claim 6, wherein retraining the machine-learned model comprises modifying weights of one or more layers of the machine-learned model by:
generating a set of token sequences representing two or more physical structures;
store activation outputs corresponding to the tokens in the token sequences;
determining an aggregated energy state for each token sequence; and
determining backpropagated gradients based on comparing the aggregated energy states for the token sequences.

### Claim 18 (depends on 6)

The computer-implemented method of claim 6, wherein the first energy difference is determined based at least in part on a Boltzmann probability ratio between the set of candidate physical structures.

### Claim 19 (depends on 6)

The computer-implemented method of claim 6, wherein the machine-learned model is configured to:
determine, for a candidate physical structure, a force associated with each atom in the one or more molecules based on a relative energy associated with a plurality of neighboring positions for the atom.

### Claim 20 (depends on 19)

The computer-implemented method of claim 19, wherein the machine-learned model is retrained based on a difference in forces associated with the candidate physical structures.

### Claim 21 (depends on 19)

The computer-implemented method of claim 19, wherein the force associated with each atom in the model is based on a collective gradient of the relative energy associated with the plurality of neighboring positions for the atom, wherein the collective gradient is determined based on a vector to a neighboring position from which a finite difference gradient is calculated.

### Claim 22 (depends on 19)

The computer-implemented method of claim 19, wherein a determined force associated with an atom in the one or more molecules is based on forces associated with one or more preceding atoms in the one or more molecules.

### Claim 23 (depends on 19)

The computer-implemented method of claim 19, wherein the relative energies associated with each of the plurality of neighboring positions for the atom are determined using a softmax output of the machine-learned model.

### Claim 24 (depends on 19)

The computer-implemented method of claim 19, wherein a determined force associated with an atom in the one or more molecules is based on, for each of the plurality of neighboring positions for the atom, the relative energy associated with the neighboring position and a distance between the atom and the neighboring position.

### Claim 25 (depends on 19)

The computer-implemented method of claim 19, wherein a determined force generated by the machine-learned model associated with an atom in the one or more molecules is based additionally on reference forces applied from a solvent or solution on the atom in the one or more molecules.
