# US 2026/0228498 A1 · Systems and methods for performing prediction tasks from tabular data using a hierarchical set of encoder machine learning models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0228498 A1
- Document ID: US20260228498A1
- Status: Published application
- Kind: A1
- Application number: 19/345,509
- Filed: 2025-09-30
- Published: 2026-08-06
- Earliest claimed: 2025-01-31
- Applicants: The Toronto-Dominion Bank
- Inventors: George Frazer STEIN; Keyu LONG; Xin NING; Michelle Erica LEVINE; Saba ZUBERI; Maksims VOLKOVS
- Web page: https://patentgrep.com/patent/US20260228498A1

## CPC

- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for performing one or more prediction task. The methods comprise: receiving a hierarchical set of encoder machine learning models configured to convert one or more tabular datasets into a token that represents the one or more tabular datasets; pre-training the hierarchical set of encoder machine learning models using self-supervised learning to generate a pre-trained hierarchical set of encoder machine learning models; fine-tuning a predictive model to perform one or more prediction task using supervised learning, the predictive model comprising a prediction head for each prediction task, each prediction head configured to generate a prediction for the corresponding prediction task based on the token generated by the pre-trained hierarchical set of encoder machine learning models; and processing one or more new tabular datasets using the fine-tuned predictive model to generate a prediction for each prediction task.

## Claims

### Claim 1

A system for performing one or more prediction task, the system comprising at least one processor configured to:
receive a hierarchical set of encoder machine learning models configured to convert one or more tabular datasets into a token that represents the one or more tabular datasets;
pre-train the hierarchical set of encoder machine learning models using self-supervised learning to generate a pre-trained hierarchical set of encoder machine learning models;
fine-tune a predictive model to perform the one or more prediction task using supervised learning, the predictive model comprising a prediction head for each of the one or more prediction task, each prediction head configured to generate a prediction for the corresponding prediction task based on the token generated by the pre-trained hierarchical set of encoder machine learning models; and
process one or more new tabular datasets using the fine-tuned predictive model to generate a prediction for the one or more prediction task.

### Claim 2 (depends on 1)

The system of claim 1, wherein pre-training the hierarchical set of encoder machine learning models using self-supervised learning comprises training the hierarchical set of encoder machine learning models to generate one or more element of the one or more tabular datasets from other elements of the one or more tabular datasets.

### Claim 3 (depends on 2)

The system of claim 2, wherein the one or more element of the one or more tabular datasets is identified by a mask.

### Claim 4 (depends on 1)

The system of claim 1, wherein each tabular dataset of the one or more tabular datasets comprises data related to a same historical time period.

### Claim 5 (depends on 4)

The system of claim 4, wherein pre-training the hierarchical set of encoder machine-learning models using self-supervised learning comprises training the hierarchical set of encoder machine learning models to generate one or more elements of one or more second tabular datasets that comprise data related to a next historical time period.

### Claim 6 (depends on 1)

The system of claim 1, wherein the one or more tabular datasets comprises a plurality of tabular datasets and the hierarchical set of encoder machine learning models comprises:
a plurality of first layer encoder machine learning models configured to generate, for each tabular dataset in the plurality of tabular datasets, one or more intermediate token that represents that tabular dataset; and
a second layer encoder machine learning model that is configured to generate the token from the intermediate tokens generated by the plurality of first layer encoder machine learning models.

### Claim 7 (depends on 6)

The system of claim 6, wherein the plurality of first layer encoder machine learning models comprises, for each different type of tabular dataset in the plurality of tabular datasets, a first layer encoder machine learning model that is configured to generate the one or more intermediate token for each tabular dataset in the plurality of tabular datasets of that type.

### Claim 8 (depends on 6)

The system of claim 6, wherein each tabular dataset of the plurality of tabular datasets comprises data related to a same historical time period, the historical time period is sub-divided into a plurality of sub-time periods, and the one or more intermediate token that represents a tabular dataset comprises an intermediate token for each of the plurality of sub-time periods that represents data in the tabular dataset related to that sub-time period.

### Claim 9 (depends on 8)

The system of claim 8, wherein the historical time period is a 12-month period, and the plurality of sub-time periods comprises a sub-time period for each month of the 12-month period.

### Claim 10 (depends on 6)

The system of claim 6, wherein each intermediate token is a same size.

### Claim 11 (depends on 6)

The system of claim 6, wherein at least one first layer encoder machine learning model of the plurality of first layer encoder machine learning models comprises a multi-layer perceptron neural network.

### Claim 12 (depends on 6)

The system of claim 6, wherein at least one first layer encoder machine learning model of the plurality of first layer encoder machine learning models comprises an encoder-only transformer model.

### Claim 13 (depends on 1)

The system of claim 1, wherein the token is a multi-element vector.

### Claim 14 (depends on 13)

The system of claim 13, wherein each element of the multi-element vector is a floating-point number.

### Claim 15 (depends on 13)

The system of claim 13, wherein the multi-element vector comprises 128 elements.

### Claim 16 (depends on 1)

The system of claim 1, wherein each tabular dataset of the one or more tabular datasets comprises time series data.

### Claim 17 (depends on 1)

The system of claim 1, wherein at least one of the one or more prediction task comprises predicting a future event.

### Claim 18 (depends on 1)

The system of claim 1, wherein the prediction head for at least one of the one or more prediction task comprises a multi-layer perceptron neural network.

### Claim 19

A method for performing one or more prediction task, the method executed in a computing environment comprising at least one processor, the method comprising:
receiving a hierarchical set of encoder machine learning models configured to convert one or more tabular datasets into a token that represents the one or more tabular datasets;
pre-training the hierarchical set of encoder machine learning models using self-supervised learning to generate a pre-trained hierarchical set of encoder machine learning models;
fine-tuning a predictive model to perform the one or more prediction task using supervised learning, the predictive model comprising a prediction head for each of the one or more prediction task, each prediction head configured to generate a prediction for the corresponding prediction task based on the token generated by the pre-trained hierarchical set of encoder machine learning models; and
processing a plurality of new tabular datasets using the fine-tuned predictive model to generate a prediction for the one or more prediction task.

### Claim 20

A non-transitory computer readable medium storing computer executable instructions which, when executed by at least one computer processor, cause the at least one computer processor to carry out a method for performing one or more prediction task, the method comprising:
receiving a hierarchical set of encoder machine learning models configured to convert one or more tabular datasets into a token that represents the one or more tabular datasets;
pre-training the hierarchical set of encoder machine learning models using self-supervised learning to generate a pre-trained hierarchical set of encoder machine learning models;
fine-tuning a predictive model to perform the one or more prediction task using supervised learning, the predictive model comprising a prediction head for each of the one or more prediction task, each prediction head configured to generate a prediction for the corresponding prediction task based on the token generated by the pre-trained hierarchical set of encoder machine learning models; and
processing a plurality of new tabular datasets using the fine-tuned predictive model to generate a prediction for the one or more prediction task.
