# US 2026/0073720 A1 · Methods, apparatus, and computer program products for progressively training a model and generating natural language outputs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0073720 A1
- Document ID: US20260073720A1
- Status: Published application
- Kind: A1
- Application number: 19/327,492
- Filed: 2025-09-12
- Published: 2026-03-12
- Earliest claimed: 2024-09-12
- Assignees: The University of North Carolina at Charlotte
- Applicants: The University of North Carolina at Charlotte
- Inventors: Srijan Das; Dominick Reilly
- Web page: https://patentgrep.com/patent/US20260073720A1

## CPC

- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06T 2207/20044: Skeletonization; Medial axis transform
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Described herein are apparatuses, methods, and computer program products for progressively training a model using video data comprising a plurality of modalities and corresponding natural language labels. The plurality of modalities comprise at least a video modality, an object modality, and a skeleton modality. Aa first stage includes individually projecting each of the video modality, the object modality, and the skeleton modality into an embedding space of the model. A second stage includes combining and projecting the video modality and the skeleton modality into the embedding space. A third stage includes combining and projecting the video modality, the object modality, and the skeleton modality into the embedding space. A language vision prediction system accesses the progressively trained model to ingest video data and to generate a natural language output associated with the video data.

## Claims

### Claim 1

An apparatus comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to:
progressively train a model using (a) video data comprising a plurality of modalities, the plurality of modalities comprising at least a video modality, an object modality, and a skeleton modality, and (b) corresponding natural language labels, wherein progressively training the model comprises:
in a first stage, individually projecting each of the video modality, the object modality, and the skeleton modality into an embedding space of the model;
in a second stage, combining and projecting the video modality and the skeleton modality into the embedding space; and
in a third stage, combining and projecting the video modality, the object modality, and the skeleton modality into the embedding space.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein progressively training the model further comprises aligning each modality with the embedding space using modality specific connectors.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein progressively training the model further comprises projecting each of the plurality of modalities into the embedding space using a linear projection layer to generate input token representations for each of the plurality of modalities.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein video data undergoes a semi-automated data curation process.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the semi-automated data curation process comprises person augmented generation, temporal stitching, and weakly supervised video descriptions.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the person augmented generation utilizes skeleton data to crop bounding boxes around individuals.

### Claim 7 (depends on 5)

The apparatus of claim 5, wherein the temporal stitching constructs long, untrimmed video sequences by stitching together shorter clips.

### Claim 8 (depends on 5)

The apparatus of claim 5, wherein the weakly supervised video descriptions generate image captions for each frame in a video and the image captions for each frame are synthesized into a cohesive video description.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the cohesive video descriptions are utilized to generate question answer pairs.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein progressively training the model further comprises extracting human object interaction features for the object modality.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein extracting human object interaction includes action-conditioned object detection and object localization and tracking.

### Claim 12 (depends on 10)

The apparatus of claim 10, wherein to extract skeleton features for the skeleton modality, a dual-encoder framework combines a skeleton backbone and a frozen text encoder.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the skeleton backbone is pretrained on trimmed clips for skeleton action classification.

### Claim 14

A method comprising:
progressively training a model using (a) video data comprising a plurality of modalities, the plurality of modalities comprising at least a video modality, an object modality, and a skeleton modality, and (b) corresponding natural language labels, wherein progressively training the model comprises:
a first stage that individually projects each of the video modality, the object modality, and the skeleton modality into an embedding space of the model,
a second stage that combines and projects the video modality and the skeleton modality into the embedding space, and
a third stage that combines and projects the video modality, the object modality, and the skeleton modality into the embedding space.

### Claim 15

An apparatus comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to:
access a model progressively trained with (a) video data comprising a plurality of modalities, the plurality of modalities comprising at least a video modality, an object modality, and a skeleton modality, and (b) corresponding natural language labels; and
ingest input video data into the model to generate a natural language output.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the input video data lacks at least one of the skeleton modality or the object modality.

### Claim 17 (depends on 15)

The apparatus of claim 15, wherein the model is trained progressively with a first stage that individually projects each of the video modality, the object modality, and the skeleton modality into an embedding space of the model, a second stage that combines and projects the video modality and the skeleton modality into the embedding space, and a third stage that combines and projects the video modality, the object modality, and the skeleton modality into the embedding space.

### Claim 18 (depends on 15)

The apparatus of claim 15, wherein generating the natural language output comprises answering a question about an action in the input video data.

### Claim 19 (depends on 15)

The apparatus of claim 15, wherein the model predicts a missing action in a temporal sequence of the video.

### Claim 20 (depends on 15)

The apparatus of claim 15, wherein the missing action is a subsequent action that occurs after the end of the video.
