# US 2025/0384652 A1 · Grounded prompting and adaptation for referring video object segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0384652 A1
- Document ID: US20250384652A1
- Status: Published application
- Kind: A1
- Application number: 19/072,880
- Filed: 2025-03-06
- Published: 2025-12-18
- Earliest claimed: 2024-06-17
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Min-Hung Chen; Ci-Siang Lin; Chien-Yi Wang; Sifei Liu; Yu-Chiang Wang
- Web page: https://patentgrep.com/patent/US20250384652A1

## CPC

- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/40: Extraction of image or video features
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/776: Validation; Performance evaluation
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06V 10/82: using neural networks
- G06V 2201/07: Target detection
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Referring Video Object Segmentation (RVOS) aims to segment an object referred to by a sentence query throughout an entire video. In contrast to Referring Image Segmentation (RIS), RVOS is particularly faced with dynamic visual challenges, such as position and size variation, pose deformation, object occlusion or exit, and scene variation. Moreover, the referring sentence may contain long-term motions or actions, which may not be easily recognized from a single frame. Existing works that address this challenging task generally require end-to-end training for vision-language models, which can be computationally expensive and time-consuming, while the requirement of dense mask annotations for training impedes the scalability of those approaches. The present disclosure uses grounded prompting to adapt image-based segmentation models to video object segmentation tasks, which can be achieved with relying only on weak supervision.

## Claims

### Claim 1

A method, comprising
receiving an input comprised of a video and a text prompt referring to an object in the video that is to be edited;
processing at least one frame of the video and the text prompt, using a first model, to generate location information for the target object in the video, wherein the model is trained to:
generate a frame-level bounding box in the at least one frame of the video for the object referred to by the text prompt, and
provide a video-level alignment of the frame-level bounding box with the text prompt to generate the location information for the target object in the video;
processing the location information, the at least one frame of the video, and the text prompt, by a second model, to generate an object mask for the target object in the at least one frame of the video;
determining an instruction for editing the object in the video; and
based on the instruction, editing a portion of the at least one frame of the video corresponding to the object mask.

### Claim 2 (depends on 1)

The method of claim 1, wherein the location information is a proposed frame-level bounding box resulting from the video-level alignment.

### Claim 3 (depends on 1)

The method of claim 1, wherein the instruction for editing the object in the video is determined from the text prompt.

### Claim 4 (depends on 1)

The method of claim 1, wherein for each frame of the at least one frame, the editing is performed on the portion of the frame corresponding to the object mask generated for the frame.

### Claim 5 (depends on 4)

The method of claim 4, wherein the at least one frame is a plurality of frames.

### Claim 6 (depends on 1)

The method of claim 1, wherein the instruction for editing the object includes an instruction to change at least one feature of the object.

### Claim 7 (depends on 5)

The method of claim 5, wherein the at least one feature of the object includes a color of the object.

### Claim 8 (depends on 5)

The method of claim 5, wherein the at least one feature of the object includes a size of the object.

### Claim 9 (depends on 1)

The method of claim 1, wherein the second model is an image-based foundation segmentation model.

### Claim 10 (depends on 1)

The method of claim 1, wherein the editing is performed by a video editing application configured to use the object mask generated for the at least one frame of the video.

### Claim 11

A method, comprising:
at a device, training a model to generate location information for a target object in a video based on a text prompt referring to the target object in the video by:
accessing a training video in a dataset of training videos each labeled with a text prompt referring to an object in the training video and per-frame bounding boxes corresponding to the object in the training video;
training the model to generate frame-level bounding boxes for the object referred to by the text prompt labeled in the training video; and
training the model to provide a video-level alignment of the frame-level bounding boxes with the text prompt labeled in the training video.

### Claim 12 (depends on 11)

The method of claim 11, wherein training the model to generate the frame-level bounding boxes for the object referred to by the text prompt labeled in the training video includes iteratively:
using the model to generate from the training video a set of frame-level bounding boxes for the object referred to by the text prompt labeled in the training video,
computing a loss between the set of frame-level bounding boxes and the per-frame bounding boxes labeled in the training video, and
updating the model based on the loss.

### Claim 13 (depends on 12)

The method of claim 12, wherein using the model to generate from the training video the set of frame-level bounding boxes for the object referred to by the text prompt labeled in the training video includes:
extracting frame-level visual features for each frame of the training video and linguistic features of the text prompt labeled in the training video,
obtaining a set of object queries using the frame-level visual features of each frame of the training video and the linguistic features of the text prompt labeled in the training video,
using the set of object queries to generate the frame-level bounding boxes for the object referred to by the text prompt labeled in the training video,
computing a loss between the frame-level bounding boxes and the per-frame bounding boxes labeled in the training video, and
updating the model based on the loss.

### Claim 14 (depends on 13)

The method of claim 13, wherein each object query in the set of object queries corresponds to a frame of the training video and includes the visual features of the frame as a key and the linguistic features of the text prompt labeled in the training video as a value.

### Claim 15 (depends on 13)

The method of claim 13, wherein each object query in the set of object queries corresponds to a frame of the training video and is used to generate a plurality of candidate bounding boxes in the frame for the object referred to by the text prompt labeled in the training video, and wherein one of the candidate bounding boxes having a highest confidence score from among the plurality of candidate bounding boxes is selected as the frame-level bounding box for the frame.

### Claim 16 (depends on 11)

The method of claim 11, wherein contrastive learning is used to train the model to generate the frame-level bounding boxes for the object referred to by the text prompt labeled in the training video.

### Claim 17 (depends on 16)

The method of claim 16, wherein the contrastive learning is performed using a different set of frame-level bounding boxes generated by the model for a different object referred to by a different text prompt.

### Claim 18 (depends on 11)

The method of claim 11, wherein training the model to provide the video-level alignment of the frame-level bounding boxes with the text prompt labeled in the training video includes:
extracting video-level visual features for the training video, and
using the video-level visual features to train the model to provide the video-level alignment of the frame-level bounding boxes with the text prompt labeled in the training video.

### Claim 19 (depends on 18)

The method of claim 18, wherein the video-level visual features for the training video are extracted by:
extracting frame-level visual features for each frame of the training video,
performing cross-attention at each frame of the training video by taking the frame-level bounding box for the frame as a query and the frame-level visual features for the frame as keys and values, and
applying an average pooling operation for temporal aggregation of a result of the performance of the cross-attention at each frame of the training video to generate the video-level visual features for the training video.

### Claim 20 (depends on 18)

The method of claim 18, wherein contrastive learning is used train the model to provide the video-level alignment of the frame-level bounding boxes with the text prompt labeled in the training video.

### Claim 21 (depends on 20)

The method of claim 20, wherein the contrastive learning is performed using linguistic features of the text prompt labeled in the training video and linguistic features of a different text prompt referring to a different object in the training video.

### Claim 22 (depends on 11)

The method of claim 11, wherein the location information for the target object is configured to be provided as a prompt to an image-based foundation segmentation model to provide referring video object segmentation for the video.

### Claim 23 (depends on 11)

The method of claim 11, further comprising, at the device:
deploying the trained model.

### Claim 24 (depends on 23)

The method of claim 23, wherein the trained model is deployed with an image-based foundation segmentation model to adapt the image-based foundation segmentation model to provide referring video object segmentation.

### Claim 25 (depends on 24)

The method of claim 24, wherein at inference time:
the trained model generates location information for the target object in the video based on the text prompt referring to the target object in the video,
the trained model inputs the location information, the video, and the text prompt to the image-based foundation segmentation model to cause the image-based foundation segmentation model to generate object masks for the target object in the video.

### Claim 26 (depends on 25)

The method of claim 25, wherein the object masks are used by a downstream application.

### Claim 27 (depends on 26)

The method of claim 26, wherein the downstream application is a video editing application.

### Claim 28 (depends on 26)

The method of claim 26, wherein the downstream application is a video analysis application.

### Claim 29

A system, comprising:
a non-transitory memory storage comprising instructions; and
one or more processors in communication with the memory, wherein the one or more processors execute the instructions to train a model to generate location information for a target object in a video based on a text prompt referring to the target object in the video by:
accessing a training video in a dataset of training videos each labeled with a text prompt referring to an object in the training video and per-frame bounding boxes corresponding to the object in the training video;
training the model to generate frame-level bounding boxes for the object referred to by the text prompt labeled in the training video; and
training the model to provide a video-level alignment of the frame-level bounding boxes with the text prompt labeled in the training video.

### Claim 30 (depends on 29)

The system of claim 29, wherein the one or more processors further execute the instructions to:
deploy the trained model.

### Claim 31 (depends on 30)

The system of claim 30, wherein the trained model is deployed with an image-based foundation segmentation model to adapt the image-based foundation segmentation model to provide referring video object segmentation.

### Claim 32 (depends on 31)

The system of claim 31, wherein at inference time:
the trained model generates location information for the target object in the video based on the text prompt referring to the target object in the video,
the trained model inputs the location information, the video, and the text prompt to the image-based foundation segmentation model to cause the image-based foundation segmentation model to generate object masks for the target object in the video.

### Claim 33 (depends on 32)

The system of claim 32, wherein the object masks are used by a downstream application.

### Claim 34 (depends on 33)

The system of claim 33, wherein the downstream application is one of:
a video editing application, or
a video analysis application.

### Claim 35

A non-transitory computer-readable media storing computer instructions which when executed by one or more processors of a device cause the device to train a model to generate location information for a target object in a video based on a text prompt referring to the target object in the video by:
accessing a training video in a dataset of training videos each labeled with a text prompt referring to an object in the training video and per-frame bounding boxes corresponding to the object in the training video;
training the model to generate frame-level bounding boxes for the object referred to by the text prompt labeled in the training video; and
training the model to provide a video-level alignment of the frame-level bounding boxes with the text prompt labeled in the training video.

### Claim 36 (depends on 35)

The non-transitory computer readable medium of claim 35, wherein the one or more processors further cause the device to:
deploy the trained model.

### Claim 37 (depends on 36)

The non-transitory computer readable medium of claim 36, wherein the trained model is deployed with an image-based foundation segmentation model to adapt the image-based foundation segmentation model to provide referring video object segmentation.

### Claim 38 (depends on 37)

The non-transitory computer readable medium of claim 37, wherein at inference time:
the trained model generates location information for the target object in the video based on the text prompt referring to the target object in the video,
the trained model inputs the location information, the video, and the text prompt to the image-based foundation segmentation model to cause the image-based foundation segmentation model to generate object masks for the target object in the video.

### Claim 39 (depends on 38)

The non-transitory computer readable medium of claim 38, wherein the object masks are used by a downstream application.

### Claim 40 (depends on 39)

The non-transitory computer readable medium of claim 39, wherein the downstream application is one of:
a video editing application, or
a video analysis application.
