# US 12,602,928 B2 · Mask propagation for video segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,928 B2
- Document ID: US12602928B2
- Status: Granted
- Kind: B2
- Application number: 18/356,892
- Filed: 2023-07-21
- Published: 2025-01-23
- Granted: 2026-04-14
- Earliest claimed: 2023-07-21
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Joon-Young Lee; Seoung Wug Oh; Ho Kei Cheng; Brian Price
- Web page: https://patentgrep.com/patent/US12602928B2

## CPC

- G06V 20/48: {Matching video sequences}
- G06T 7/174: involving the use of two or more images
- G06T 7/187: involving region growing; involving region merging; involving connected component labelling
- G06T 7/194: involving foreground-background segmentation
- G06T 7/215: Motion-based segmentation
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T 2207/30232: Surveillance
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Embodiments are disclosed for performing universal segmentation to mask objects across multiple frames of a video. The method may include determining an image segmentation mask which masks an object of a frame of a video sequence using the frame and an image segmentation module of a segmentation system. The method further includes determining a mask propagation mask which masks the object of the frame of the video sequence using the frame, a representation of a previous frame of the video sequence, and a mask propagation module of the segmentation system. The method further includes determining a frame mask which masks the object of the frame of the video sequence based on a comparison of the image segmentation mask and the mask propagation mask. The method further includes displaying the frame mask of the video sequence.

## Claims

### Claim 1

A method comprising:
determining an image segmentation mask which masks an object of a frame of a video sequence using the frame and an image segmentation module of a segmentation system;
determining a mask propagation mask which masks the object of the frame of the video sequence using the frame, a representation of a previous frame of the video sequence, and a mask propagation module of the segmentation system;
determining a frame mask which masks the object of the frame of the video sequence based on a comparison of the image segmentation mask and the mask propagation mask; and
displaying the frame mask of the video sequence.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the frame mask further comprises:
determining that the image segmentation mask matches the mask propagation mask based on a similarity of the image segmentation mask and the mask propagation mask satisfying a threshold.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
determining a representation of the frame mask using the frame, the frame mask, and the representation of the previous frame of the video sequence; and
storing the representation of the frame mask.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
determining a second image segmentation mask which masks each object of a plurality of objects of a second frame of the video sequence using the second frame and the image segmentation module;
determining a second mask propagation mask which masks each object of the plurality of objects of the second frame using the second frame, the representation of the frame mask, and the mask propagation module of the segmentation system; and
determining a second frame mask which masks each object of the plurality of objects of the frame based on a comparison of the second image segmentation mask and the second mask propagation mask.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
determining an intermediary mask by merging each masked object of the second image segmentation mask and each masked object of the second mask propagation mask that satisfy a similarity threshold; and
determining the second frame mask using the intermediary mask, the second frame, the representation of the frame mask, and the mask propagation module of the segmentation system.

### Claim 6 (depends on 5)

The method of claim 5, further comprising adding an unmatched mask objected from the second image segmentation mask or the second mask propagation mask to the intermediary mask.

### Claim 7 (depends on 3)

The method of claim 3, further comprising:
receiving a second frame of the video sequence including the object, wherein the second frame occurs at a time after the frame of the video sequence; and
determining a second frame mask of the object of the second frame using the second frame, the representation of the frame mask, and the mask propagation module of the segmentation system.

### Claim 8 (depends on 1)

The method of claim 1, wherein the representation of the previous frame is a feature map of the previous frame determined by the image segmentation module, wherein the previous frame occurs at a time before the frame of the video sequence.

### Claim 9 (depends on 1)

The method of claim 1, wherein the image segmentation module and the mask propagation module of the segmentation system are each trained independently.

### Claim 10

A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
determining an image segmentation mask which masks an object of a frame of a video sequence using the frame and an image segmentation module of a segmentation system;
determining a mask propagation mask which masks the object of the frame of the video sequence using the frame, a representation of a previous frame of the video sequence, and a mask propagation module of the segmentation system;
determining a frame mask which masks the object of the frame of the video sequence based on a comparison of the image segmentation mask and the mask propagation mask; and
displaying the frame mask of the video sequence.

### Claim 11 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein determining the frame mask further comprises:
determining that the image segmentation mask matches the mask propagation mask based on a similarity of the image segmentation mask and the mask propagation mask satisfying a threshold.

### Claim 12 (depends on 10)

The non-transitory computer-readable medium of claim 10, storing instructions that further cause the processing device to perform operations comprising:
determining a representation of the frame mask using the frame, the frame mask, and the representation of the previous frame of the video sequence; and
storing the representation of the frame mask.

### Claim 13 (depends on 12)

The non-transitory computer-readable medium of claim 12, storing instructions that further cause the processing device to perform operations comprising:
determining a second image segmentation mask which masks each object of a plurality of objects of a second frame of the video sequence using the second frame and the image segmentation module;
determining a second mask propagation mask which masks each object of the plurality of objects of the second frame using the second frame, the representation of the frame mask, and the mask propagation module of the segmentation system; and
determining a second frame mask which masks each object of the plurality of objects of the frame based on a comparison of the second image segmentation mask and the second mask propagation mask.

### Claim 14 (depends on 13)

The non-transitory computer-readable medium of claim 13, storing instructions that further cause the processing device to perform operations comprising:
determining an intermediary mask by merging each masked object of the second image segmentation mask and each masked object of the second mask propagation mask that satisfy a similarity threshold; and
determining the second frame mask using the intermediary mask, the second frame, the representation of the frame mask, and the mask propagation module of the segmentation system.

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14, storing instructions that further cause the processing device to perform operations comprising:
adding an unmatched mask objected from the second image segmentation mask or the second mask propagation mask to the intermediary mask.

### Claim 16 (depends on 12)

The non-transitory computer-readable medium of claim 12, storing instructions that further cause the processing device to perform operations comprising:
receiving a second frame of the video sequence including the object, wherein the second frame occurs at a time after the frame of the video sequence; and
determining a second frame mask of the object of the second frame using the second frame, the representation of the frame mask, and the mask propagation module of the segmentation system.

### Claim 17

A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
determining a first mask which masks each object of a plurality of objects of a frame of a video sequence using the frame and an image segmentation module;
determining a second mask which masks each object of the plurality of objects of the frame using the frame, a representation of a previous frame of the video sequence, and a mask propagation module;
comparing the first mask and the second mask to obtain a coarse estimation of a third mask;
refining the coarse estimation of the third mask to obtain a refined estimation using the frame, the coarse estimation of the third mask, the representation of the previous frame of the video sequence, and the mask propagation module.

### Claim 18 (depends on 17)

The system of claim 17, wherein the coarse estimation of the third mask includes an estimated masked object that shares one or more pixels of a first object of the plurality of objects and a second object of the plurality of objects.

### Claim 19 (depends on 17)

The system of claim 17, wherein the representation of the previous frame is a feature map of the previous frame determined by the image segmentation module, wherein the previous frame occurs at a time before the frame of the video sequence.

### Claim 20 (depends on 17)

The system of claim 17, wherein an object of the plurality of objects is a background object or a foreground object.