# US 2022/0284590 A1 · Disentangled feature transforms for video object segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284590 A1
- Document ID: US20220284590A1
- Status: Published application
- Kind: A1
- Application number: 17/192,599
- Filed: 2021-03-04
- Published: 2022-09-08
- Earliest claimed: 2021-03-04
- Assignees: Lemon Inc.
- Applicants: Lemon Inc.
- Inventors: Linjie Yang; Ziyu Jiang; Ding Liu; Longyin Wen
- Web page: https://patentgrep.com/patent/US20220284590A1

## CPC

- G06T 7/11: Region-based segmentation
- G06T 7/174: involving the use of two or more images
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06T 7/194: involving foreground-background segmentation
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and method directed to performing video object segmentation are provided. In examples, video data representing a sequence of image frames and video data representing an object mask may be received at a video object segmentation server. Image features may be generated based on a first image frame of the sequence of image frames, image features may be generated based on a second image frame of the sequence of image frames; and object features may be generated based on the object mask. A transform matrix may be computed based on the image features of the first image frame and image features of the second image frame; the transform matrix may be applied to the object features resulting in transformed object features. A predicted object mask associated with the second image frame may be obtained by decoding the transformed object features.

## Claims

### Claim 1

A method for performing video object segmentation, the method comprising:
receiving video data representing a sequence of image frames;
receiving an object mask;
generating image features based on a first image frame of the sequence of image frames;
generating object features based on the object mask;
generating image features based on a second image frame of the sequence of image frames;
computing a transform matrix based on the image features of the first image frame and the image features of the second image frame;
applying the transform matrix to the object features resulting in transformed object features; and
obtaining a predicted object mask associated with the second image frame by decoding the transformed object features.

### Claim 2 (depends on 1)

The method of claim 1, wherein the predicted object mask associated with the second image frame is obtained based on the transformed object features and decoded image features associated with the second image frame.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
receiving a second object mask;
generating second object features based on the second object mask;
applying the transform matrix to the second object features based on the second object mask resulting in second transformed object features; and
obtaining a second predicted object mask associated with the second image frame by decoding the second transformed object features.

### Claim 4 (depends on 3)

The method of claim 3, wherein the object features are generated based on the object mask and the image features based on the first image frame and the second object features are generated based on the second object mask and the image features based on the first image frame.

### Claim 5 (depends on 4)

The method of claim 4, wherein the predicted object mask associated with the second image frame is based on the transformed object features and decoded image features associated with the second image frame, and wherein the second predicted object mask associated with the second image frame is based on the second transformed object features and the decoded image features associated with the second image frame.

### Claim 6 (depends on 5)

The method of claim 5 further comprising obtaining the predicted object mask associated with the second image frame by decoding the transformed object features at a decoder that is different than a decoder that decodes the object features associated with the second image frame.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
generating the image features based on the first image frame of the sequence of image frames at an encoder that is different than an encoder that generates the object features based on the object mask.

### Claim 8

A system configured to perform video object segmentation, the system comprising:
a processor; and
memory including instructions, which when executed by the processor, causes the processor to:
receive video data representing a sequence of image frames;
receive an object mask;
generate image features based on a first image frame of the sequence of image frames;
generate object features based on the object mask;
generate image features based on a second image frame of the sequence of image frames;
compute a transform matrix based on the image features of the first image frame and the image features of the second image frame;
apply the transform matrix to the object features resulting in transformed object features; and
obtain a predicted object mask associated with the second image frame by decoding the transformed object features.

### Claim 9 (depends on 8)

The system of claim 8, wherein the instructions, when executed by the processor, cause the processor to:
receive a second object mask;
generate second object features based on the second object mask;
apply the transform matrix to the second object features based on the second object mask resulting in second transformed object features; and
obtain a second predicted object mask associated with the second image frame by decoding the second transformed object features.

### Claim 10 (depends on 9)

The system of claim 9, wherein the object features are generated based on the object mask and image features of the first image frame and the second object features are generated based on the second object mask and the image features of the first image frame.

### Claim 11 (depends on 10)

The system of claim 10, wherein the predicted object mask associated with the second image frame is based on the transformed object features and decoded image features associated with the second image frame, and wherein the second predicted object mask associated with the second image frame is based on the second transformed object features and the decoded image features associated with the second image frame.

### Claim 12 (depends on 11)

The system of claim 11, wherein the instructions, when executed by the processor, cause the processor to obtain the predicted object mask associated with the second image frame by decoding the transformed object features at a decoder that is different than a decoder that decodes the object features associated with the second image frame.

### Claim 13 (depends on 8)

The system of claim 8, wherein the instructions, when executed by the processor, cause the processor to generate the image features based on the first image frame of the sequence of image frames at an encoder that is different than an encoder that generates the object features based on the object mask.

### Claim 14 (depends on 8)

The system of claim 8, wherein the transformation matrix includes similarity information indicating a similarity between the first image frame and the second image frame.

### Claim 15

A computer-readable storage medium including instructions, which when executed by a processor, cause the processor to:
receive video data representing a sequence of image frames;
receive an object mask;
generate image features based on a first image frame of the sequence of image frames;
generate object features based on the object mask;
generate image features based on a second image frame of the sequence of image frames;
compute a transform matrix based on the image features of the first image frame and the image features of the second image frame;
apply the transform matrix to the object features resulting in transformed object features; and
obtain a predicted object mask associated with the second image frame by decoding the transformed object features.

### Claim 16 (depends on 15)

The computer-readable storage medium of claim 15, wherein the instructions, which when executed by a processor, cause the processor to:
receive a second object mask;
generate second object features based on the second object mask;
apply the transform matrix to the second object features based on the second object mask resulting in second transformed object features; and
obtain a second predicted object mask associated with the second image frame by decoding the second transformed object features.

### Claim 17 (depends on 16)

The computer-readable storage medium of claim 16, wherein the object features are generated based on the object mask and the image features based on the first image frame, and the second object features are generated based on the second object mask and the image features based on the first image frame.

### Claim 18 (depends on 17)

The computer-readable storage medium of claim 17, wherein the predicted object mask associated with the second image frame is based on the transformed object features and decoded image features associated with the second image frame, and wherein the second predicted object mask associated with the second image frame is based on the second transformed object features and the decoded image features associated with the second image frame.

### Claim 19 (depends on 18)

The computer-readable storage medium of claim 18, wherein the instructions, which when executed by a processor, cause the processor to obtain the predicted object mask associated with the second image frame by decoding the transformed object features at a decoder that is different than a decoder that decodes the object features associated with the second image frame.

### Claim 20 (depends on 15)

The computer-readable storage medium of claim 15, wherein the instructions, which when executed by a processor, cause the processor to generate the image features based on the first image frame of the sequence of image frames at an encoder that is different than an encoder that generates the object features based on the object mask.
