# US 2025/0157055 A1 · Method and apparatus with scene flow estimation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0157055 A1
- Document ID: US20250157055A1
- Status: Published application
- Kind: A1
- Application number: 18/944,556
- Filed: 2024-11-12
- Published: 2025-05-15
- Earliest claimed: 2023-11-15
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Xiongfeng PENG; Zhihua LIU; Qiang WANG; Soon-Yong CHO
- Web page: https://patentgrep.com/patent/US20250157055A1

## CPC

- G06T 7/248: {involving reference images or patches}
- G06T 7/215: Motion-based segmentation
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/269: using gradient-based methods
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A scene flow estimation method includes: inputting a frame pair into an artificial intelligence (AI) network, and obtaining therefrom a motion embedding feature and a non-occluded-category label embedding feature corresponding to a target pixel in the frame pair; and estimating a scene flow corresponding to the frame pair based on the motion embedding feature and the non-occluded-category label embedding feature, the frame pair includes a first frame and a second frame, the first frame including a first color image and a first depth image and the second frame including a second color image and a second depth image, the non-occluded-category label embedding feature includes category information of an object corresponding to a pixel pair in the frame pair, the pixel pair includes a first pixel of the first frame and a second pixel of the second frame, and the second pixel corresponds to the first pixel.

## Claims

### Claim 1

A processor-implemented scene flow estimation method, comprising:
inputting a frame pair into an artificial intelligence (AI) network, and obtaining therefrom a motion embedding feature and a non-occluded-category label embedding feature corresponding to a target pixel in the frame pair; and
estimating a scene flow corresponding to the frame pair based on the motion embedding feature and the non-occluded-category label embedding feature,
wherein the frame pair comprises a first frame and a second frame,
wherein the first frame comprises a first color image and a first depth image and the second frame comprises a second color image and a second depth image,
wherein the non-occluded-category label embedding feature comprises category information of an object corresponding to a pixel pair in the frame pair,
wherein the pixel pair comprises a first pixel of the first frame and a second pixel of the second frame, the second pixel corresponding to the first pixel.

### Claim 2 (depends on 1)

The scene flow estimation method of claim 1, wherein the estimating of the scene flow corresponding to the frame pair comprises:
obtaining a motion field corresponding to the frame pair;
obtaining the motion embedding feature and the non-occluded-category label embedding feature by inputting the motion field into the AI network;
fusing the motion embedding feature with the non-occluded-category label embedding feature to obtain a fused embedding feature;
obtaining a target motion field by updating the motion field based on the fused embedding feature; and
estimating the scene flow corresponding to the frame pair based on the target motion field.

### Claim 3 (depends on 2)

The scene flow estimation method of claim 2, wherein the obtaining of the target motion field comprises:
determining a neighboring point set comprising pixels in the second frame that correspond to the target pixel in the first frame;
determining a matching level between the target pixel and the pixels in the neighboring point set, based on a fused embedding feature corresponding to the target pixel and based on fused embedding features of the respective pixels in the neighboring point set; and
obtaining the target motion field by updating the motion field based on the matching level.

### Claim 4 (depends on 2)

The scene flow estimation method of claim 2, wherein the estimating of the scene flow corresponding to the frame pair comprises:
obtaining a weight that adjusts the motion field, based on similarity levels of similarity between pixels of the first color image; and
estimating the scene flow corresponding to the frame pair, based on the target motion field obtained by applying the weight to the motion field.

### Claim 5 (depends on 4)

The scene flow estimation method of claim 4, wherein the obtaining of the weight comprises:
obtaining a first correlation between the pixels of the first color image by inputting the first color image into an attention encoder of the AI network; and
determining a first weight corresponding to the first color image based on the first correlation.

### Claim 6 (depends on 5)

The scene flow estimation method of claim 5, wherein the obtaining of the weight comprises:
obtaining a second correlation between pixels of the second color image by inputting the color image comprised in the second frame into the attention encoder;
determining a second weight corresponding to the second color image based on the second correlation; and
obtaining a fused weight that adjusts the motion field by fusing the first weight with the second weight.

### Claim 7 (depends on 2)

The scene flow estimation method of claim 2, wherein the obtaining of the motion embedding feature and the non-occluded-category label embedding feature comprises:
extracting, based on a feature encoder comprised in the AI network, a first frame feature from the first frame and a second frame feature from the second frame;
generating a correlation volume corresponding to the frame pair, based on a correlation between the first frame feature and the second frame feature;
extracting, based on a context encoder comprised in the AI network, a context feature and a hidden state corresponding to the first frame from the first frame data; and
obtaining the motion embedding feature and the non-occluded-category label embedding feature, by an operation of a convolutional gated recurrent unit (CGRU)-based update network comprised in the AI network, the operation based on the context feature, the hidden state, the motion field, and the correlation volume.

### Claim 8 (depends on 1)

The scene flow estimation method of claim 1, further comprising:
training the AI network by iteratively performing a training operation on the AI network with a training set until a training end condition is satisfied.

### Claim 9 (depends on 8)

The scene flow estimation method of claim 8, wherein the training set comprises a sample frame pair, and a ground truth (GT) scene flow value and a GT non-occluded-category label mask value corresponding to the sample frame pair,
wherein sample frame pair comprises a first sample frame comprising a first sample color image and a first sample depth image, and further comprises a second sample frame comprising a second sample color image and a second sample depth image,
wherein the GT non-occluded-category label mask value comprises category information corresponding to a sample pixel pair in the sample frame pair,
wherein the sample pixel pair comprises a first sample pixel in the first sample frame and a second sample pixel in the second sample frame, the first sample pixel corresponding to the second sample pixel.

### Claim 10 (depends on 9)

The scene flow estimation method of claim 9, wherein the obtaining of the trained AI network by iteratively performing the training operation comprises:
obtaining a sample motion embedding feature and a sample non-occluded-category label embedding feature corresponding to the sample frame pair by applying the AI network to be trained to the sample frame pair;
obtaining an estimated scene flow value corresponding to the sample frame pair, based on the sample motion embedding feature and the sample non-occluded-category label embedding feature;
obtaining an estimated non-occluded-category label mask value corresponding to the sample frame pair, based on the sample non-occluded-category label embedding feature;
determining a first training loss, based on the predetermined GT scene flow value and the estimated scene flow value corresponding to the sample frame pair;
determining a second training loss, based on the predetermined GT non-occluded-category label mask value and the estimated non-occluded-category label mask value corresponding to the sample frame pair;
determining a combined training loss based on the first training loss and the second training loss; and
adjusting a model parameter of the AI network to be trained, based on the combined training loss.

### Claim 11 (depends on 10)

The scene flow estimation method of claim 10, wherein the obtaining of the estimated non-occluded-category label mask value corresponding to the sample frame pair, based on the sample non-occluded-category label embedding feature comprises:
determining an average non-occluded-category label feature of pixels corresponding to a category comprised in the sample frame pair, based on the GT non-occluded-category label mask value and the sample non-occluded-category label embedding feature; and
based on a difference between a sample non-occluded-category label embedding feature corresponding to a pixel comprised in the first sample frame and an average non-occluded-category label feature of pixels corresponding to a category to which the pixel comprised in the first sample frame belongs, obtaining the estimated non-occluded-category label mask value corresponding to the category of the pixel comprised in the first sample frame data.

### Claim 12 (depends on 9)

The scene flow estimation method of claim 9, wherein the GT non-occluded-category label mask value corresponding to the sample frame pair is determined based on:
obtaining an object instance segmentation result for object instances comprised in the sample frame pair;
determining a first optical error between matched pixels of a pixel pair in the sample frame pair, based on the GT scene flow value corresponding to the sample frame pair;
determining a GT non-occluded-category label mask value corresponding to each of the object instances comprised in the sample frame pair, based on the first optical error and the object instance segmentation result; and
obtaining the GT non-occluded-category label mask value corresponding to the sample frame pair, based on the GT non-occluded-category label mask value corresponding to each of the object instances comprised in the sample frame pair.

### Claim 13 (depends on 12)

The scene flow estimation method of claim 12, wherein the GT non-occluded-category label mask value corresponding to the sample frame pair is determined based on:
obtaining a GT motion field value corresponding to the sample frame pair;
determining a second optical error and a depth error between the matched pixels of the pixel pair in the sample frame pair, based on the GT motion field value;
determining a GT non-occluded-category label mask value corresponding to a non-occluded background region of the sample frame pair, based on the second optical error and the depth error; and
obtaining the GT non-occluded-category label mask value corresponding to the sample frame pair by fusing the GT non-occluded-category label mask value corresponding to the non-occluded background region and the GT non-occluded-category label mask value corresponding to each of the object instances comprised in the sample frame pair.

### Claim 14 (depends on 1)

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the scene flow estimation method of claim 1.

### Claim 15

A scene flow estimation device, comprising:
one or more processors configured to input a frame pair into an artificial intelligence (AI) network and obtain a motion embedding feature and a non-occluded-category label embedding feature corresponding to the frame pair, and estimate a scene flow corresponding to the frame pair based on the motion embedding feature and the non-occluded-category label embedding feature,
wherein the frame pair comprises a first frame and a second frame, the first frame comprising a first color image and a first depth image, and the second frame comprising a second color image and a second depth image,
wherein the non-occluded-category label embedding feature comprises category information of an object corresponding to a pixel pair in the frame pair, and
wherein the pixel pair comprises a first pixel of the first frame and a second pixel of the second frame, the first pixel corresponding to the second pixel.

### Claim 16 (depends on 15)

The scene flow estimation device of claim 15, wherein the one or more processors are further configured to:
obtain a motion field corresponding to the frame pair;
obtain the motion embedding feature and the non-occluded-category label embedding feature corresponding to the frame pair by inputting the motion field into the AI network;
fuse the motion embedding feature with the non-occluded-category label embedding feature to obtain a fused embedding feature;
obtain a target motion field by updating the motion field based on the fused embedding feature; and
estimate the scene flow corresponding to the frame pair based on the target motion field.

### Claim 17 (depends on 16)

The scene flow estimation device of claim 16, wherein the one or more processors are configured to:
determine a neighboring point set comprising pixels in the second frame that correspond to a target pixel in the first frame;
determine a matching level between the target pixel and pixels comprised in the determined neighboring point set based on a fused embedding feature corresponding to the target pixel and fused embedding features of the pixels in the neighboring point set; and
obtain the target motion field by updating the motion field based on the matching level.

### Claim 18 (depends on 16)

The scene flow estimation device of claim 16, wherein the one or more processors are further configured to:
obtain a weight that adjusts the motion field based on a similarity level between pixels in the first color image; and
estimate the scene flow corresponding to the frame pair based on the target motion field obtained by applying the weight to the motion field.

### Claim 19 (depends on 18)

The scene flow estimation device of claim 18, wherein the one or more processors are further configured to:
obtain a first correlation between the pixels of the first color image by inputting the color first image into an attention encoder of the AI network; and
determine a first weight corresponding to the first color image based on the first correlation.

### Claim 20 (depends on 19)

The scene flow estimation device of claim 19, wherein the one or more processors are further configured to:
obtain a second correlation between pixels of the second color image by inputting the second color image into the attention encoder;
determine a second weight corresponding to the second color image based on the second correlation; and
obtain a fused weight that adjusts the motion field by fusing the first weight and the second weight.
