# US 2023/0267632 A1 · Stereo matching method and image processing device performing same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267632 A1
- Document ID: US20230267632A1
- Status: Published application
- Kind: A1
- Application number: 18/309,234
- Filed: 2023-04-28
- Published: 2023-08-24
- Earliest claimed: 2020-10-30
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Taehwa HONG; Dowan KIM; Kimin KIM
- Web page: https://patentgrep.com/patent/US20230267632A1

## CPC

- G06T 7/593: from stereo images
- G06T 7/73: using feature-based methods
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/327: Calibration thereof
- H04N 13/332: Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
- G06T 2207/10012: Stereo images
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An image processing device is provided. The image processing device includes a camera configured to obtain a stereo image, an eye-tracking sensor configured to obtain gaze information of a user, a memory storing one or more instructions, and at least one processor. The at least one processor is configured to, by executing one or more instructions, extract feature points from the stereo image and generate gaze coordinate information in which gaze coordinates corresponding to the gaze information of the user are accumulated on the stereo image, and perform stereo matching based on the feature points and the gaze coordinate information.

## Claims

### Claim 1

An image processing device comprising:
a camera configured to obtain a stereo image;
an eye-tracking sensor configured to obtain gaze information of a user;
a memory storing one or more instructions; and
at least one processor configured to execute the one or more instructions,
wherein the at least one processor is configured to:
extract feature points from the stereo image and generate gaze coordinate information in which gaze coordinates corresponding to the gaze information of the user are accumulated on the stereo image, and
perform stereo matching based on the feature points and the gaze coordinate information.

### Claim 2 (depends on 1)

The image processing device of claim 1, wherein the at least one processor is further configured to execute the one or more instructions to:
perform the stereo matching by restricting a search range of a second image to be a certain range from a second gaze coordinate corresponding to a first gaze coordinate near a first feature point of a first image.

### Claim 3 (depends on 2)

The image processing device of claim 2, wherein the first gaze coordinate is a gaze coordinate closest to a coordinate of the first feature point, from among the gaze coordinates forming the gaze coordinate information.

### Claim 4 (depends on 1)

The image processing device of claim 1, wherein the at least one processor is further configured to execute the one or more instructions to:
obtain a second feature point of a second image corresponding to a first feature point, based on a restricted range on an epipolar line of the second image corresponding to a coordinate of the first feature point of a first image, and on a certain range from a second gaze coordinate of the second image corresponding to a first gaze coordinate near the first feature point.

### Claim 5 (depends on 1)

The image processing device of claim 1, wherein the at least one processor is further configured to execute the one or more instructions to:
identify a first gaze coordinate near a first feature point of a first image;
determine a search range of a second image based on a result of the identification; and
obtain a second feature point of the second image corresponding to the first feature point within the search range.

### Claim 6 (depends on 5)

The image processing device of claim 5, wherein the at least one processor is further configured to execute the one or more instructions to:
when there is the first gaze coordinate, determine the search range to be within a certain range from a second gaze coordinate in the second image corresponding to the first gaze coordinate; and
when there is no first gaze coordinate, determine the search range to be a predefined range.

### Claim 7 (depends on 5)

The image processing device of claim 5, wherein the at least one processor is further configured to execute the one or more instructions to:
obtain as the second feature point, from among feature points within the search range, a feature point having highest similarity to feature information of the first feature point.

### Claim 8 (depends on 1)

The image processing device of claim 1, wherein the at least one processor is further configured to:
execute the one or more instructions to obtain a coordinate pair of gaze coordinates from the stereo image, based on the gaze information obtained by using the eye-tracking sensor, and accumulate, in the memory, three-dimensional (3D) gaze coordinates obtained from the coordinate pair; and
generate the gaze coordinate information by re-projecting the accumulated 3D gaze coordinates onto the stereo image.

### Claim 9 (depends on 1)

The image processing device of claim 1, wherein the at least one processor is further configured to execute the one or more instructions to perform a process of generating the gaze coordinate information in parallel with a process of extracting the feature points.

### Claim 10

A stereo matching method comprising:
obtaining a stereo image by using a camera;
extracting feature points from the stereo image;
generating gaze coordinate information in which gaze coordinates corresponding to gaze information of a user are accumulated on the stereo image; and
performing stereo matching based on the feature points and the gaze coordinate information.

### Claim 11 (depends on 10)

The stereo matching method of claim 10, wherein the performing of the stereo matching comprises:
performing the stereo matching by restricting a search range of a second image to a certain range from a second gaze coordinate corresponding to a first gaze coordinate near a first feature point of a first image.

### Claim 12 (depends on 10)

The stereo matching method of claim 10, wherein the performing of the stereo matching comprises:
obtaining a second feature point of a second image corresponding to a first feature point, based on a restricted range on an epipolar line of the second image corresponding to a coordinate of the first feature point of a first image, and on a certain range from a second gaze coordinate of the second image corresponding to a first gaze coordinate near the first feature point.

### Claim 13 (depends on 10)

The stereo matching method of claim 10, wherein the performing of the stereo matching comprises:
identifying a first gaze coordinate near a first feature point of a first image;
determining a search range of a second image, based on a result of the identification; and
obtaining a second feature point of the second image from the search range, the second feature point corresponding to the first feature point.

### Claim 14 (depends on 10)

The stereo matching method of claim 10, further comprising:
obtaining the gaze information by using an eye-tracking sensor;
obtaining a coordinate pair of gaze coordinates from the stereo image, based on the gaze information; and
accumulating three-dimensional (3D) gaze coordinates obtained from the coordinate pair,
wherein the generating of the gaze coordinate information comprises:
generating the gaze coordinate information by re-projecting the accumulated 3D gaze coordinates onto the stereo image.

### Claim 15

A computer-readable recording medium having recorded thereon a program to be executed by a computer, the computer-readable recording medium comprising instructions for:
obtaining a stereo image by using a camera;
extracting feature points from the stereo image;
generating gaze coordinate information in which gaze coordinates corresponding to gaze information of a user are accumulated on the stereo image; and
performing stereo matching, based on the feature points and the gaze coordinate information.
