# US 2019/0318196 A1 · Guided sparse feature matching via coarsely defined dense matches

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318196 A1
- Document ID: US20190318196A1
- Status: Published application
- Kind: A1
- Application number: 16/455,980
- Filed: 2019-06-28
- Published: 2019-10-17
- Earliest claimed: 2019-06-28
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Oscar Nestares; Niloufar Pourian
- Web page: https://patentgrep.com/patent/US20190318196A1

## CPC

- G06V 10/758: {Involving statistics of pixels or of feature values, e.g. histogram matching}
- G06V 10/757: {Matching configurations of points or features}
- G06T 2207/10012: Stereo images
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An example method is described herein. The method includes executing dense feature matching on an image pair that is down sampled to obtain a first set of feature correspondences for each pixel of the down sampled image pair. The method also includes calculating a neighborhood correspondence based on the first set of feature correspondences for each pixel in a first image of the image pair. Further, the method includes executing sparse feature matching on stereoscopic patch pairs from the image pair based on the neighborhood correspondence for each pixel to obtain correspondence estimates for each stereoscopic patch pair. Finally, the method includes refining the correspondence estimates for each stereoscopic patch pair to obtain a semi-dense set of feature correspondences by applying a geometric constraint to the correspondence estimates and retaining correspondences that satisfy the geometric constraint.

## Claims

### Claim 1

A method, comprising:
executing dense feature matching on an image pair that is down sampled to obtain a first set of feature correspondences for each pixel of the down sampled image pair;
calculating a neighborhood correspondence based on the first set of feature correspondences for each pixel in a first image of the image pair;
executing sparse feature matching on stereoscopic patch pairs from the image pair based on the neighborhood correspondence for each pixel to obtain correspondence estimates for each stereoscopic patch pair; and
refining the correspondence estimates for each stereoscopic patch pair to obtain a semi-dense set of feature correspondences by applying a geometric constraint to the correspondence estimates and retaining correspondences that satisfy the geometric constraint.

### Claim 2 (depends on 1)

The method of claim 1, wherein the geometric constraint is a multi-layer refinement stage that comprises a geometrical cost function that is defined by a distance from its initial coarse correspondence estimate, as well as the visual similarity score between the corresponding points.

### Claim 3 (depends on 1)

The method of claim 1, wherein the image pair is down sampled by a factor of four.

### Claim 4 (depends on 1)

The method of claim 1, wherein the geometric constraint comprises computing a number of encounters of a given feature in a certain radius surrounding it, and retaining features that consistently appear as a match via sparse matching of multi-scale overlapping stereo patches.

### Claim 5 (depends on 1)

The method of claim 1, wherein the geometric constraint comprises:
defining a single feature for each 1-pixel×1-pixel grid of each image of the image pair; and
retaining the feature that has the largest visual similarity with its match.

### Claim 6 (depends on 1)

The method of claim 1, wherein the sparse feature matching on stereoscopic patch pairs is enabled by a ratio test.

### Claim 7 (depends on 1)

The method of claim 1, wherein the dense feature matching on an image pair that is down sampled defines central points for a large group of overlapping windows with variable sizes across multiple scales to obtain the first set of feature correspondences.

### Claim 8 (depends on 1)

The method of claim 1, wherein the second level multi-scale sparse feature matching limits the number of outliers typically raised due to the absence of geometrical awareness in classical sparse feature matching approaches

### Claim 9 (depends on 1)

The method of claim 1, wherein the dense feature matching may be according to SIFT dense feature matching, local binary pattern (LBP), or any combination thereof.

### Claim 10 (depends on 1)

The method of claim 1, wherein the sparse feature matching is according to SIFT sparse feature matching, SURF feature matching, or any combination thereof.

### Claim 11

A system for game status detection, comprising:
a dense feature estimator to execute dense feature matching on an image pair that is down sampled to obtain a first set of feature correspondences for each pixel of the down sampled image pair and calculate a neighborhood correspondence based on the first set of feature correspondences for each pixel in a first image of the image pair;
a sparse feature estimator to execute sparse feature matching on stereoscopic patch pairs from the image pair based on the neighborhood correspondence for each pixel to obtain correspondence estimates for each stereoscopic patch pair; and
a refinement manager to refine the correspondence estimates for each stereoscopic patch pair to obtain a semi-dense set of feature correspondences by applying a geometric constraint to the correspondence estimates and retaining correspondences that satisfy the geometric constraint.

### Claim 12 (depends on 11)

The system of claim 11, wherein the geometric constraint is a multi-layer refinement stage that comprises a geometrical cost function that is defined by a distance from its initial coarse correspondence estimate, as well as the visual similarity score between the corresponding points.

### Claim 13 (depends on 11)

The system of claim 11, wherein the image pair is down sampled by a factor of four.

### Claim 14 (depends on 11)

The system of claim 11, wherein the geometric constraint comprises computing a number of encounters of a given feature in a certain radius surrounding it, and retaining features that consistently appear as a match via sparse matching of multi-scale overlapping stereo patches.

### Claim 15 (depends on 11)

The system of claim 11, wherein the geometric constraint comprises:
defining a single feature for each 1-pixel×1-pixel grid of each image of the image pair; and
retaining the feature that has the largest visual similarity with its match.

### Claim 16 (depends on 11)

The system of claim 11, wherein the sparse feature matching on stereoscopic patch pairs is enabled by a ratio test.

### Claim 17 (depends on 11)

The system of claim 11, wherein the dense feature matching on an image pair that is down sampled defines central points for a large group of overlapping windows with variable sizes across multiple scales to obtain the first set of feature correspondences.

### Claim 18 (depends on 11)

The system of claim 11, wherein the second level multi-scale sparse feature matching limits the number of outliers typically raised due to the absence of geometrical awareness in classical sparse feature matching approaches

### Claim 19 (depends on 11)

The system of claim 11, wherein the dense feature matching may be according to SIFT dense feature matching, local binary pattern (LBP), or any combination thereof.

### Claim 20 (depends on 11)

The system of claim 11, wherein the sparse feature matching is according SIFT sparse feature matching, SURF feature matching, or any combination thereof.

### Claim 21

At least one non-transitory computer-readable medium, comprising instructions to direct a processor to:
execute dense feature matching on an image pair that is down sampled to obtain a first set of feature correspondences for each pixel of the down sampled image pair;
calculate a neighborhood correspondence based on the first set of feature correspondences for each pixel in a first image of the image pair;
execute sparse feature matching on stereoscopic patch pairs from the image pair based on the neighborhood correspondence for each pixel to obtain correspondence estimates for each stereoscopic patch pair; and
refine the correspondence estimates for each stereoscopic patch pair to obtain a semi-dense set of feature correspondences by applying a geometric constraint to the correspondence estimates and retaining correspondences that satisfy the geometric constraint.

### Claim 22 (depends on 21)

The computer readable medium of claim 21, wherein the geometric constraint is a multi-layer refinement stage that comprises a geometrical cost function that is defined by a distance from its initial coarse correspondence estimate, as well as the visual similarity score between the corresponding points.

### Claim 23 (depends on 21)

The computer readable medium of claim 21, wherein the geometric constraint comprises computing a number of encounters of a given feature in a certain radius surrounding it, and retaining features that consistently appear as a match via sparse matching of multi-scale overlapping stereo patches.

### Claim 24 (depends on 21)

The computer readable medium of claim 21, wherein the geometric constraint comprises:
defining a single feature for each 1-pixel×1-pixel grid of each image of the image pair; and
retaining the feature that has the largest visual similarity with its match.

### Claim 25 (depends on 21)

The computer readable medium of claim 21, wherein the sparse feature matching on stereoscopic patch pairs is enabled by a ratio test.