# US 2021/0174134 A1 · Methods and apparatus to match images using semantic features

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174134 A1
- Document ID: US20210174134A1
- Status: Published application
- Kind: A1
- Application number: 16/768,559
- Filed: 2018-03-01
- Published: 2021-06-10
- Earliest claimed: 2018-03-01
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Yimin ZHANG; Haibing REN; Wei HU; Ping GUO
- Web page: https://patentgrep.com/patent/US20210174134A1

## CPC

- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06F 18/211: Selection of the most significant subset of features
- G06V 10/757: {Matching configurations of points or features}
- G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and apparatus to match images using semantic features are disclosed. An example apparatus includes a semantic labeler to determine a semantic label for each of a first set of points of a first image and each of a second set of points of a second image; a binary robust independent element features (BRIEF) determiner to determine semantic BRIEF descriptors for a first subset of the first set of points and a second subset of the second set of points based on the semantic labels; and a point matcher to match first points of the first subset of points to second points of the second subset of points based on the semantic BRIEF descriptors.

## Claims

### Claim 1

An apparatus to match images using semantic features, the apparatus comprising:
a semantic labeler to determine a semantic label for respective ones of a first set of points of a first image and respective ones of a second set of points of a second image;
a binary robust independent element features (BRIEF) determiner to determine semantic BRIEF descriptors for a first subset of the first set of points and a second subset of the second set of points based on the semantic labels; and
a point matcher to match first points of the first subset of points to second points of the second subset of points based on the semantic BRIEF descriptors.

### Claim 2 (depends on 1)

The apparatus of claim 1, further including a mask generator to generate a semantic segmentation of the first and second images, the semantic labeler to identify semantic objects based on the semantic segmentation, the mask generator to determine the semantic labels for the first and second sets of points based on the semantic objects.

### Claim 3 (depends on 1)

The apparatus of claim 1, further including a key point determiner to select the first subset of points and the second subset of points based on an ordering of the points of the first and second sets of points, the ordering being based on where the points are located in the first and second images.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the BRIEF determiner is to determine the semantic BRIEF descriptors for the first and second sets of points based on semantic binary tests.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the BRIEF determiner is to determine a BRIEF descriptor of a first point of the first subset of points by:
selecting sample position pairs in a patch around a center of the first point;
determining whether semantic labels of the sample position pairs match;
setting semantic binary values based on the determinations of whether the semantic labels match; and
determining the BRIEF descriptor of the first point based on the semantic binary values.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the point matcher is to match the first points of the first set of points to the second points of the second set of points based on a Hamming distance between the semantic BRIEF descriptors of candidate point pairs between the first subset of points and the second subset of points.

### Claim 7 (depends on 6)

The apparatus of claim 6, wherein the point matcher is to determine candidate point pairs based on at least one of corresponding to a common matching label or corresponding to a same group.

### Claim 8 (depends on 7)

The apparatus of claim 7, wherein the semantic labeler is to determine the matching labels for the first subset of points and the second subset of points, the matching labels corresponding to at least one of semantic labels of the points or semantic labels of neighboring points.

### Claim 9 (depends on 7)

The apparatus of claim 7, further including a key point determiner to divide points in the first and second sets of points into groups based on positions of the points in the first and second images.

### Claim 10

A non-transitory computer readable storage medium comprising instructions which, when executed, cause a machine to at least:
determine a semantic label for respective ones of a first set of points of a first image and respective ones of a second set of points of a second image;
determine semantic BRIEF descriptors for a first subset of the first set of points and a second subset of the second set of points based on the semantic labels; and
match first points of the first subset of points to second points of the second subset of points based on the semantic BRIEF descriptors.

### Claim 11 (depends on 10)

The computer readable storage medium of claim 10, wherein the instructions cause the machine to:
create a semantic segmentation of the first and second images;
identify semantic objects based on the semantic segmentation; and
determine the semantic labels for the first and second sets of points based on the semantic objects.

### Claim 12 (depends on 10)

The computer readable storage medium of claim 10, wherein the instructions cause the machine to select the first subset of points and the second subset of points based on an ordering of the points of the first and second sets of points, the ordering being based on a location of the points within the first and second images.

### Claim 13 (depends on 10)

The computer readable storage medium of claim 10, wherein the instructions cause the machine to determine the semantic BRIEF descriptors by performing semantic binary tests with the first and second sets of points.

### Claim 14 (depends on 10)

The computer readable storage medium of claim 10, wherein the instructions cause the machine to determine a BRIEF descriptor of a first point of the first subset of points by:
selecting sample position pairs in a patch corresponding to the first point;
determining whether a union of semantic labels of the sample position pairs include a matching label;
determining semantic binary values based on whether the semantic labels match; and
determining the BRIEF descriptor of the first point based on the semantic binary values.

### Claim 15 (depends on 10)

The computer readable storage medium of claim 10, wherein the instructions cause the machine to match the first points of the first set of points to the second points of the second set of points based on a distance function corresponding to the semantic BRIEF descriptors of candidate point pairs between the first subset of points and the second subset of points.

### Claim 16 (depends on 15)

The computer readable storage medium of claim 15, wherein the instructions cause the machine to select candidate point pairs based on at least one of corresponding to a common matching label or corresponding to a same group.

### Claim 17 (depends on 16)

The computer readable storage medium of claim 16, wherein the instructions cause the machine to determine the matching labels for the first subset of points and the second subset of points, the matching labels corresponding to at least one of semantic labels of the points or semantic labels of neighboring points.

### Claim 18 (depends on 16)

The computer readable storage medium of claim 16, wherein the instructions cause the machine to group points in the first and second sets of points into groups based on positions of the points in the first and second images.

### Claim 19

A method to match images using semantic features, the method comprising:
determining a semantic label for respective ones of a first set of points of a first image and respective ones of a second set of points of a second image;
determining semantic BRIEF descriptors for a first subset of the first set of points and a second subset of the second set of points based on the semantic labels; and
matching first points of the first subset of points to second points of the second subset of points based on the semantic BRIEF descriptors.

### Claim 20 (depends on 19)

The method of claim 19, further including:
generating a semantic segmentation of the first and second images;
identify semantic objects based on the semantic segmentation; and
determining the semantic labels for the first and second sets of points based on the semantic objects.

### Claim 21 (depends on 19)

The method of claim 19, further including selecting the first subset of points and the second subset of points based on an ordering of the points of the first and second sets of points, the ordering being based on where the points are located in the first and second images.

### Claim 22 (depends on 19)

The method of claim 19, further including determining the semantic BRIEF descriptors for the first and second sets of points based on semantic binary tests.

### Claim 23 (depends on 19)

The method of claim 19, further including determining a BRIEF descriptor of a first point of the first subset of points by:
selecting sample position pairs in a patch around a center of the first point;
determining whether semantic labels of the sample position pairs match;
setting semantic binary values based on the determinations of whether the semantic labels match; and
determining the BRIEF descriptor of the first point based on the semantic binary values.

### Claim 24 (depends on 19)

The method of claim 19, further including matching the first points of the first set of points to the second points of the second set of points based on a Hamming distance between the semantic BRIEF descriptors of candidate point pairs between the first subset of points and the second subset of points.

### Claim 25 (depends on 24)

The method of claim 24, further including determining candidate point pairs based on at least one of corresponding to a common matching label or corresponding to a same group.
