# US 2023/0274577 A1 · Device and method with image matching

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274577 A1
- Document ID: US20230274577A1
- Status: Published application
- Kind: A1
- Application number: 18/312,997
- Filed: 2023-05-05
- Published: 2023-08-31
- Earliest claimed: 2018-02-12
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: SungUn PARK; Jihye KIM; Jaejoon HAN; Minsu KO; Seungju HAN; Jinwoo SON; Changyong SON
- Web page: https://patentgrep.com/patent/US20230274577A1

## CPC

- G06V 40/172: {Classification, e.g. identification}
- G06F 16/5838: {using colour}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 21/32: using biometric data, e.g. fingerprints, iris scans or voiceprints
- G06T 7/74: {involving reference images or patches}
- G06V 40/45: {Detection of the body part being alive}
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06T 2207/10024: Color image
- G06T 2207/10048: Infrared image
- G06T 2207/20024: Filtering details
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V 10/759: {Region-based matching}
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An image matching method includes extracting, from a first image of an object, a landmark patch including a landmark point of the object; extracting, from a second image of the object, a target patch corresponding to the landmark patch; and determining a target point in the second image corresponding to the landmark point based on a matching between the landmark patch and the target patch.

## Claims

### Claim 1

A processor-implemented image matching method, comprising:
photographing, by a first image sensor, an object to acquire a first image;
photographing, by a second image sensor, the object to acquire a second image;
determining a landmark point of the first image based on a neural network model stored in a memory;
extracting, from the first image of the object, a landmark patch including the landmark point of the object;
extracting, from the second image of the object, a target patch corresponding to the landmark patch, wherein a size of the target patch is greater than a size of the landmark patch; and
determining a target point in the second image corresponding to the landmark point based on a matching between the landmark patch extracted from the first image and the target patch extracted from the second image.

### Claim 2 (depends on 1)

The image matching method of claim 1, wherein the photographing by the first image sensor comprises acquiring, using a color image sensor, a color image as the first image; and
the photographing by the second image sensor comprises acquiring, using an infrared (IR) image sensor, an IR image as the second image.

### Claim 3 (depends on 1)

The image matching method of claim 1, further comprising:
determining, based on the target point, whether the object is an anatomical structure of the user.

### Claim 4 (depends on 1)

The image matching method of claim 1, further comprising:
allowing access, through the user interface of a device, to one or more features of the device in response to the object being determined to be either one or both of a live anatomical structure and a recognized user.

### Claim 5 (depends on 1)

The image matching method of claim 1, wherein the extracting of the target patch comprises determining the target patch in the second image based on a location of the landmark patch in the first image.

### Claim 6 (depends on 1)

The image matching method of claim 1, wherein the extracting of the target patch comprises extracting the target patch in response to the landmark point being detected in a determined region of the first image.

### Claim 7 (depends on 6)

The image matching method of claim 6, wherein the extracting of the target patch comprises determining the determined region based on a difference between a field of view (FOV) of a first image sensor used to capture the first image and an FOV of a second image sensor used to capture the second image.

### Claim 8 (depends on 1)

The image matching method of claim 1, wherein the determining of the target point comprises:
retrieving, from the target patch, a partial region that matches the landmark patch; and
determining a center point of the retrieved partial region as the target point.

### Claim 9 (depends on 8)

The image matching method of claim 8, wherein the retrieving of the partial region comprises:
calculating a similarity level between the landmark patch and each of a plurality of partial regions of the target patch; and
determining a partial region with a highest calculated similarity level among the plurality of partial regions as the partial region that matches the landmark patch.

### Claim 10 (depends on 9)

The image matching method of claim 9, wherein the calculating of the similarity level comprises calculating a correlation level between values of pixels included in each of the plurality of partial regions of the target patch and pixels included in the landmark patch as the similarity level.

### Claim 11 (depends on 1)

The image matching method of claim 1, wherein:
the first image is a color image and the second image is an infrared (IR) image; and
the extracting of the landmark patch comprises:
selecting a channel image from the first image; and
extracting the landmark patch from the selected channel image.

### Claim 12 (depends on 1)

The image matching method of claim 1, wherein:
the first image includes a plurality of channel images; and
the extracting of the landmark patch comprises extracting the landmark patch from a channel image with a minimum wavelength difference between the channel image and the second image among the plurality of channel images.

### Claim 13 (depends on 1)

The image matching method of claim 1, wherein:
the extracting of the landmark patch comprises extracting the landmark patch from the first image for each of a plurality of landmarks of the object,
the extracting of the target patch comprises extracting the target patch from the second image for each of the plurality of landmarks, and
the determining of the target point comprises determining the target point corresponding to the landmark point for each of the plurality of landmarks.

### Claim 14 (depends on 1)

The image matching method of claim 1, wherein the determining of the landmark point comprises:
determining an object region corresponding to the object in the first image;
identifying the landmark point of the object in the object region.

### Claim 15 (depends on 1)

The image matching method of claim 1, further comprising:
matching the first image and the second image based on the landmark point of the first image and the target point of the second image.

### Claim 16 (depends on 1)

The image matching method of claim 1, wherein the extracting of the target patch comprises determining the target patch in the second image based on a location of the landmark patch in the first image and a distance between an image matching device and the object.

### Claim 17 (depends on 1)

The image matching method of claim 1, further comprising:
determining a point corresponding to a remaining landmark in the second image based on the target point associated with one of a plurality of landmarks of the object in response to the plurality of landmarks of the object being detected in the first image.

### Claim 18 (depends on 1)

The image matching method of claim 1, further comprising:
recognizing the object, in the second image, based on the target point; and
authenticating a user based on the recognized object,
wherein the landmark point comprises both ends of an eye of the user.

### Claim 19 (depends on 1)

The image matching method of claim 1, further comprising:
verifying a liveness of the object, in the second image, based on the target point.

### Claim 20

An image matching device, comprising:
a first image sensor configured to capture an object for acquiring a first image;
a second image sensor configured to capture the object for acquiring a second image;
a memory configured to store a neural network model; and
one or more processors configured to:
determine a landmark point of the first image based on the neural network model stored in the memory,
obtain the first image of the object and the second image of the object;
extract, from the first image, a landmark patch including the landmark point of the object,
extract, from the second image, a target patch corresponding to the landmark patch, and
determine a target point in the second image corresponding to the landmark point based on a matching between the landmark patch extracted from the first image and the target patch extracted from the second image, and
wherein a size of the target patch is greater than a size of the landmark patch.
