# US 2021/0174138 A1 · Device and method with sensor-specific image recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174138 A1
- Document ID: US20210174138A1
- Status: Published application
- Kind: A1
- Application number: 16/856,221
- Filed: 2020-04-23
- Published: 2021-06-10
- Earliest claimed: 2019-12-06
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Jiho CHOI; Solae LEE; Hana LEE; Youngjun KWAK; Byung In YOO; Yong-Il LEE
- Web page: https://patentgrep.com/patent/US20210174138A1

## CPC

- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 20/80: Recognising image objects characterised by unique random patterns
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method with image recognition includes: extracting, using a feature extraction layer, feature data from an input image received by an image sensor; and outputting a recognition result of an object appearing in the input image, by applying a fixed mask and a variable mask to the extracted feature data, wherein the variable mask is adjusted in response to the extracted feature data.

## Claims

### Claim 1

A method with image recognition, comprising:
extracting, using a feature extraction layer, feature data from an input image received by an image sensor; and
outputting a recognition result of an object appearing in the input image, by applying a fixed mask and a variable mask to the extracted feature data,
wherein the variable mask is adjusted in response to the extracted feature data.

### Claim 2 (depends on 1)

The method of claim 1, wherein the outputting of the recognition result comprises:
calculating first recognition data from the extracted feature data, by the applying of the fixed mask to the extracted feature data;
calculating second recognition data from the extracted feature data, by the applying of the variable mask to the extracted feature data; and
determining the recognition result based on the first recognition data and the second recognition data.

### Claim 3 (depends on 2)

The method of claim 2, wherein the calculating of the first recognition data comprises:
generating a generic feature map related to an object region of interest by the applying of the fixed mask to the extracted feature data; and
calculating the first recognition data from the generic feature map.

### Claim 4 (depends on 2)

The method of claim 2, wherein the calculating of the second recognition data comprises:
generating a sensor-specific feature map related to a region of interest of the image sensor by the applying of the variable mask to a target feature map corresponding to the extracted feature data; and
calculating the second recognition data from the sensor-specific feature map.

### Claim 5 (depends on 4)

The method of claim 4, wherein the generating of the sensor-specific feature map comprises:
applying a corresponding value in the variable mask for an individual value of the target feature map.

### Claim 6 (depends on 2)

The method of claim 2, further comprising:
calculating, from the extracted feature data, third recognition data using a softmax function and a fully connected layer,
wherein the determining of the recognition result comprises:
determining the recognition result further based on the third recognition data in addition to the first recognition data and the second recognition data.

### Claim 7 (depends on 1)

The method of claim 1, wherein the outputting of the recognition result comprises:
adjusting one or more values of the variable mask based on the extracted feature data, using at least a portion of a sensor-specific layer including the variable mask.

### Claim 8 (depends on 7)

The method of claim 7, wherein the adjusting of the one or more values of the variable mask comprises:
determining a value of the variable mask using a softmax function from a result of multiplication between a transposed query feature map and a key feature map corresponding to a result of applying convolutional filtering to the extracted feature data.

### Claim 9 (depends on 1)

The method of claim 1, wherein the outputting of the recognition result comprises:
determining, to be the recognition result, a weighted sum of first recognition data based on the fixed mask and second recognition data based on the variable mask.

### Claim 10 (depends on 9)

The method of claim 9, wherein the determining of the weighted sum to be the recognition result comprises:
applying, to the second recognition data, a weight greater than a weight applied to the first recognition data.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
receiving, from an external server, in response to an update command, a parameter of a sensor-specific layer including the variable mask; and
updating a sensor-specific layer with the received parameter.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
requesting, from the external server, a sensor-specific parameter corresponding to an optical characteristic similar or identical to an optical characteristic of the image sensor.

### Claim 13 (depends on 11)

The method of claim 11, further comprising:
maintaining a value of the fixed mask while a parameter of the sensor-specific layer is updated.

### Claim 14 (depends on 1)

The method of claim 1, wherein the outputting of the recognition result comprises:
calculating the recognition result based on the fixed mask and a plurality of variable masks.

### Claim 15 (depends on 14)

The method of claim 14, wherein a parameter of a sensor-specific layer including one variable mask among the plurality of variable masks is different from a parameter of another sensor-specific layer including another variable mask among the plurality of variable masks.

### Claim 16 (depends on 1)

The method of claim 1, wherein the outputting of the recognition result comprises:
generating, as the recognition result, authenticity information indicating whether the object is a real object or a fake object.

### Claim 17 (depends on 1)

The method of claim 1, further comprising:
granting an authority based on the recognition result; and
allowing an access to either one or both of an operation of an electronic terminal and data of the electronic terminal, based on the authority.

### Claim 18 (depends on 1)

The method of claim 1, wherein the outputting of the recognition result comprises:
visualizing the recognition result on a display after the recognition result is generated.

### Claim 19 (depends on 1)

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1.

### Claim 20

A device with image recognition, comprising:
an image sensor configured to receive an input image; and
a processor configured to:
extract feature data from the input image using a feature extraction layer; and
output a recognition result of an object appearing in the input image, by applying a fixed mask and a variable mask to the extracted feature data,
wherein the variable mask is adjusted in response to the extracted feature data.

### Claim 21 (depends on 20)

The device of claim 20, wherein the outputting of the recognition result comprises:
calculating first recognition data from the extracted feature data, by the applying of the fixed mask to the extracted feature data;
calculating second recognition data from the extracted feature data, by the applying of the variable mask to the extracted feature data; and
determining the recognition result based on a sum of the first recognition data and the second recognition data.

### Claim 22 (depends on 21)

The device of claim 21, wherein the sum is determined by applying a weight to the second recognition data that is greater than a weight applied to the first recognition data.

### Claim 23 (depends on 21)

The device of claim 21, wherein the calculating of the first recognition data comprises:
generating a generic feature map related to an object region of interest by the applying of the fixed mask to the extracted feature data; and
calculating the first recognition data from the generic feature map, and
wherein the calculating of the second recognition data comprises:
generating a sensor-specific feature map related to a region of interest of the image sensor by the applying of the variable mask to a target feature map corresponding to the extracted feature data; and
calculating the second recognition data from the sensor-specific feature map.

### Claim 24

A system with image recognition, comprising:
an image recognition device configured to:
extract feature data from a received input image using a feature extraction layer; and
output a recognition result of an object appearing in the input image, by applying a fixed mask and a variable mask to the extracted feature data,
wherein the variable mask is included in a sensor-specific layer of the image recognition device, and is adjusted in response to the extracted feature data; and
a server configured to distribute a parameter of an additionally trained sensor-specific layer to the image recognition device, in response to either one or both of an update request from the image recognition device and completion of additional training for a sensor-specific layer of a recognition model of the server,
wherein the image recognition device is configured to update the sensor-specific layer of the image recognition device based on the distributed parameter.

### Claim 25 (depends on 24)

The system of claim 24, wherein the server is further configured to distribute the parameter of the additionally trained sensor-specific layer to another image recognition device comprising an image sensor determined to be similar to an image sensor of the image recognition device.
