# US 2021/0174203 A1 · Neural network device, operation method thereof, and neural network system including the same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174203 A1
- Document ID: US20210174203A1
- Status: Published application
- Kind: A1
- Application number: 17/033,054
- Filed: 2020-09-25
- Published: 2021-06-10
- Earliest claimed: 2019-12-10
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: CHANGGWUN LEE; KYOUNGYOUNG KIM; BYEOUNGSU KIM; JAEGON KIM; SANGHYUCK HA
- Web page: https://patentgrep.com/patent/US20210174203A1

## CPC

- G06N 3/08: Learning methods
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/063: using electronic means
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The inventive concept relates to a neural network device, an operation method of the neural network device, and a neural network system including the neural network device, the operation method of the neural network device includes, receiving raw data from a sensor, converting the raw data to a feature map suitable for a first hidden layer among a plurality of hidden layers of a target neural network using a sensor response network adaptively trained to the sensor, inputting the feature map to the first hidden layer of the target neural network, and driving the neural network on a basis of the feature map.

## Claims

### Claim 1

An operation method of a neural network device comprising:
receiving raw data from a sensor;
converting the raw data to features suitable for a first hidden layer among a plurality of hidden layers of a target neural network using a sensor response network adaptively trained to the sensor;
inputting the features to the first hidden layer of the target neural network; and
driving the target neural network using the features.

### Claim 2 (depends on 1)

The operation method of claim 1, wherein the converting of the raw data to the features comprises generating feature maps as many as a number of channels of the first hidden layer based on the raw data.

### Claim 3 (depends on 1)

The operation method of claim 1, wherein the target neural network is trained in advance on a basis of a sensing data set, and a configuration of the target neural network is not changed in a process in which the sensor response network is adaptively trained to the sensor.

### Claim 4 (depends on 1)

The operation method of claim 1, further comprising generating the sensor response network through reinforcement learning based on a parameter of the sensor.

### Claim 5 (depends on 4)

The operation method of claim 4, wherein the parameter comprises at least one among a pattern of the raw data generated from the sensor, physical characteristics of the sensor, and a type of the sensor.

### Claim 6 (depends on 5)

The operation method of claim 5, wherein the sensor includes an image sensor, and the physical characteristics of the sensor comprise at least one among a focal length of a lens provided in the sensor, a degree of lens distortion, a pixel pitch of a sensing array provided in the sensor, and a dynamic range of the sensor.

### Claim 7 (depends on 4)

The operation method of claim 4, wherein the reinforcement learning generates a reward and adjusts the sensor response network based on the reward, wherein the reward indicates a suitability of the features for the target neural network or an accuracy of the target neural network.

### Claim 8 (depends on 4)

The operation method of claim 4, wherein the generating of the sensor response network comprises determining the first hidden layer to which the features are to be provided from among the plurality of hidden layers of the target neural network.

### Claim 9 (depends on 4)

The operation method of claim 4, wherein the reinforcement learning is performed as a background operation of the neural network device.

### Claim 10 (depends on 1)

The operation method of claim 1, further comprising:
receiving processed data generated by processing the raw data, wherein the sensor is an image sensor;
inputting the processed data to an input layer of the target neural network; and
driving the neural network using the processed data.

### Claim 11 (depends on 10)

The operation method of claim 10, wherein the receiving of the raw data and the receiving of the processed data are selectively performed on a basis of whether to the processed data to be presented on a display.

### Claim 12 (depends on 1)

The operation method of claim 1, wherein the target neural network comprises a neural network trained to classify input data into one of a plurality of classes.

### Claim 13

A neural network device comprising:
a memory configured to store at least one program; and
a processor configured to receive raw data from a sensor, execute the at least one program to perform operations of a sensor response network trained adaptively to the sensor, convert the raw data to a feature map to be input to a hidden layer among a plurality of layers of a target neural network, and perform operations of the target neural network using the feature map to generate a neural network processing result.

### Claim 14 (depends on 13)

The neural network device of claim 13, wherein the sensor response network is generated through reinforcement learning based on a parameter of the sensor.

### Claim 15

(canceled)

### Claim 16 (depends on 14)

The neural network device of claim 14, wherein the reinforcement learning executes the target neural network on a basis of the feature map to generate a result, determines a reward imparted by the result, and adjusts the sensor response network based on the reward.

### Claim 17 (depends on 16)

The neural network device of claim 16, wherein the reward is a suitability of the feature map for the target neural network or an accuracy of the result.

### Claim 18 (depends on 14)

The neural network device of claim 14, wherein the target neural network is trained in advance on a basis of a sensing data set, and parameters of the target neural network are stored in the memory.

### Claim 19 (depends on 14)

The neural network device of claim 14, wherein, when the reinforcement learning is performed, the processor converts processed data generated on basis of the raw data to a data format suitable for an input layer among the plurality of layers of the target neural network and performs operations of the target neural network on a basis of the data format.

### Claim 20 (depends on 14)

The neural network device of claim 14, wherein the processor comprises an arithmetic circuit comprising processing units configured to operate in parallel.

### Claim 21

A neural network system comprising:
an image sensor configured to generate a Bayer pattern on a basis of a received optical signal;
an image signal processor configured to convert the Bayer pattern received from the image sensor to red-green-blue (RGB) data;
a neural network device configured to selectively drive a sensor response network and a target neural network on a basis of the Bayer pattern, or to drive the target neural network on a basis of the RGB data; and
a display configured to display an image using the RGB data.

### Claim at position 22

22-25. (canceled)
