# US 12,118,764 B2 · Method and apparatus for processing image based on partial images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,764 B2
- Document ID: US12118764B2
- Status: Granted
- Kind: B2
- Application number: 17/177,702
- Filed: 2021-02-17
- Published: 2022-03-03
- Granted: 2024-10-15
- Earliest claimed: 2020-08-31
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Jaehyoung Yoo; Changyong Son; Dongwook Lee; Jaejoon Han; Chang Kyu Choi
- Web page: https://patentgrep.com/patent/US12118764B2

## CPC

- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06F 18/2413: based on distances to training or reference patterns
- G06F 18/253: {of extracted features}
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/806: {of extracted features}
- G06V 10/82: using neural networks
- H04N 23/611: where the recognised objects include parts of the human body
- H04N 23/651: {for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera}
- H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method and apparatus for processing an image based on partial images. The method includes extracting a feature of a current partial processing region of an input image frame by inputting pixel data of the current partial processing region into a convolutional neural network (CNN), updating a hidden state of a recurrent neural network (RNN) for a context between the current partial processing region and at least one previous partial processing region by inputting the extracted feature into the RNN, and generating an image processing result for the input image frame based on the updated hidden state.

## Claims

### Claim 1

A method of processing an image, the method comprising:
extracting a feature of a current partial processing region of an input image frame by inputting pixel data of the current partial processing region into a convolutional neural network (CNN) being a multi-head CNN (MCNN);
updating a hidden state of a recurrent neural network (RNN) for a context between the current partial processing region and at least one previous partial processing region of the input image frame by inputting the extracted feature into the RNN; and
generating an image processing result for the input image frame based on the updated hidden state,
wherein the extracting comprises extracting a plurality of intermediate features of the current partial processing region by inputting the pixel data of the current partial processing region into each of plural heads of the MCNN, and
generating an encoded feature of the current partial processing region by fusing the plurality of extracted intermediate features.

### Claim 2 (depends on 1)

The method of claim 1, wherein the updating comprises converting the extracted feature to a one-dimensional vector and inputting the one-dimensional vector into the RNN.

### Claim 3 (depends on 2)

The method of claim 2, wherein the one-dimensional vector comprises at least one of a channel direction vector and a width direction vector.

### Claim 4 (depends on 1)

The method of claim 1, wherein the generating comprises:
generating final context data based on the updated hidden state in response to the current partial processing region corresponding to a final partial processing region of the input image frame; and
generating the image processing result based on the final context data.

### Claim 5 (depends on 1)

The method of claim 1, wherein the image processing result indicates whether there is a target object in the input image frame.

### Claim 6 (depends on 5)

The method of claim 5, wherein the target object is a human.

### Claim 7 (depends on 5)

The method of claim 5, wherein the target object is a human face.

### Claim 8 (depends on 1)

The method of claim 1, wherein the generating comprises:
generating intermediate context data based on the updated hidden state in response to the current partial processing region not corresponding to a final partial processing region of the input image frame; and
generating an intermediate detection result based on the intermediate context data.

### Claim 9 (depends on 1)

The method of claim 1, wherein each partial processing region including the current partial processing region corresponds to a pixel line group for image signal processing (ISP) of an image sensor.

### Claim 10 (depends on 1)

The method of claim 1, wherein the extracting, the updating, and the generating are performed in a low power mode.

### Claim 11 (depends on 1)

The method of claim 1, wherein the CNN is trained in advance using object position information related to each partial processing region of a training image frame.

### Claim 12 (depends on 1)

The method of claim 1, wherein the updated hidden state represents a hidden vector corresponding to the context.

### Claim 13 (depends on 1)

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1.

### Claim 14

A method of processing an image, the method comprising:
extracting a feature of a current partial processing region of an input image frame by inputting pixel data of the current partial processing region into a convolutional neural network (CNN);
updating a hidden state of a recurrent neural network (RNN) for a context between the current partial processing region and at least one previous partial processing region of the input image frame by inputting the extracted feature into the RNN; and
generating an image processing result for the input image frame based on the updated hidden state,
wherein the generating comprises:
generating intermediate context data based on the updated hidden state in response to the current partial processing region not corresponding to a final partial processing region of the input image frame; and
generating an intermediate detection result based on the intermediate context data,
wherein in response to the intermediate detection result indicating that there is a target object in the input image frame, an image processing process on the input image frame is terminated without performing an additional image processing process on at least one subsequent partial processing region.

### Claim 15

An apparatus for processing an image, the apparatus comprising:
a processor configured to execute instructions; and
a memory to store the instructions,
wherein the execution of the instructions by the processor configures the processor to:
extract a feature of a current partial processing region of an input image frame by inputting pixel data of the current partial processing region into a convolutional neural network (CNN) being a multi-head CNN (MCNN);
update a hidden state of a recurrent neural network (RNN) for a context between the current partial processing region and at least one previous partial processing region of the input image frame by inputting the extracted feature into the RNN; and
generate an image processing result for the input image frame based on the updated hidden state, and
wherein the processor is further configured to extract a plurality of intermediate features of the current partial processing region by inputting the pixel data of the current partial processing region into each of plural heads of the MCNN, and generate an encoded feature of the current partial processing region by fusing the plurality of extracted intermediate features.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the processor is further configured to generate final context data based on the updated hidden state in response to the current partial processing region corresponding to a final partial processing region of the input image frame, and generate the image processing result based on the final context data.

### Claim 17 (depends on 15)

The apparatus of claim 15, wherein the processor is further configured to generate intermediate context data based on the updated hidden state in response to the current partial processing region not corresponding to a final partial processing region of the input image frame, and generate an intermediate detection result based on the intermediate context data.

### Claim 18 (depends on 15)

The apparatus of claim 15, wherein each partial processing region including the current partial processing region corresponds to a pixel line group for image signal processing (ISP) of an image sensor.

### Claim 19 (depends on 15)

The apparatus of claim 15,
wherein, for the update of the hidden state of the RNN, the processor is configured to:
determine whether the current partial processing region of the input image frame is a final partial processing region of the input image frame,
in a case in which the current partial processing region is not the final partial processing region, repeatedly update the hidden state of the RNN through performance of the inputting of the extracted feature of the current partial processing region into the RNN, a generation of intermediate context data based on the updated hidden state, and a generation of an intermediate detection result based on the intermediate context data until the current partial processing region is determined to be the final partial processing region; and
in a case in which the current partial processing region is the final partial processing region, update the hidden state of the RNN by inputting the extracted feature of the current partial processing region into the RNN, generate final context data based on the updated hidden state, and
wherein, for the generation of the image processing result, the processor is configured to generate the image processing result for the input image frame based on the final context data.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein, after each iteration in the case in which the current partial processing region is not the final partial processing region, the processor is configured to determine whether the intermediate detection result indicates the presence of a target object in the input image frame, and terminate any further processing of the input image frame in a case in which the intermediate detection result indicates the presence of the target object in the input image frame.

### Claim 21

An electronic device, comprising:
a camera configured to generate an input image frame;
a processor configured to execute instructions;
a memory to store the instructions,
wherein the execution of the instructions by the processor configures the processor to:
extract a feature of a current partial processing region of an input image frame by inputting pixel data of the current partial processing region into a convolutional neural network (CNN) being a multi-head CNN (MCNN);
update a hidden state of a recurrent neural network (RNN) for a context between the current partial processing region and at least one previous partial processing region of the input image frame by inputting the extracted feature into the RNN; and
generate an image processing result for the input image frame based on the updated hidden state, and
wherein the processor is further configured to extract a plurality of intermediate features of the current partial processing region by inputting the pixel data of the current partial processing region into each of plural heads of the MCNN, and generates an encoded feature of the current partial processing region by fusing the plurality of extracted intermediate features.

### Claim 22 (depends on 21)

The electronic device of claim 21, wherein the extracting, the updating, and the generating are performed in a low power mode, and
the electronic device is configured to:
wake up in response to a result of the generation of the image processing result being that a target object is detected in the input image frame; and
in response to the electronic device being woken up in response to the result, perform an operation associated with further image processing in a normal mode with respect to the target object.

### Claim 23 (depends on 22)

The electronic device of claim 22, wherein the low power mode comprises an always-on mode.

### Claim 24 (depends on 22)

The electronic device of claim 22, wherein the image processing comprises object detection, and
the operation associated with the image processing comprises any one or any combination of object tracking, object recognition, and unlocking.

### Claim 25 (depends on 21)

The electronic device of claim 21, wherein, for the generation of the image processing result, the processor is configured to:
generate context data based on the updated hidden state;
generate a detection result based on the context data;
determine whether the detection result indicates the presence of target data in the input data frame;
in a case in which the detection result indicates the presence of the target data in the input data frame, terminate any further processing of the input data frame; and
in a case in which the detection result does not indicate the presence of the target data in the input data frame, continue processing of the input data frame with respect to a partial processing region of the input data frame that is subsequent to the current partial processing region.

### Claim 26 (depends on 25)

The apparatus of claim 25, wherein, in the case in which the detection result does not indicate the presence of the target data in the input data frame and the current partial processing region is a final partial processing region of the input data frame, the processor is configured to generate an image processing result for the input data frame based on the context data.