# US 10,769,485 B2 · Framebuffer-less system and method of convolutional neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,485 B2
- Document ID: US10769485B2
- Status: Granted
- Kind: B2
- Application number: 16/012,133
- Filed: 2018-06-19
- Published: 2019-12-19
- Granted: 2020-09-08
- Earliest claimed: 2018-06-19
- Assignees: Himax Technologies Limited
- Applicants: Himax Technologies Limited
- Inventors: Der-Wei Yang
- Web page: https://patentgrep.com/patent/US10769485B2

## CPC

- G06T 1/00: General purpose image data processing
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 5/046: Forward inferencing; Production systems
- G06T 3/4046: using neural networks
- G06T 7/11: Region-based segmentation
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
- G06V 10/82: using neural networks
- G06V 10/955: {using specific electronic processors}
- G06N 3/048: Activation functions
- G06N 3/063: using electronic means
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A framebuffer-less system of convolutional neural network (CNN) includes a region of interest (ROI) unit that extracts features, according to which a region of interest in an input image frame is generated; a convolutional neural network (CNN) unit that processes the region of interest of the input image frame to detect an object; and a tracking unit that compares the features extracted at different times, according to which the CNN unit selectively processes the input image frame.

## Claims

### Claim 1

A framebuffer-less system of convolutional neural network (CNN), comprising:
a region of interest (ROI) unit that extracts features, according to which a region of interest in an input image frame is generated;
a convolutional neural network (CNN) unit that processes the region of interest of the input image frame to detect an object;
a tracking unit that compares features extracted at different times, according to which the CNN unit selectively processes the input image frame; and
a temporary storage for storing the features extracted by the ROI unit;
wherein the ROI unit adopts scan-line based technique and block-based scheme to find the region of interest in the input image frame, which is divided into a plurality of blocks of image;
wherein the ROI unit comprises:
a feature extractor that extracts the features from the input image frame; and
a classifier that makes decision whether to perform CNN for each block of image, thus generating a decision map, according to which the region of interest is determined.

### Claim 2 (depends on 1)

The system of claim 1, wherein the ROI unit generates block-based features, according to which decision of whether to perform CNN is made for each block of image.

### Claim 3 (depends on 1)

The system of claim 1, wherein the feature extractor generates shallow features of the blocks of image according to block-based histogram or frequency analysis.

### Claim 4 (depends on 1)

The system of claim 1, wherein the temporary storage comprises a first feature map storing features of a previous image frame, and a second feature map storing features of a current image frame.

### Claim 5 (depends on 4)

The system of claim 4, wherein the tracking unit compares the first feature and the second feature map, and accordingly updates the decision map.

### Claim 6 (depends on 1)

The system of claim 1, wherein the temporary storage comprises a sliding window storing a block of the input image frame.

### Claim 7 (depends on 1)

The system of claim 1, wherein the CNN unit comprises:
a convolutional unit including a plurality of convolution engines to perform convolution operation on the region of interest;
an activation unit that performs activation function when predefined features are detected; and
a pooling unit that performs down-sampling on the input image frame.

### Claim 8

A framebuffer-less method of convolutional neural network (CNN), comprising:
extracting features to generate a region of interest (ROI) in an input image frame;
performing convolutional neural network (CNN) on the region of interest of the input image frame to detect an object; and
comparing features extracted at different times and accordingly processing the input image frame selectively;
wherein the ROI is generated by adopting scan-line based technique and block-based scheme, the input image frame being divided into a plurality of blocks of image;
wherein the step of generating the ROI comprises:
extracting the features from the input image frame; and
making decision by classification whether to perform CNN for each block of image, thus generating a decision map, according to which the region of interest is determined.

### Claim 9 (depends on 8)

The method of claim 8, wherein the step of generating the ROI comprises:
generating block-based features, according to which decision of whether to perform CNN is made for each block of image.

### Claim 10 (depends on 8)

The method of claim 8, wherein the step of extracting the features comprises:
generating shallow features of the blocks of image according to block-based histogram or frequency analysis.

### Claim 11 (depends on 8)

The method of claim 8, further comprising a step of temporarily storing the features that generates the ROI.

### Claim 12 (depends on 11)

The method of claim 11, wherein the step of temporarily storing the features comprises:
generating a first feature map storing features of a previous image frame; and
generating a second feature map storing features of a current image frame.

### Claim 13 (depends on 12)

The method of claim 12, wherein the step of comparing the features comprises:
comparing the first feature and the second feature map, and accordingly updating the decision map.

### Claim 14 (depends on 11)

The method of claim 11, wherein the step of temporarily storing the features comprises:
generating a sliding window storing a block of the input image frame.

### Claim 15 (depends on 8)

The method of claim 8, wherein the step of performing convolutional neural network (CNN) comprises:
using a plurality of convolution engines to perform convolution operation on the region of interest;
performing activation function when predefined features are detected; and
performing down-sampling on the input image frame.