# US 2020/0394418 A1 · Image processing method, an image processing apparatus, and a surveillance system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394418 A1
- Document ID: US20200394418A1
- Status: Published application
- Kind: A1
- Application number: 16/776,660
- Filed: 2020-01-30
- Published: 2020-12-17
- Earliest claimed: 2019-06-14
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: WEIHENG LIU; Yinlong QIAN; Hyunku LEE; Qiang WANG; Keunjoo Paul PARK; Hyunsurk Eric RYU
- Web page: https://patentgrep.com/patent/US20200394418A1

## CPC

- G08B 29/186: {Fuzzy logic; neural networks}
- G06F 18/2136: based on sparsity criteria, e.g. with an overcomplete basis
- G06F 18/24: Classification techniques
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 30/19173: {Classification techniques}
- G08B 13/19602: {Image analysis to detect motion of the intruder, e.g. by frame subtraction}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/61: based on recognised objects
- H04N 23/745: Detection of flicker frequency or suppression of flicker wherein the flicker is caused by illumination, e.g. due to fluorescent tube illumination or pulsed LED illumination
- G06V 10/513: {Sparse representations}
- G08B 13/19613: {Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion}
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An image processing method including: capturing changes in a monitored scene; and performing a sparse feature calculation on the changes in the monitored scene to obtain a sparse feature map.

## Claims

### Claim 1

An image processing method, comprising:
capturing changes in a monitored scene; and
performing a sparse feature calculation on the changes in the monitored scene to obtain a sparse feature map.

### Claim 2 (depends on 1)

The image processing method of claim 1, wherein performing the sparse feature calculation comprises:
generating a pixel-based event when the changes in the monitored scene are captured; and
projecting the generated pixel-based event along a time axis, and performing the sparse feature calculation based on a two-dimensional image formed by the projecting to obtain the sparse feature map.

### Claim 3

(canceled)

### Claim 4 (depends on 2)

The image processing method of claim 2, wherein generating the pixel-based event comprises:
capturing the changes in the monitored scene by a first camera to generate the pixel-based event,
or, generating the pixel-based event by performing frame difference processing on a frame image captured by a second camera.

### Claim at position 5

5-6. (canceled)

### Claim 7 (depends on 2)

The image processing method of claim 2, wherein a texture of an object generating the pixel-based event is reconstructed using the pixel-based event.

### Claim 8 (depends on 2)

The image processing method of claim 2, wherein the two-dimensional image is formed based on an event generated by object motion,
wherein the event generated by the object motion is obtained by filtering out events generated by light source flicker or by internal noise of a camera for capturing the changes in the monitored scene.

### Claim 9 (depends on 2)

The image processing method of claim 2, wherein performing the sparse feature calculation comprises:
obtaining the sparse feature map by performing the sparse feature calculation based on the two-dimensional image by a sparse convolution neural network.

### Claim 10 (depends on 9)

The image processing method of claim 9, wherein obtaining the sparse feature map by performing the sparse feature calculation based on the two-dimensional image by the sparse convolution neural network comprises:
determining a feature activity map corresponding to each convolution layer of the sparse convolution neural network based on the structure of the sparse convolution neural network and the two-dimensional image; and
obtaining the sparse feature map by the sparse convolution neural network using each feature activity map and the two-dimensional image.

### Claim at position 10

11-15. (canceled)

### Claim 16 (depends on 1)

The image processing method of claim 1, wherein the image processing method further comprises:
performing region feature extraction on the sparse feature map;
identifying an object that changes the monitored scene based on extracted region features; and
determining a location, motion trajectory, behavior, or identity of the object if the identified object is a person.

### Claim at position 12

17-19. (canceled)

### Claim 20

A surveillance system, comprising:
a front-end camera that captures changes in a monitored scene; and
a processor that is configured to perform a sparse feature calculation on the changes in the monitored scene to obtain a sparse feature map.

### Claim 21 (depends on 20)

The surveillance system of claim 20, wherein the processor projects a pixel-based event generated by capturing the changes in the monitored scene along a time axis and performs the sparse feature calculation based on a two-dimensional image formed by the projecting to obtain the sparse feature map.

### Claim at position 15

22-34. (canceled)

### Claim 35 (depends on 20)

The surveillance system of claim 20 wherein the processor is further configured to:
perform a region feature extraction on the sparse feature map;
identify an object that changes the monitored scene based on extracted region features; and
determine a location, motion trajectory, behavior, or identity of the object if the identified object is a person.

### Claim at position 17

36-38. (canceled)

### Claim 39

An image processing apparatus, comprising:
a dynamic event information collection module that captures changes in a monitored scene; and
a sparse feature map generation module that performs a sparse feature calculation on the changes in the monitored scene to obtain a sparse feature map.

### Claim 40 (depends on 39)

The image processing apparatus of claim 39, wherein the sparse feature map generation module comprises:
a dynamic event map generation sub-module that projects a pixel-based event generated by capturing the changes in the monitored scene along a time axis to form a two-dimensional image; and
a sparse feature map operation sub-module that performs the sparse feature calculation based on the two-dimensional image formed by the projecting to obtain the sparse feature map.

### Claim 41

(canceled)

### Claim 42 (depends on 40)

The image processing apparatus of claim 40, wherein,
the dynamic event information collection module generates the pixel-based event when capturing the changes in the monitored scene,
or, the sparse feature map generation module further includes a frame difference processing sub-module that generates the pixel-based event by performing a frame difference processing on the captured frame image of the monitored scene.

### Claim 43

(canceled)

### Claim 44 (depends on 40)

The image processing apparatus of claim 40, wherein the sparse feature map generation module further comprises a texture reconstruction sub-module that reconstructs a texture of an object generating the pixel-based event using the pixel-based event.

### Claim 45 (depends on 40)

The image processing apparatus of claim 40, wherein the sparse feature map generation module further comprises a dynamic event pre-processing sub-module that obtains an event generated by object motion by filtering out events generated by light source flicker or by internal noise of a camera for capturing the changes in the monitored scene,
wherein the dynamic event map generation sub-module forms the two-dimensional image based on the event generated by the object motion.

### Claim 46 (depends on 40)

The image processing apparatus of claim 40, wherein the sparse feature map operation sub-module performs a sparse feature calculation based on the two-dimensional image to obtain the sparse feature map by using a sparse convolution neural network.

### Claim 47 (depends on 46)

The image processing apparatus of claim 46, wherein the sparse feature map operation sub-module comprises:
a feature activity map generation unit that determines a feature activity map corresponding to each convolution layer of the sparse convolution neural network based on the structure of the sparse convolution neural network and the two-dimensional image; and
a sparse feature map operation unit that obtains the sparse feature map by the sparse convolution neural network using each feature activity map and the two-dimensional image.

### Claim at position 27

48-52. (canceled)

### Claim 53 (depends on 39)

The image processing apparatus of claim 39, wherein the image processing apparatus further comprises a feature map analysis module that performs region feature extraction on the obtained sparse feature map, identifies an object that changes the monitored scene based on extracted region features and determines a location, motion trajectory, behavior, or identity of the object if the identified object is a person.

### Claim at position 29

54-56. (canceled)

### Claim 57 (depends on 1)

A computer-readable storage medium storing a computer program, when the computer program is performed by a processor, to implement the image processing method of claim 1.
