# US 2023/0410328 A1 · Target tracking method and system of spiking neural network based on event camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410328 A1
- Document ID: US20230410328A1
- Status: Published application
- Kind: A1
- Application number: 18/240,526
- Filed: 2023-08-31
- Published: 2023-12-21
- Earliest claimed: 2022-04-07
- Assignees: Zhejiang Lab; Zhejiang University
- Applicants: Zhejiang Lab; Zhejiang University
- Inventors: Wenyi ZHAO; Huajin TANG; Chaofei HONG; Xiao WANG; Mengwen YUAN; Yujing LU; Mengxiao ZHANG; Gang PAN
- Web page: https://patentgrep.com/patent/US20230410328A1

## CPC

- H04N 23/617: Upgrading or updating of programs or applications for camera control
- G06F 18/213: Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06N 3/045: Combinations of networks
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/08: Learning methods
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 3/4046: using neural networks
- G06T 7/248: {involving reference images or patches}
- G06T 7/74: {involving reference images or patches}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/77: Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
- H04N 25/47: Image sensors with pixel address output; Event-driven image sensors; Selection of pixels to be read out based on image data
- H04N 25/707: Pixels for event detection
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V 2201/07: Target detection
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A target tracking method and a target tracking system of a spiking neural network based on an event camera are provided. The method includes: acquiring a data stream of asynchronous events in a high dynamic scene of a target by an event camera as input data; dividing the data stream of the asynchronous events into synchronous event frames with millisecond time resolution; training a twin network based on a spiking neural network by a gradient substitution algorithm with a target image as a template image and a complete image as a searched image; and tracking the target by a trained twin network with interpolating a result of feature mapping to up-sample and obtaining the position of the target in an original image. The twin network includes a feature extractor and a cross-correlation calculator.

## Claims

### Claim 1

A target tracking method of a spiking neural network based on an event camera, the method comprising:
acquiring a data stream of asynchronous events in a high dynamic scene of a target by an event camera as input data;
dividing the data stream of the asynchronous events into synchronous event frames with millisecond time resolution by asynchronous event accumulation, wherein the synchronous event frames are binary images similar to a spiking;
training a twin network based on a spiking neural network by a gradient substitution algorithm with a target image as a template image denoted as z and a complete image as a searched image denoted as x, wherein the twin network comprises a feature extractor with weight sharing and a cross-correlation calculator for calculating a position of the target, the target image is an image of the target in the synchronous event frames, and the complete image comprises any one of all the synchronous event frames; and
tracking the target by a trained twin network with interpolating a result of feature mapping to up-sample and obtaining the position of the target in an original image.

### Claim 2 (depends on 1)

The target tracking method of the spiking neural network based on the event camera of claim 1, wherein the synchronous event frames are generated by dividing the asynchronous events according to a set size and number of time steps, accumulating the data stream of the asynchronous events within each time step, setting a pixel of a coordinate to 1 as long as the number of the asynchronous events generated at the coordinate within the same time step is greater than 0, otherwise setting the pixel of the coordinate to 0, and ultimately generating event frame images divided by the time steps.

### Claim 3 (depends on 1)

The target tracking method of the spiking neural network based on the event camera of claim 1, wherein the feature extractor is generated by adopting a spiking convolutional neural network as the feature extractor, wherein a network structure of the spiking convolutional neural network is 96C5-2S-256C3-2S-384C3-384C3-256C3, wherein 96C5 represents a spiking convolutional layer with a convolutional kernel size of 5 and an output channel of 96, 2S represents a pooling layer with a down-sampling of 2 times, and the rest network structure is in a similar manner; a convolutional step of a first convolutional layer is 2, convolutional steps of the rest convolutional layers are 1, and all of convolutional layers of the feature extractor are followed by a spiking neuron.

### Claim 4 (depends on 3)

The target tracking method of the spiking neural network based on the event camera of claim 3, wherein the spiking neuron is a Leaky integrate and fire neuron model, i.e.,
τ
m
⁢
d
⁢
V
d
⁢
t
=
V
r
⁢
e
⁢
s
⁢
t
-
V
+
R
m
⁢
I
,
wherein τm represents a membrane time constant, V represents a membrane potential, t represents a spiking time, Vrest represents a resting potential, and Rm and I represent impedance and input current of a cell membrane, respectively;
the feature extractor is denoted as φ, a size of the template image z is 255*255*3, a size of the searched image x is 127*127*3, an output after an operation of the feature extractor is φ(z) with a size of 6*6*256 and φ(x) with a size of 22*22*256.

### Claim 5 (depends on 1)

The target tracking method of the spiking neural network based on the event camera of claim 1, wherein an operation of the cross-correlation calculator comprises:
configuring a feature mapping denoted as φ(z) after extracting features from the template image z to be a convolutional kernel, configuring a feature mapping φ(x) after extracting features from the searched image x to be a feature map to be convolved, and performing a convolution operation on the convolutional kernel and the feature map to be convolved, wherein a result produced after the convolution operation of the current convolutional layer is a similarity heatmap that represents a prediction probability of a predicted center position of the target, and a position of a maximum spiking issuance rate is the predicted center position of the target.

### Claim 6 (depends on 1)

The target tracking method of the spiking neural network based on the event camera of claim 1, wherein the twin network is generated by:
adopting a brain-inspired computing development framework, and putting a padded template image and the searched image into the same batch sequentially based on batch training, so that the number of neurons in an input layer for the padded template image is the same as that for the searched image, and the padded template image and the searched image share the same network connection;
after operation of the feature extractor denoted as φ, cropping an output of an odd-numbered sample that is an output of a z-branch denoted as φ(z) to delete edge-padding of φ(z), and obtaining the feature mapping with a due size of 6*6*256.

### Claim 7 (depends on 1)

The target tracking method of the spiking neural network based on the event camera of claim 1, further comprising:
performing no update on the target image that is the template image, performing an operation φ(z) of the feature extractor for an initial target once, configuring the searched image to be an image equivalent to 4 times a size of the template image, wherein the searched image is centered on the position of the target and cropped from a previous synchronous event frame, and a search area is narrowed to improve real-time performance;
adopting bicubic interpolation to up-sample and revert a size of the similarity heatmap, determining a predicted position of the target, adopting three scales to search, that is, scaling the similarity heatmap to 1.03{−1,0,1} respectively; and selecting a position of a maximum spiking issuance rate from a scaling output as a final result, wherein the maximum spiking issuance rate is a maximum similarity.

### Claim 8

A target tracking system of a spiking neural network based on an event camera, comprising a data stream acquiring module, a high time resolution event frame generating module, a network training module, and a network output module;
wherein the data stream acquiring module is configured for acquiring a data stream of asynchronous events in a high dynamic scene of a target by an event camera as input data;
the high time resolution event frame generating module is configured for dividing the data stream of the asynchronous events into synchronous event frames with millisecond time resolution by asynchronous event accumulation, wherein the synchronous event frames are binary images similar to a spiking;
the network training module is configured for training a twin network based on a spiking neural network by a gradient substitution algorithm with a target image as a template image denoted as z and a complete image as a searched image denoted as x, wherein the twin network comprises a feature extractor with weight sharing and a cross-correlation calculator for calculating a position of the target, the target image is an image of the target in the synchronous event frames, and the complete image comprises any one of all the synchronous event frames; and
the network output module is configured for tracking the target by a trained twin network with interpolating a result of feature mapping to up-sample and obtaining the position of the target in an original image.

### Claim 9 (depends on 1)

An electronic device, comprising a processor and a memory, wherein the memory stores a computer program executable by the processor to implement the steps of the target tracking method of the spiking neural network based on the event camera of claim 1.

### Claim 10 (depends on 1)

A computer-readable storage medium having stored a computer program, wherein the computer program is executed by a processor to implement the steps of the target tracking method of the spiking neural network based on the event camera of claim 1.

### Claim 11 (depends on 9)

The electronic device of claim 9, wherein the synchronous event frames are generated by dividing the asynchronous events according to a set size and number of time steps, accumulating the data stream of the asynchronous events within each time step, setting a pixel of a coordinate to 1 as long as the number of the asynchronous events generated at the coordinate within the same time step is greater than 0, otherwise setting the pixel of the coordinate to 0, and ultimately generating event frame images divided by the time steps.

### Claim 12 (depends on 9)

The electronic device of claim 9, wherein the feature extractor is generated by adopting a spiking convolutional neural network as the feature extractor, wherein a network structure of the spiking convolutional neural network is 96C5-2S-256C3-2S-384C3-384C3-256C3, wherein 96C5 represents a spiking convolutional layer with a convolutional kernel size of 5 and an output channel of 96, 2S represents a pooling layer with a down-sampling of 2 times, and the rest network structure is in a similar manner; a convolutional step of a first convolutional layer is 2, convolutional steps of the rest convolutional layers are 1, and all of convolutional layers of the feature extractor are followed by a spiking neuron.

### Claim 13 (depends on 12)

The electronic device of claim 12, wherein the spiking neuron is a Leaky integrate and fire neuron model, i.e.,
τ
m
⁢
d
⁢
V
d
⁢
t
=
V
r
⁢
e
⁢
s
⁢
t
-
V
+
R
m
⁢
I
wherein τm represents a membrane time constant, V represents a membrane potential, t represents a spiking time, Vrest represents a resting potential, and Rm and I represent impedance and input current of a cell membrane, respectively;
the feature extractor is denoted as φ, a size of the template image z is 255*255*3, a size of the searched image x is 127*127*3, an output after an operation of the feature extractor is φ(z) with a size of 6*6*256 and φ(x) with a size of 22*22*256.

### Claim 14 (depends on 9)

The electronic device of claim 9, wherein an operation of the cross-correlation calculator comprises:
configuring a feature mapping denoted as φ(z) after extracting features from the template image z to be a convolutional kernel, configuring a feature mapping φ(x) after extracting features from the searched image x to be a feature map to be convolved, and performing a convolution operation on the convolutional kernel and the feature map to be convolved, wherein a result produced after the convolution operation of the current convolutional layer is a similarity heatmap that represents a prediction probability of a predicted center position of the target, and a position of a maximum spiking issuance rate is the predicted center position of the target.

### Claim 15 (depends on 9)

The electronic device of claim 9, wherein the twin network is generated by:
adopting a brain-inspired computing development framework, and putting a padded template image and the searched image into the same batch sequentially based on batch training, so that the number of neurons in an input layer for the padded template image is the same as that for the searched image, and the padded template image and the searched image share the same network connection;
after operation of the feature extractor denoted as φ, cropping an output of an odd-numbered sample that is an output of a z-branch denoted as φ(z) to delete edge-padding of φ(z), and obtaining the feature mapping with a due size of 6*6*256.

### Claim 16 (depends on 9)

The electronic device of claim 9, further comprising:
performing no update on the target image that is the template image, performing an operation φ(z) of the feature extractor for an initial target once, configuring the searched image to be an image equivalent to 4 times a size of the template image, wherein the searched image is centered on the position of the target and cropped from a previous synchronous event frame, and a search area is narrowed to improve real-time performance;
adopting bicubic interpolation to up-sample and revert a size of the similarity heatmap, determining a predicted position of the target, adopting three scales to search, that is, scaling the similarity heatmap to 1.03{−1,0,1}, respectively; selecting a position of a maximum spiking issuance rate from a scaling output as a final result, wherein the maximum spiking issuance rate is a maximum similarity.

### Claim 17 (depends on 10)

The computer-readable storage medium of claim 10, wherein the synchronous event frames are generated by dividing the asynchronous events according to a set size and number of time steps, accumulating the data stream of the asynchronous events within each time step, setting a pixel of a coordinate to 1 as long as the number of the asynchronous events generated at the coordinate within the same time step is greater than 0, otherwise setting the pixel of the coordinate to 0, and ultimately generating event frame images divided by the time steps.

### Claim 18 (depends on 10)

The computer-readable storage medium of claim 10, wherein the feature extractor is generated by adopting a spiking convolutional neural network as the feature extractor, wherein a network structure of the spiking convolutional neural network is 96C5-2S-256C3-2S-384C3-384C3-256C3, wherein 96C5 represents a spiking convolutional layer with a convolutional kernel size of 5 and an output channel of 96, 2S represents a pooling layer with a down-sampling of 2 times, and the rest network structure is in a similar manner; a convolutional step of a first convolutional layer is 2, convolutional steps of the rest convolutional layers are 1, and all of convolutional layers of the feature extractor are followed by a spiking neuron.

### Claim 19 (depends on 18)

The computer-readable storage medium of claim 18, wherein the spiking neuron is a Leaky integrate and fire neuron model, i.e.,
τ
m
⁢
d
⁢
V
d
⁢
t
=
V
r
⁢
e
⁢
s
⁢
t
-
V
+
R
m
⁢
I
wherein τm represents a membrane time constant, V represents a membrane potential, t represents a spiking time, Vrest represents a resting potential, and Rm and I represent impedance and input current of a cell membrane, respectively;
the feature extractor is denoted as φ, a size of the template image z is 255*255*3, a size of the searched image x is 127*127*3, an output after an operation of the feature extractor is φ(z) with a size of 6*6*256 and φ(x) with a size of 22*22*256.

### Claim 20 (depends on 10)

The computer-readable storage medium of claim 10, wherein an operation of the cross-correlation calculator comprises:
configuring a feature mapping denoted as φ(z) after extracting features from the template image z to be a convolutional kernel, configuring a feature mapping φ(x) after extracting features from the searched image x to be a feature map to be convolved, and performing a convolution operation on the convolutional kernel and the feature map to be convolved, wherein a result produced after the convolution operation of the current convolutional layer is a similarity heatmap that represents a prediction probability of a predicted center position of the target, and a position of a maximum spiking issuance rate is the predicted center position of the target.
