# US 2019/0244017 A1 · Gesture recognition method and system using siamese neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244017 A1
- Document ID: US20190244017A1
- Status: Published application
- Kind: A1
- Application number: 16/121,008
- Filed: 2018-09-04
- Published: 2019-08-08
- Earliest claimed: 2018-02-04
- Assignees: KaiKuTek Inc.
- Applicants: KaiKuTek Inc.
- Inventors: Tsung-Ming TAI; Yun-Jie JHANG; Wen-Jyi HWANG; Chun-Hsuan KUO
- Web page: https://patentgrep.com/patent/US20190244017A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G01S 7/414: {Discriminating targets with respect to background clutter}
- G01S 7/417: {involving the use of neural networks}
- G01S 13/584: {adapted for simultaneous range and velocity measurements}
- G01S 13/89: for mapping or imaging
- G06F 9/5027: {the resource being a machine, e.g. CPUs, Servers, Terminals}
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/24: Classification techniques
- G06F 18/24133: {Distances to prototypes}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- H03B 21/02: by plural beating, i.e. for frequency synthesis {; Beating in combination with multiplication or division of frequency (digital frequency synthesis using a ROM G06F1/02; digital frequency synthesis in general H03K; indirect frequency synthesis using a PLL H03L7/16)}
- G01S 7/356: {involving particularities of FFT processing}
- G01S 7/415: {Identification of targets based on measurements of movement associated with the target}
- G01S 13/50: Systems of measurement based on relative movement of target
- G06F 2218/12: Classification; Matching
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H03B: GENERATION OF OSCILLATIONS, DIRECTLY OR BY FREQUENCY-CHANGING, BY CIRCUITS EMPLOYING ACTIVE ELEMENTS WHICH OPERATE IN A NON-SWITCHING MANNER; GENERATION OF NOISE BY SUCH CIRCUITS (generators adapted for electrophonic musical instruments G10H; masers or lasers H01S; generation of oscillations in plasma H05H)

## Abstract

A gesture recognition system using siamese neural network executes a gesture recognition method. The gesture recognition method includes steps of: receiving a first training signal to calculate a first feature; receiving a second training signal to calculate a second feature; determining a distance between the first feature and the second feature in a feature space; adjusting the distance between the first feature and the second feature in feature space according to a predetermined parameter. Two neural networks are used to generate the first feature and the second feature, and determine the distance between the first feature and the second feature in the feature space for training the neural networks. Therefore, the gesture recognition system does not need a big amount of data to train one neural network for classifying a sensing signal. A user may easily define a new personalized gesture.

## Claims

### Claim 1

A gesture recognition method using siamese neural network, comprising steps of:
controlling weight of a first neural network unit and weight of a second neural network unit to be the same by a weight sharing unit;
receiving a first training signal from a sensor to calculate a first feature by the first neural network unit;
receiving a second training signal from the sensor to calculate a second feature by the second neural network unit;
determining a distance between the first feature and the second feature in a feature space by a similarity analysis unit;
controlling the weight of the first neural network unit and the weight of the second neural network unit through the weight sharing unit to adjust the distance between the first feature and the second feature in the feature space according to a predetermined parameter by a weight controlling unit.

### Claim 2 (depends on 1)

The gesture recognition method as claimed in claim 1, wherein the weight controlling unit reduces the distance between the first feature and the second feature in the feature space when the predetermined parameter is 1.

### Claim 3 (depends on 1)

The gesture recognition method as claimed in claim 1, wherein the weight controlling unit increases the distance between the first feature and the second feature in the feature space when the predetermined parameter is 0.

### Claim 4 (depends on 1)

The gesture recognition method as claimed in claim 1, further comprising step of:
receiving a sensing signal to calculate a sensing feature by the first neural network unit;
receiving an anchor signal to calculate a reference feature by the second neural network unit;
generating a distance between the sensing feature and the reference feature in the feature space by the similarity analysis unit;
when the distance between the sensing feature and the reference feature is smaller than a threshold value, the similarity analysis classifies a gesture event.

### Claim 5 (depends on 1)

The gesture recognition method as claimed in claim 1, wherein the first neural network unit and the second neural network unit execute convolutional neural networks (CNNs) or recurrent neural networks (RNNs).

### Claim 6 (depends on 1)

The gesture recognition method as claimed in claim 1, wherein the first training signal and the second training signal are Range Doppler Image (RDI) signals.

### Claim 7 (depends on 1)

The gesture recognition method as claimed in claim 1, wherein the distance generated by the similarity analysis unit is calculated by a contrastive loss function.

### Claim 8

A gesture recognition system using siamese neural network, comprising:
a sensor, configured to sense a first training signal and a second training signal;
a first neural network unit, configured to be electrically connected to the sensor to receive the first training signal, and calculate a first feature;
a second neural network unit, configured to be electrically connected to the sensor to receive the second training signal, and calculate a second feature;
a weight sharing unit, configured to be electrically connected to the first neural network unit and the second neural network unit, and control weight of the first neural network unit and weight of the second neural network unit to be the same;
a similarity analysis unit, configured to be electrically connected to the first neural network unit and the second neural network unit, and receive the first feature and the second feature; wherein the similarity analysis unit determines a distance between the first feature and the second feature in a feature space;
a weight controlling unit, configured to be electrically connected to the similarity analysis unit and the weight sharing unit; wherein the weight controlling unit controls the weight of the first neural network unit and the weight of the second neural network unit through the weight sharing unit to adjust the distance between the first feature and the second feature in the feature space according to a predetermined parameter.

### Claim 9 (depends on 8)

The gesture recognition system as claimed in claim 8, wherein the weight controlling unit reduces the distance between the first feature and the second feature in the feature space when the predetermined parameter is 1.

### Claim 10 (depends on 8)

The gesture recognition system as claimed in claim 8, wherein the weight controlling unit increases the distance between the first feature and the second feature in the feature space when the predetermined parameter is 0.

### Claim 11 (depends on 8)

The gesture recognition system as claimed in claim 8, wherein:
the first neural network unit further receives a sensing signal to calculate a sensing feature;
the second neural network unit further receives an anchor signal to calculate a reference feature;
the similarity analysis unit further generates a distance between the sensing feature and the reference feature in the feature space;
when the distance between the sensing feature and the reference feature is smaller than a threshold value, the similarity analysis unit classifies a gesture event.

### Claim 12 (depends on 8)

The gesture recognition system as claimed in claim 8, wherein the first neural network unit and the second neural network unit execute convolutional neural networks (CNNs) or recurrent neural networks (RNNs).

### Claim 13 (depends on 8)

The gesture recognition system as claimed in claim 8, wherein the first training signal and the second training signal are Range Doppler Image (RDI) signals.

### Claim 14 (depends on 8)

The gesture recognition system as claimed in claim 8, wherein the distance determined by the similarity analysis unit is calculated by a contrastive loss function.
