# US 2023/0068523 A1 · Radar-Based Gesture Classification Using a Variational Auto-Encoder Neural Network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0068523 A1
- Document ID: US20230068523A1
- Status: Published application
- Kind: A1
- Application number: 17/886,264
- Filed: 2022-08-11
- Published: 2023-03-02
- Earliest claimed: 2021-08-12
- Assignees: Infineon Technologies AG
- Applicants: Infineon Technologies AG
- Inventors: Avik Santra; Souvik Hazra; Thomas Reinhold Stadelmayer
- Web page: https://patentgrep.com/patent/US20230068523A1

## CPC

- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G01S 7/415: {Identification of targets based on measurements of movement associated with the target}
- G01S 7/417: {involving the use of neural networks}
- G01S 13/584: {adapted for simultaneous range and velocity measurements}
- G06F 18/24133: {Distances to prototypes}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/763: {Non-hierarchical techniques, e.g. based on statistics of modelling distributions}
- G06V 10/82: using neural networks
- G01S 13/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S13/46)
- G01S 13/589: {measuring the velocity vector}
- G06V 2201/12: Acquisition of 3D measurements of objects
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

In an embodiment, a method includes: obtaining one or more positional time spectrograms of a radar measurement of a scene comprising an object; and based on the one or more positional time spectrograms and based on a feature embedding of a variational auto-encoder neural network, predicting a gesture class of a gesture performed by the object.

## Claims

### Claim 1

A method comprising:
obtaining one or more positional time spectrograms of a radar measurement of a scene comprising an object; and
based on the one or more positional time spectrograms and based on a feature embedding of a variational auto-encoder neural network, predicting a gesture class of a gesture performed by the object.

### Claim 2 (depends on 1)

The method of claim 1, wherein the gesture class is predicted based on a comparison of a mean of a distribution of the feature embedding of the variational auto-encoder neural network with one or more regions predefined in a feature space of the feature embedding.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
monitoring a cluster formation of the means of the distributions of the feature embedding of the variational auto-encoder neural network obtained for multiple sets of the one or more positional time spectrograms, the cluster formation being outside of the one or more predefined regions; and
based on the monitoring of the cluster formation, determining a further predefined region in the feature space to enclose a respective cluster.

### Claim 4 (depends on 1)

The method of claim 1, wherein the one or more positional time spectrograms are obtained by time gating measurement data of the radar measurement based on at least one trigger event.

### Claim 5 (depends on 4)

The method of claim 4, wherein the at least one trigger event comprises a comparison between a change rate of a positional observable captured by the measurement data and at least one predefined threshold.

### Claim 6 (depends on 4)

The method of claim 4, wherein the at least one trigger event comprises an output of a gesture detection algorithm.

### Claim 7 (depends on 1)

The method of claim 1, wherein the one or more positional time spectrograms are selected from the group consisting of: a range time spectrogram, a velocity time spectrogram, an azimuthal angle time spectrogram, and an elevation angle time spectrogram.

### Claim 8 (depends on 1)

The method of claim 1, wherein the one or more positional time spectrograms comprise one or more raw positional time spectrograms, and wherein the variational auto-encoder neural network has been trained to reconstruct one or more filtered positional time spectrograms.

### Claim 9

A method for training a variational auto-encoder neural network for predicting a gesture class of a gesture performed by an object of a scene, the gesture class being selected from a plurality of gesture classes, the method comprising:
obtaining multiple training sets of one or more training positional time spectrograms (of a radar measurement of the scene comprising the object, each one of the multiple training sets being associated with a respective ground-truth label indicative of the respective gesture class; and
training the variational auto-encoder neural network based on the multiple training sets and the associated ground-truth labels.

### Claim 10 (depends on 9)

The method of claim 9, wherein the training of the variational auto-encoder neural network uses at least one loss that is determined based on at least one statistical distance between a distribution of a feature embedding of the variational auto-encoder neural network obtained for a first training set of the multiple training sets that is associated with a first gesture class of the plurality of gesture classes, and at least one mean of the at least one distribution of the feature embedding of the variational auto-encoder neural network obtained for at least one second training set of the multiple training sets that is associated with at least one of the first gesture class or a second gesture class of the plurality of gesture classes.

### Claim 11 (depends on 10)

The method of claim 10, wherein the at least one loss comprises a statistical distance triplet loss determined based on a first statistical distance and a second statistical distance, the first statistical distance being between the distribution of the feature embedding of the variational auto-encoder neural network obtained for an anchor training set of the multiple training sets and the mean of the distribution of the feature embedding of the variational auto-encoder neural network obtained for a positive training set of the multiple training sets, the second statistical distance being between the distribution of the feature embedding of the variational auto-encoder neural network obtained for the anchor training set and the mean of the distribution of the feature embedding of the variational auto-encoder neural network obtained for a negative training set of the multiple training sets.

### Claim 12 (depends on 10)

The method of claim 10, wherein the at least one loss comprises a statistical distance center loss determined based on a statistical distance between a class distribution associated with the first gesture class and means of the distributions of the feature embedding of the variational auto-encoder neural network obtained for all training sets of the multiple training sets associated with the first gesture class.

### Claim 13 (depends on 10)

The method claim 10, wherein the statistical distance is a Mahalanobis distance.

### Claim 14 (depends on 9)

The method of claim 9, wherein the one or more training positional time spectrograms comprise one or more raw training positional time spectrograms, the method further comprising:
applying an unscented Kalman filter to the one or more raw training positional time spectrograms to obtain one or more filtered training positional time spectrograms, wherein the training of the variational auto-encoder neural network uses at least one reconstruction loss which is based on a difference between a reconstruction of the one or more raw training positional time spectrograms output by the variational auto-encoder neural network and the one or more filtered training positional time spectrograms.

### Claim 15 (depends on 9)

The method of claim 9, further comprising, based on class distributions of a feature embedding of the variational auto-encoder neural network obtained for the training sets associated with each one of the plurality of gesture classes, determining predefined regions in a feature space of the feature embedding for gesture class prediction.

### Claim 16

A radar system comprising:
a millimeter-wave radar sensor configured to transmit radar signals towards a scene, receive reflected radar signals from the scene, and generate radar measurement data based on the reflected radar signals; and
a processor configured to:
generate one or more positional time spectrograms based on the radar measurement data, and
based on the one or more positional time spectrograms and based on a feature embedding of a variational auto-encoder neural network, predict a gesture class of a gesture performed by an object in the scene.

### Claim 17 (depends on 16)

The radar system of claim 16, wherein the gesture class is predicted based on a comparison of a mean of a distribution of the feature embedding of the variational auto-encoder neural network with one or more regions predefined in a feature space of the feature embedding.

### Claim 18 (depends on 17)

The radar system of claim 17, wherein the processor is further configured to:
monitor a cluster formation of the means of the distributions of the feature embedding of the variational auto-encoder neural network obtained for multiple sets of the one or more positional time spectrograms, the cluster formation being outside of the one or more predefined regions; and
based on the monitoring of the cluster formation, determine a further predefined region in the feature space to enclose a respective cluster.

### Claim 19 (depends on 16)

The radar system of claim 16, wherein the one or more positional time spectrograms are obtained by time gating measurement data of the radar measurement based on at least one trigger event, and wherein the at least one trigger event comprises a comparison between a change rate of a positional observable captured by the radar measurement data and at least one predefined threshold.

### Claim 20 (depends on 16)

The radar system of claim 16, wherein the one or more positional time spectrograms are selected from the group comprising: a range time spectrogram, a velocity time spectrogram, an azimuthal angle time spectrogram, and an elevation angle time spectrogram.
