# US 12,657,261 B2 · Method and device for classifying sensor data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,657,261 B2
- Document ID: US12657261B2
- Status: Granted
- Kind: B2
- Application number: 18/032,133
- Filed: 2021-09-09
- Published: 2025-09-04
- Granted: 2026-06-16
- Earliest claimed: 2020-10-14
- Assignees: Microvision, Inc.
- Applicants: MicroVision, Inc.
- Inventors: Julia Nitsch
- Web page: https://patentgrep.com/patent/US12657261B2

## CPC

- G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/2414: {Smoothing the distance, e.g. radial basis function networks [RBFN]}
- G06F 18/2431: Multiple classes
- G06F 18/2433: Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method and a device for classifying sensor data are proposed, wherein the method comprises providing a respective averaged feature vector for a multiplicity of classes wherein the method further comprises the following steps by means of a classifier, wherein the classifier comprises at least one neural network trained on the basis of training data: determining a feature vector on the basis of sensor data, respective determining of a cosine similarity between the feature vector and a respective averaged feature vector for the multiplicity of classes, comparing the respective cosine similarity with a threshold value established for each class beforehand, detecting a scenario not represented by the multiplicity of classes if the threshold values are not reached for all classes, and wherein the device comprises a classifier and is designed for carrying out the method.

## Claims

### Claim 1

A light detection and ranging (LiDAR) device, comprising:
a classifier unit including a processor and a memory that stores executable instructions that, when executed by the classifier unit, facilitate performance of operations, the operations comprising:
generating point cloud data representing an environment;
detecting a cluster of reflections in the point cloud data;
storing, in the memory, an averaged feature vector and a threshold value for each class of a multiplicity of predefined classes, wherein for each class, the averaged feature vector is determined by averaging feature vectors of first training samples of the class, and wherein the threshold value is determined by:
determining a cosine similarity between the feature vector of each first training sample of the class and the averaged feature vector of the class,
determining an averaged cosine similarity or a smallest cosine similarity of the first training samples of the class, and
establishing the averaged cosine similarity or the smallest cosine similarity as the threshold value of the class;
determining a feature vector from the cluster of reflections in the point cloud data;
determining a similarity measure between the feature vector from the cluster of reflections and the averaged feature vector for each of the multiplicity of predefined classes;
comparing the similarity measure with the threshold value for each of the multiplicity of predefined classes; and
determining that the feature vector is not represented by any of the multiplicity of predefined classes when the similarity measure is not greater than any of the threshold values.

### Claim 2 (depends on 1)

The LIDAR device according to claim 1, the operations further comprising determining a Mahalanobis distance between a covariance matrix and the averaged feature vector and the feature vector for each class of the multiplicity of predefined classes.

### Claim 3 (depends on 2)

The LiDAR device according to claim 2, wherein a reference value for each class of the multiplicity of predefined classes is determined based on the Mahalanobis distance and a χ2-distribution,
wherein the reference value is compared with a second threshold value established for each class of the multiplicity of predefined classes,
the method further comprising determining that the feature vector is not represented by any of the multiplicity of predefined classes when the reference value is not greater than any of the second threshold values.

### Claim 4 (depends on 1)

The LiDAR device according to claim 1, wherein the classifier includes a neural network.

### Claim 5 (depends on 4)

The LIDAR device according to claim 4, wherein the operations further comprise storing of the cluster of reflections in the point cloud data as a supplement to the first training samples.

### Claim 6 (depends on 4)

The LIDAR device according to claim 4
wherein the classifier is trained using second training samples,
wherein the second training samples are produced by a generative adversarial network (GAN), and
wherein the GAN comprises a generator and a discriminator.

### Claim 7 (depends on 6)

The LiDAR device according to claim 6, wherein training of the classifier is carried out based on a cross entropy loss based on the first training samples added with a Kullback-Leibler divergence based on the second training samples.

### Claim 8 (depends on 6)

The LIDAR device according to claim 6, wherein the operations further comprise a training of the GAN, namely of the generator and of the discriminator,
wherein the training of the classifier and the training of the GAN take place jointly.

### Claim 9 (depends on 8)

The LIDAR device according to claim 8, wherein the training of the generator is carried out based on a binary cross entropy loss based on the second training samples added with a Kullback-Leibler divergence based on the second training samples.

### Claim 10 (depends on 6)

The LIDAR device according to claim 6,
wherein the averaged feature vector of a class is determined by the
following steps:
determining a feature vector for each first training sample of the class,
adding the feature vectors of all first training samples of the class for obtaining an added feature vector
dividing the added feature vector by the number of the first training samples of the class.

### Claim 11

A light detection and ranging (LiDAR) device, comprising:
at least one solid state LiDAR sensor configured to generate a point cloud data representing an environment;
a pre-processing unit to detect a cluster of reflections in the point cloud data;
a memory storing an averaged feature vector and a threshold value for each class of a multiplicity of predefined classes, wherein, for each class, the averaged feature vector is determined by averaging feature vectors of first training samples of the class, and wherein the threshold value is determined by:
determining a cosine similarity between the feature vector of each first training sample of the class and the averaged feature vector of the class,
determining an averaged cosine similarity or a smallest cosine similarity of the first training samples of the class, and
establishing the averaged cosine similarity or the smallest cosine similarity as the threshold value of the class;
a classifier unit to determine a feature vector from the cluster of reflections in the point cloud data, to determine a similarity measure between the feature vector and the averaged feature vector for each of the multiplicity of predefined classes, to compare the similarity measure with the threshold value for each of the multiplicity of predefined classes, and to determine that the feature vector is not represented by any of the multiplicity of predefined classes when the similarity measure is not greater than any of the threshold values.

### Claim 12

A non-transitory computer-readable storage medium on which a program is stored which enables a computer, after it has been loaded into a memory of the computer, to carry out a method of operating a light detection and ranging (LiDAR) device, the method comprising:
generating point cloud data representing an environment;
detecting a cluster of reflections in the point cloud data;
storing, in a memory, an averaged feature vector and a threshold value for each of a multiplicity of predefined classes, wherein, for each class, the averaged feature vector is determined by averaging feature vectors of first training samples of the class, and wherein the threshold value is determined by:
determining a cosine similarity between the feature vector of each first training sample of the class and the averaged feature vector of the class,
determining an averaged cosine similarity or a smallest cosine similarity of the first training samples of the class, and
establishing the averaged cosine similarity or the smallest cosine similarity as the threshold value of the class;
determining a feature vector from the cluster of reflections in the point cloud data;
determining a similarity measure between the feature vector and the averaged feature vector for each of the multiplicity of predefined classes;
comparing the similarity measure with the threshold value for each of the multiplicity of predefined classes; and
determining that the feature vector is not represented by any of the multiplicity of predefined classes when the similarity measure is not greater than any of the threshold values.

### Claim 13 (depends on 11)

The LiDAR device according to claim 11, wherein the similarity measure comprises a Mahalanobis distance between a covariance matrix and the averaged feature vector and the feature vector for each class.

### Claim 14 (depends on 11)

The LiDAR device according to claim 11, wherein the classifier unit includes a neural network, and the cluster of reflections in the point cloud data are stored as training data to train the neural network.

### Claim 15 (depends on 12)

The non-transitory computer-readable storage medium according to claim 12, wherein the similarity measure comprises a Mahalanobis distance between a covariance matrix and the averaged feature vector and the feature vector for each class.

### Claim 16 (depends on 12)

The non-transitory computer-readable storage medium according to claim 12, wherein the determining the feature vector, the determining the similarity measure, the comparing, and the determining that the feature vector is not represented by any of the multiplicity of predefined classes are performed by a classifier that includes a neural network, and the cluster of reflections in the point cloud data are stored as training data to train the neural network.