# US 2022/0277173 A1 · Function testing for movable objects in safety critical applications with spatial representation learning and adversarial generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277173 A1
- Document ID: US20220277173A1
- Status: Published application
- Kind: A1
- Application number: 17/188,907
- Filed: 2021-03-01
- Published: 2022-09-01
- Earliest claimed: 2021-03-01
- Assignees: Robert Bosch GmbH
- Applicants: Robert Bosch GmbH
- Inventors: Wenbin He; Liang Gou; Lincan Zou; Liu Ren
- Web page: https://patentgrep.com/patent/US20220277173A1

## CPC

- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06V 10/82: using neural networks
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods and systems for performing function testing for moveable objects. One system includes an electronic processor configured to access a driving scene including a moveable object. The electronic processor is also configured to perform spatial representation learning on the driving scene. The electronic processor is also configured to generate an adversarial example based on the learned spatial representation. The electronic processor is also configured to retrain the deep learning model using the adversarial example and the driving scene.

## Claims

### Claim 1

A system for performing function testing for moveable objects, the system comprising:
an electronic processor configured to
access a driving scene including a moveable object,
perform spatial representation learning on the driving scene,
generate an adversarial example based on the learned spatial representation, and
retrain a deep learning model using the adversarial example and the driving scene.

### Claim 2 (depends on 1)

The system of claim 1, wherein the electronic processor is configured to perform the spatial representation learning using a conditional variational autoencoder.

### Claim 3 (depends on 1)

The system of claim 1, wherein the spatial representation learning includes
training an encoder configured to encode a bounding box of the moveable object included in the driving scene into a latent representation, and
training a decoder configured to reconstruct the bounding box given the latent representation and the driving scene.

### Claim 4 (depends on 1)

The system of claim 1, wherein the deep learning model is a semantic segmentation model.

### Claim 5 (depends on 1)

The system of claim 1, wherein the deep learning model identifies a bounding box representing the moveable object included in the driving scene.

### Claim 6 (depends on 1)

The system of claim 1, wherein the electronic processor is configured to generate the adversarial example by
generating and inserting a new moveable object into the driving scene, and
perturbing a spatial representation of the new moveable object within the driving scene.

### Claim 7 (depends on 6)

The system of claim 6, wherein the electronic processor is configured to generate and insert the new moveable object by
determining a spatial representation of the moveable object in the driving scene by sampling a latent representation for the driving scene,
selecting the new moveable object based on a similarity of the new moveable object with the moveable object of the driving scene, and
inserting the new moveable object into the driving scene as a new driving scene.

### Claim 8 (depends on 7)

The system of claim 7, wherein the electronic processor is further configured to
obtain, using a decoder, a corresponding bounding box for the moveable object,
wherein the new moveable object is inserted into the corresponding bounding box.

### Claim 9 (depends on 7)

The system of claim 7, wherein the electronic processor is configured to blend the new moveable object into the driving scene.

### Claim 10 (depends on 6)

The system of claim 6, wherein the electronic processor is configured to perturb the spatial representation of the new moveable object by
generating a set of new moveable objects,
determining a performance of the deep learning model by applying the deep learning model to the set of new moveable objects, and
determining the adversarial example based on the performance of the deep learning model on the set of new moveable objects.

### Claim 11 (depends on 10)

The system of claim 10, wherein the adversarial example includes a new moveable object from the set of new moveable objects.

### Claim 12

A method for performing function testing for moveable objects, the method comprising:
accessing a driving scene including a moveable object;
performing, with an electronic processor, spatial representation learning on the driving scene;
generating, with the electronic processor, an adversarial example based on the learned spatial representation learning; and
retraining, with the electronic processor, the deep learning model using the adversarial example and the driving scene.

### Claim 13 (depends on 12)

The method of claim 12, wherein performing the spatial representation learning includes
training an encoder configured to encode a bounding box of the moveable object included in the driving scene into a latent representation, and
training a decoder configured to reconstruct the bounding box given the latent representations and the driving scene.

### Claim 14 (depends on 12)

The method of claim 12, wherein generating the adversarial example includes
generating and inserting a new moveable object into the driving scene, and
perturbing a spatial representation of the new moveable object within the driving scene.

### Claim 15 (depends on 14)

The method of claim 14, wherein generating and inserting the new moveable object includes
determining a spatial representation of the moveable object included in the driving scene by sampling a latent representation for the driving scene,
selecting the new moveable object based on a similarity of the new moveable object with the moveable object of the driving scene, and
inserting the new moveable object into the driving scene as a new driving scene.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
obtain, using a decoder, a corresponding bounding box for the moveable object,
wherein the new moveable object is inserted into the corresponding bounding box.

### Claim 17 (depends on 14)

The method of claim 14, wherein perturbing the spatial representation of the new moveable object includes
generating a set of new moveable objects,
determining a performance of the deep learning model by applying the deep learning model to the set of new moveable objects, and
determining the adversarial example based on the performance of the deep learning model on the set of new moveable objects.

### Claim 18 (depends on 12)

The method of claim 12, further comprising:
generating a visual summary for display, the visual summary including at least one selected from a group consisting of a robustness quantification for the deep learning model and a summarization of adversarial attack patterns.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
determining the robustness quantification for the deep learning model, wherein the robustness quantification is a ratio of a performance drop of the adversarial example over an amount of change for the adversarial example in latent space.

### Claim 20

A non-transitory, computer-readable medium storing instructions that, when executed by an electronic processor, perform a set of functions, the set of functions comprising:
accessing a driving scene including a moveable object;
performing spatial representation learning on the driving scene;
generating an adversarial example based on the learned spatial representation; and
retraining the deep learning model using the adversarial example and the driving scene.
