# US 2024/0199071 A1 · Generating a driving assistant model using synthetic data generated using historical shadow driver failures and generative rendering with physical constraints

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0199071 A1
- Document ID: US20240199071A1
- Status: Published application
- Kind: A1
- Application number: 18/542,857
- Filed: 2023-12-18
- Published: 2024-06-20
- Earliest claimed: 2022-12-18
- Assignees: Cognata Ltd.
- Applicants: Cognata Ltd.
- Inventors: Dan ATSMON
- Web page: https://patentgrep.com/patent/US20240199071A1

## CPC

- B60W 60/001: {Planning or execution of driving tasks}
- B60W 50/06: Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot
- G06F 30/27: using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
- B60W 2050/0019: {Control system elements or transfer functions}
- B60W 2050/0075: {Automatic parameter input, automatic initialising or calibrating means}
- B60W 2556/00: Input parameters relating to data
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method for generating a driving assistant model, comprising: computing at least one semantic driving scenario by computing at least one permutation of at least one initial semantic driving scenario; providing the at least one semantic driving scenario to a simulation generator to produce simulated driving data describing at least one simulated driving environment; training a driving assistant model using the simulated driving data to produce a trained driving assistant model; and providing by the trained driving assistant model at least one driving instruction to at least one autonomous driving model while the at least one autonomous driving model is operating.

## Claims

### Claim 1

A method for generating a driving assistant model, comprising:
computing at least one semantic driving scenario by computing at least one permutation of at least one initial semantic driving scenario;
providing the at least one semantic driving scenario to a simulation generator to produce simulated driving data describing at least one simulated driving environment;
training a driving assistant model using the simulated driving data to produce a trained driving assistant model; and
providing by the trained driving assistant model at least one driving instruction to at least one autonomous driving model while the at least one autonomous driving model is operating.

### Claim 2 (depends on 1)

The method of claim 1, wherein at least one of the at least one permutation is computed by providing at least one of the at least one initial semantic driving scenario to a generative machine learning model trained to compute, in response to input comprising a semantic driving scenario, a permutation of the semantic driving scenario.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
in response to the at least one autonomous driving model receiving the at least one driving instruction providing input, by the at least one autonomous driving model, to at least one control circuit of a vehicle.

### Claim 4 (depends on 3)

The method of claim 3, wherein the trained driving assistant model is installed in the vehicle.

### Claim 5 (depends on 1)

The method of claim 1, wherein the trained driving assistant model provides the at least one driving instruction to the at least one autonomous driving model via at least one digital communication network.

### Claim 6 (depends on 1)

The method of claim 1, further comprising receiving, by the trained driving assistant from the at least one autonomous driving model, driving data collected while the autonomous driving model is operating;
wherein providing the at least one driving instruction to the at least one autonomous driving model is in response to receiving the driving data from the at least one autonomous driving model.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
accessing driving event data describing at least one driving event detected in other driving data collected during operation of at least one other autonomous driving model (shadow driver) in another vehicle driven by a human driver; and
computing the at least one initial semantic driving scenario using the driving event data.

### Claim 8 (depends on 7)

The method of claim 7, wherein the driving event data comprises one or more of:
at least one signal, captured while the other vehicle is driven by the human driver, by at least one sensor installed in the other vehicle;
captured driving data collected while the other vehicle is driven by the human driver and while the at least one signal is captured; and
computed driving data computed by the shadow driver, using the at least one signal, while the other vehicle is driven by the human driver.

### Claim 9 (depends on 8)

The method of claim 8, wherein the driving event data further comprises annotation data describing one or more relations between the at least one signal, the captured driving data and the computed driving data.

### Claim 10 (depends on 8)

The method of claim 8, wherein computing the at least one initial semantic driving scenario is further using at least one object identified in the at least one signal and not identified in the computed driving data; and
wherein the at least one initial semantic driving scenario comprises the at least one object.

### Claim 11 (depends on 10)

The method of claim 10, further comprising identifying the at least one object in the at least one signal.

### Claim 12 (depends on 9)

The method of claim 9, wherein the annotation data comprises an indication of at least one object identified in the at least one signal and not identified in the computed driving data.

### Claim 13 (depends on 10)

The method of claim 10, wherein computing the at least one permutation comprises changing at least one property of the at least one object.

### Claim 14 (depends on 1)

The method of claim 1, wherein the simulated driving data comprises a plurality of synthetic signals, each simulating one of a plurality of signals captured from at least one physical driving environment equivalent to the at least one simulated driving environment by a plurality of sensors mounted on yet another vehicle while traversing the at least one physical driving environment.

### Claim 15 (depends on 1)

The method of claim 1, wherein the simulated driving data comprises a ground truth of the at least one simulated driving environment.

### Claim 16 (depends on 8)

The method of claim 8, wherein the at least one sensor comprises at least one of: a camera, an electromagnetic radiation sensor, a microphone, a thermometer, an acceleration sensor, a rolling shutter camera, a velocity sensor, an audio sensor, a radio detection and ranging sensor (radar), a laser imaging, detection, a ranging sensor (LIDAR), an ultrasonic sensor, a thermal sensor, and a far infra-red (FIR) sensor and a video camera.

### Claim 17 (depends on 1)

The method of claim 1, further comprising validating the driving assistant model using the simulated driving data to produce the trained driving assistant model, additionally or alternatively to training the driving assistant model using the simulated driving data.

### Claim 18 (depends on 1)

The method of claim 1, further comprising verifying the driving assistant model using the simulated driving data to produce the trained driving assistant model, additionally or alternatively to training the driving assistant model using the simulated driving data.

### Claim 19 (depends on 1)

The method of claim 1, further comprising testing the driving assistant model using the simulated driving data to produce the trained driving assistant model, additionally or alternatively to training the driving assistant model using the simulated driving data.

### Claim 20 (depends on 1)

The method of claim 1, further comprising:
training a generative rendition model to generate at least one digital image according to at least one physical constraint by providing the generative rendition model with a plurality of training examples, each comprising a plurality of physical constraints of a simulated driving environment and a real digital image corresponding to the plurality of physical constraints, to produce a trained generative rendition model; and
providing the simulated driving data to the driving assistant model for the purpose of one or more of: training the driving assistant model, verifying the driving assistant model, testing the driving assistant model and validating the driving assistant model;
wherein producing the simulated driving data comprises computing at least one synthetic digital image using the trained generative rendition model by providing the trained generative rendition model with another plurality of physical constraints of another simulated driving environment.

### Claim 21 (depends on 20)

The method of claim 20, wherein the plurality of physical constraints comprise a plurality of three-dimensional (3D) placements of a plurality of objects in the simulated driving environment.

### Claim 22 (depends on 21)

The method of claim 21, wherein the other plurality of physical constraints comprises another plurality of 3D placements of another plurality of objects in the other simulated driving environment.

### Claim 23 (depends on 20)

The method of claim 20, wherein the plurality of physical constraints comprises text in at least one natural language.

### Claim 24 (depends on 23)

The method of claim 23, wherein the other plurality of physical constraints comprises another text in the at least one natural language.

### Claim 25 (depends on 20)

The method of claim 20, wherein the generative rendition model is a previously-trained generative rendition model, trained to generate at least one synthetic digital image in response to data describing an image, the previously-trained generative rendition model trained using a plurality of real digital images.

### Claim 26 (depends on 25)

The method of claim 25 wherein the data describing the image is provided in a natural language.

### Claim 27 (depends on 20)

The method of claim 20, wherein the generative rendition model is a latent diffusion deep neural network.

### Claim 28 (depends on 20)

The method of claim 20, wherein generating the simulated driving data comprises providing the trained generative rendition model with at least one environment-characteristic adjustment value.

### Claim 29

A system for generating a driving assistant model, comprising at least one hardware processor configured to:
compute at least one semantic driving scenario by computing at least one permutation of at least one initial semantic driving scenario;
provide the at least one semantic driving scenario to a simulation generator to produce simulated driving data describing at least one simulated driving environment;
train a driving assistant model using the simulated driving data to produce a trained driving assistant model; and
execute the trained driving assistant model and at least one autonomous driving model, where the trained driving assistant model provides at least one driving instruction to the at least one autonomous driving model while the at least one autonomous driving model is operating.

### Claim 30 (depends on 29)

The system of claim 29, further comprising at least one digital communication network interface connected to the at least one hardware processor;
wherein the trained driving assistant model provides the at least one driving instruction to the at least one autonomous driving model via the at least one digital communication network interface.

### Claim 31 (depends on 29)

The system of claim 29, wherein the at least one initial semantic driving scenario is computed using driving event data describing at least one driving event detected in other driving data collected during operation of at least one other autonomous driving model (shadow driver) in a vehicle driven by a human driver;
wherein at least one sensor is installed in the vehicle in an identified configuration;
wherein the at least one autonomous driving model is installed in another vehicle; and
wherein at least one other sensor is installed in the other vehicle in the identified configuration.

### Claim 32

A software program product for generating a driving assistant model, comprising:
a non-transitory computer readable storage medium;
first program instructions for computing at least one semantic driving scenario by computing at least one permutation of at least one initial semantic driving scenario;
second program instructions for providing the at least one semantic driving scenario to a simulation generator to produce simulated driving data describing at least one simulated driving environment;
third program instructions for training a driving assistant model using the simulated driving data to produce a trained driving assistant model; and
fourth program instructions for execute the trained driving assistant model and at least one autonomous driving model, where the trained driving assistant model provides at least one driving instruction to the at least one autonomous driving model while the at least one autonomous driving model is operating;
wherein the first, second, third and fourth program instructions are executed by at least one computerized processor from the non-transitory computer readable storage medium.
