# US 2021/0239812 A1 · Sensor triggering based on sensor simulation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0239812 A1
- Document ID: US20210239812A1
- Status: Published application
- Kind: A1
- Application number: 16/777,349
- Filed: 2020-01-30
- Published: 2021-08-05
- Earliest claimed: 2020-01-30
- Assignees: Pony AI Inc.
- Applicants: Pony AI Inc.
- Inventors: Pingfan Meng; Zhenhao Pan; Stephen Lee; Wei-yang Chiu; Kai Chen
- Web page: https://patentgrep.com/patent/US20210239812A1

## CPC

- G01S 7/4972: {Alignment of sensor}
- G05B 19/0426: {Programming the control sequence}
- G01S 7/4808: {Evaluating distance, position or velocity data}
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G05B 13/048: {using a predictor}
- G05B 19/0423: {Input/output}
- G06F 3/005: {Input arrangements through a video camera}
- G06N 3/126: Evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 20/00: Machine learning
- G01S 7/52004: {Means for monitoring or calibrating (short-range imaging G01S7/5205)}
- G05B 2219/25428: Field device
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Described herein are systems, methods, and non-transitory computer readable media for triggering a sensor operation of a second sensor (e.g., a camera) based on a predicted time of alignment with a first sensor (e.g., a LiDAR), where operation of the second sensor is simulated to determine the predicted time of alignment. In this manner, the sensor data captured by the two sensors is ensured to be substantially synchronized with respect to the physical environment being sensed. This sensor data synchronization based on predicted alignment of the sensors solves the technical problem of lack of sensor coordination and sensor data synchronization that would otherwise result from the latency associated with communication between sensors and a centralized controller and/or between sensors themselves.

## Claims

### Claim 1

A computer-implemented method for triggering a sensor operation, the method comprising:
simulating operation of a first sensor;
predicting, based at least in part on the simulated operation of the first sensor, a time at which the first sensor will be aligned with a second sensor; and
triggering the second sensor to perform the sensor operation based at least in part on the predicted time.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein simulating operation of the first sensor comprises:
receiving a set of initial parameters associated with the operation of the first sensor; and
generating a virtual sensor system based at least in part on the set of initial parameters.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein generating the virtual sensor system comprises training a predictive model based at least in part on the set of initial parameters, wherein the predictive model is configured to perform the predicting.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, further comprising:
determining an error between the predicted time and an actual time that the first sensor was aligned with a second sensor;
providing feedback data comprising an indication of the error to the predictive model; and
re-training the predictive model based at least in part on the feedback data to improve a predictive capability of the predictive model.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein triggering the second sensor to perform the sensor operation comprises triggering the second sensor to perform the sensor operation at the predicted time.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the time is a first predicted time, the method further comprising:
predicting, using the re-trained predictive model, a second time at which the first sensor will be aligned with the second sensor; and
triggering the second sensor to perform the sensor operation based at least in part on the second predicted time.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the error between the first predicted time and a first actual time that the first sensor was aligned with the second sensor is a first error, the method further comprising:
determining a second error between the second predicted time and a second actual time that the first sensor was aligned with the second sensor,
wherein the second error is smaller than the first error.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the first sensor is a Light Detection and Ranging (LiDAR) sensor and the second sensor is a camera.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the sensor operation is an image capture function of the camera.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, further comprising determining a delay associated with the image capture function of the camera, wherein the delay results from a rolling shutter of the camera that causes the camera to capture different portions of image data of a scene at different times.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein triggering the second sensor to perform the sensor operation comprises triggering the camera to perform the image capture function prior to or after the predicted time based at least in part on the delay.

### Claim 12

A system for triggering a sensor operation, the system comprising:
at least one processor; and
at least one memory storing computer-executable instructions, wherein the at least one processor is configured to access the at least one memory and execute the computer-executable instructions to:
simulate operation of a first sensor;
predict, based at least in part on the simulated operation of the first sensor, a time at which the first sensor will be aligned with a second sensor; and
trigger the second sensor to perform the sensor operation based at least in part on the predicted time.

### Claim 13 (depends on 12)

The system of claim 12, wherein the at least one processor is configured to simulate operation of the first sensor by executing the computer-executable instructions to:
receive a set of initial parameters associated with the operation of the first sensor; and
generate a virtual sensor system based at least in part on the set of initial parameters.

### Claim 14 (depends on 13)

The system of claim 13, wherein the at least one processor is configured to generate the virtual sensor system by executing the computer-executable instructions to train a predictive model based at least in part on the set of initial parameters, wherein the predictive model is configured to perform the predicting.

### Claim 14 (depends on 12)

The system of claim 12, wherein the at least one processor is configured to trigger the second sensor to perform the sensor operation by executing the computer-executable instructions to trigger the second sensor to perform the sensor operation at the predicted time.

### Claim 15 (depends on 14)

The system of claim 14, wherein the at least one processor is further configured to execute the computer-executable instructions to:
determine an error between the predicted time and an actual time that the first sensor was aligned with a second sensor;
provide feedback data comprising an indication of the error to the predictive model; and
re-train the predictive model based at least in part on the feedback data to improve a predictive capability of the predictive model.

### Claim 16 (depends on 15)

The system of claim 15, wherein the time is a first predicted time and the error between the first predicted time and a first actual time that the first sensor was aligned with the second sensor is a first error, and wherein the at least one processor is further configured to execute the computer-executable instructions to:
predict, using the re-trained predictive model, a second time at which the first sensor will be aligned with the second sensor;
trigger the second sensor to perform the sensor operation based at least in part on the second predicted time;
determine a second error between the second predicted time and a second actual time that the first sensor was aligned with the second sensor,
wherein the second error is smaller than the first error.

### Claim 18 (depends on 12)

The system of claim 12, wherein the first sensor is a Light Detection and Ranging (LiDAR) sensor and the second sensor is a camera, and wherein the sensor operation is an image capture function of the camera.

### Claim 19 (depends on 18)

The system of claim 18, wherein the at least one processor is further configured to execute the computer-executable instructions to determine a delay associated with the image capture function of the camera, wherein the delay results from a rolling shutter of the camera that causes the camera to capture different portions of image data of a scene at different times.

### Claim 20 (depends on 19)

The system of claim 19, wherein the at least one processor is configured to trigger the second sensor to perform the sensor operation by executing the computer-executable instructions to trigger the camera to perform the image capture function prior to or after the predicted time based at least in part on the delay.
