# US 2020/0404197 A1 · Synchronized Spinning LIDAR and Rolling Shutter Camera System

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0404197 A1
- Document ID: US20200404197A1
- Status: Published application
- Kind: A1
- Application number: 16/556,683
- Filed: 2019-08-30
- Published: 2020-12-24
- Earliest claimed: 2017-09-28
- Assignees: Waymo LLC
- Applicants: Waymo LLC
- Inventors: Blaise Gassend; Benjamin Ingram; Andreas Wendel; Mark Alexander Shand
- Web page: https://patentgrep.com/patent/US20200404197A1

## CPC

- H04N 25/531: by controlling rolling shutters in CMOS SSIS
- G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46)
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- H04N 23/71: Circuitry for evaluating the brightness variation
- H04N 23/73: by influencing the exposure time
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06T 7/90: Determination of colour characteristics
- G06T 17/05: Geographic models
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10144: Varying exposure
- G06T 2207/10152: Varying illumination
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

One example system comprises a LIDAR sensor that rotates about an axis to scan an environment of the LIDAR sensor. The system also comprises one or more cameras that detect external light originating from one or more external light sources. The one or more cameras together provide a plurality of rows of sensing elements. The rows of sensing elements are aligned with the axis of rotation of the LIDAR sensor. The system also comprises a controller that operates the one or more cameras to obtain a sequence of image pixel rows. A first image pixel row in the sequence is indicative of external light detected by a first row of sensing elements during a first exposure time period. A second image pixel row in the sequence is indicative of external light detected by a second row of sensing elements during a second exposure time period.

## Claims

### Claim at position 1

1-20: (canceled)

### Claim 21

A method, comprising:
receiving light detection and ranging (LIDAR) data obtained by a LIDAR sensor coupled to a vehicle, wherein the LIDAR sensor scans an environment of the vehicle based on rotation of the LIDAR sensor about an axis, wherein the rotation of the LIDAR sensor about the axis adjusts a field-of-view (FOV) of the LIDAR sensor, and wherein the LIDAR data comprises a plurality of LIDAR data points indicative of distances to one or more objects in the environment of the vehicle;
receiving image data indicative of the environment of the vehicle obtained by one or more cameras coupled to the vehicle, wherein the one or more cameras operate in a rolling shutter mode that is coordinated with the rotation of the LIDAR sensor, wherein the image data comprises a plurality of image pixels; and
combining the image data and the LIDAR data.

### Claim 22 (depends on 21)

The method of claim 21, wherein the one or more cameras include a plurality of rows of sensing elements, wherein each row of sensing elements is aligned with the axis of rotation of the LIDAR sensor, and wherein the one or more cameras are operated in the rolling shutter mode such that each row of sensing elements senses light in a respective FOV during a respective exposure time period.

### Claim 23 (depends on 22)

The method of claim 22, wherein the rolling shutter mode is coordinated with the rotation of the LIDAR sensor such that the FOV of the LIDAR sensor at least partially overlaps the FOV of each given row of sensing elements during the exposure time period of that given row of sensing elements as the LIDAR sensor rotates about the axis.

### Claim 24 (depends on 21)

The method of claim 21, wherein combining the image data and the LIDAR data comprises:
assigning depth values to individual image pixels in the plurality of image pixels based on the LIDAR data.

### Claim 25 (depends on 21)

The method of claim 21, wherein combining the image data and the LIDAR data comprises:
assigning colors to individual LIDAR data points in the plurality of LIDAR data points based on the image data.

### Claim 26 (depends on 21)

The method of claim 21, further comprising:
determining detection times of one or more image pixels of the plurality of image pixels; and
determining detection times of one or more LIDAR data points of the plurality of LIDAR data points.

### Claim 27 (depends on 26)

The method of claim 26, wherein combining the image data and the LIDAR data comprises:
associating the one or more image pixels with the one or more LIDAR data points based on the detection times of the one or more image pixels being within a threshold of the detection times of the one or more LIDAR data points.

### Claim 28 (depends on 21)

The method of claim 21, further comprising:
determining sensing directions of one or more image pixels of the plurality of image pixels; and
determining sensing directions of one or more LIDAR data points of the plurality of LIDAR data points.

### Claim 29 (depends on 28)

The method of claim 28, wherein combining the image data and the LIDAR data comprises:
associating the one or more image pixels with the one or more LIDAR data points based on the sensing directions of the one or more image pixels matching the sensing directions of the one or more LIDAR data points.

### Claim 30 (depends on 21)

The method of claim 21, wherein combining the image data and the LIDAR data comprises:
generating a three-dimensional representation of the environment that indicates distances to one or more features in the environment along with colors of the one or more features.

### Claim 31 (depends on 21)

The method of claim 21, wherein combining the image data and the LIDAR data comprises:
mapping individual LIDAR data points in the plurality of LIDAR data points to individual image pixels in the plurality of image pixels.

### Claim 32 (depends on 21)

The method of claim 21, wherein combining the image data and the LIDAR data comprises:
mapping individual image pixels in the plurality of image pixels to individual LIDAR data points in the plurality of LIDAR data points.

### Claim 33 (depends on 21)

The method of claim 21, wherein combining the image data and the LIDAR data comprises:
mapping individual LIDAR data points in the plurality of LIDAR data points to a three-dimensional representation of the environment; and
mapping individual image pixels in the plurality of image pixels to the three-dimensional representation of the environment.

### Claim 34

A non-transitory computer readable medium having stored therein instructions executable by a computing device to cause the computing device to perform operations, wherein the operations comprise:
receiving light detection and ranging (LIDAR) data obtained by a LIDAR sensor coupled to a vehicle, wherein the LIDAR sensor scans an environment of the vehicle based on rotation of the LIDAR sensor about an axis, wherein the rotation of the LIDAR sensor about the axis adjusts a field-of-view (FOV) of the LIDAR sensor, and wherein the LIDAR data comprises a plurality of LIDAR data points indicative of distances to one or more objects in the environment of the vehicle;
receiving image data indicative of the environment of the vehicle obtained by one or more cameras coupled to the vehicle, wherein the one or more cameras operate in a rolling shutter mode that is coordinated with the rotation of the LIDAR sensor, wherein the image data comprises a plurality of image pixels; and
combining the image data and the LIDAR data.

### Claim 35 (depends on 34)

The non-transitory computer readable medium of claim 34, wherein the one or more cameras include a plurality of rows of sensing elements, wherein each row of sensing elements is aligned with the axis of rotation of the LIDAR sensor, and wherein the one or more cameras are operated in the rolling shutter mode such that each row of sensing elements senses light in a respective FOV during a respective exposure time period.

### Claim 36 (depends on 35)

The non-transitory computer readable medium of claim 35, wherein the rolling shutter mode is coordinated with the rotation of the LIDAR sensor such that the FOV of the LIDAR sensor at least partially overlaps the FOV of each given row of sensing elements during the exposure time period of that given row of sensing elements as the LIDAR sensor rotates about the axis.

### Claim 37 (depends on 34)

The non-transitory computer readable medium of claim 34, wherein combining the image data and the LIDAR data comprises:
assigning depth values to individual image pixels in the plurality of image pixels based on the LIDAR data.

### Claim 38 (depends on 34)

The non-transitory computer readable medium of claim 34, wherein combining the image data and the LIDAR data comprises:
assigning colors to individual LIDAR data points in the plurality of LIDAR data points based on the image data.

### Claim 39

A system, comprising:
a light detection and ranging (LIDAR) sensor coupled to a vehicle, wherein the LIDAR sensor is configured to scan an environment of the vehicle based on rotation of the LIDAR sensor about an axis and thereby obtain LIDAR data, wherein the rotation of the LIDAR sensor about the axis adjusts a field-of-view (FOV) of the LIDAR sensor, and wherein the LIDAR data comprises a plurality of LIDAR data points indicative of distances to one or more objects in the environment of the vehicle;
one or more cameras coupled to the vehicle, wherein the one or more cameras are configured to obtain image data indicative of the environment of the vehicle, wherein the image data comprises a plurality of image pixels, and wherein the one or more cameras operate in a rolling shutter mode that is coordinated with the rotation of the LIDAR sensor; and
a controller, wherein the controller is configured to receive the LIDAR data from the LIDAR sensor, receive the image data from the one or more cameras, and combine the image data and the LIDAR data.

### Claim 40 (depends on 39)

The system of claim 39, wherein the one or more cameras include a plurality of rows of sensing elements, wherein each row of sensing elements is aligned with the axis of rotation of the LIDAR sensor, wherein the one or more cameras are operated in the rolling shutter mode such that each row of sensing elements senses light in a respective FOV during a respective exposure time period, and wherein the rolling shutter mode is coordinated with the rotation of the LIDAR sensor such that the FOV of the LIDAR sensor at least partially overlaps the FOV of each given row of sensing elements during the exposure time period of that given row of sensing elements as the LIDAR sensor rotates about the axis.
