# US 2022/0284627 A1 · Vehicle analysis environment with displays for vehicle sensor calibration and/or event simulation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284627 A1
- Document ID: US20220284627A1
- Status: Published application
- Kind: A1
- Application number: 17/194,882
- Filed: 2021-03-08
- Published: 2022-09-08
- Earliest claimed: 2021-03-08
- Assignees: GM Cruise Holdings LLC
- Applicants: GM Cruise Holdings, LLC
- Inventors: Christopher Allen Johnson; Kevin Keogh
- Web page: https://patentgrep.com/patent/US20220284627A1

## CPC

- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- B60Q 5/005: {automatically actuated}
- B60R 1/23: with a predetermined field of view
- B60W 10/18: including control of braking systems
- B60W 10/30: including control of auxiliary equipment, e.g. air-conditioning compressors or oil pumps
- B60W 30/143: {Speed control (B60W30/16 takes precedence)}
- B60W 30/18163: {Lane change; Overtaking manoeuvres}
- B60W 50/06: Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot
- G01S 7/4086: {in a calibrating environment, e.g. anechoic chamber}
- G01S 7/497: Means for monitoring or calibrating
- G01S 7/52004: {Means for monitoring or calibrating (short-range imaging G01S7/5205)}
- G01S 15/931: of land vehicles
- G01S 17/931: of land vehicles
- G06F 3/1423: {controlling a plurality of local displays, e.g. CRT and flat panel display}
- G06F 3/1431: {using a single graphics controller}
- G06F 3/1446: {display composed of modules, e.g. video walls}
- G06K 7/10722: {Photodetector array or CCD scanning}
- G06K 7/1413: {1D bar codes}
- G06K 7/1417: {2D bar codes}
- G06T 7/73: using feature-based methods
- G06T 11/00: Two-dimensional [2D] image generation
- G09G 5/12: Synchronisation between the display unit and other units, e.g. other display units, video-disc players
- H04N 9/31: Projection devices for colour picture display {, e.g. using electronic spatial light modulators [ESLM]}
- H04N 17/002: {for television cameras}
- H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- B60R 2300/10: characterised by the type of camera system used
- B60R 2300/105: using multiple cameras
- B60R 2300/20: characterised by the type of display used
- B60W 60/001: {Planning or execution of driving tasks}
- G01S 7/40: Means for monitoring or calibrating
- G01S 13/931: of land vehicles
- G01S 2013/9323: {Alternative operation using light waves}
- G01S 2013/9324: {Alternative operation using ultrasonic waves}
- G06T 2207/30204: Marker
- G06T 2207/30248: Vehicle exterior or interior
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G09G 3/03: {specially adapted for displays having non-planar surfaces, e.g. curved displays}
- G09G 2380/10: Automotive applications
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A vehicle analysis environment includes one or more display screens, such as a display screen wall or an array of display screens. While a vehicle is in the vehicle analysis environment, a vehicle analysis system renders and displays one or more vehicle sensor calibration targets and/or one or more simulated events on the one or more display screens. Vehicle sensors of the vehicle capture sensor data while in the vehicle analysis environment. The sensor data depict the vehicle sensor calibration targets and/or the simulated events that are displayed on the one or more display screens. The vehicle can output actions based on the simulated event and/or can calibrate its vehicle sensors based on the vehicle sensor calibration targets.

## Claims

### Claim 1

A system for sensor calibration, the system comprising:
a camera coupled to a housing, wherein the camera captures a plurality of camera datasets over a course of a calibration time period while the housing is in an analysis environment;
one or more memory units storing instructions; and
one or more processors that execute the instructions, wherein execution of the instructions by the one or more processors causes the one or more processors to:
receive the plurality of camera datasets from the camera,
identify, within at least a subset of the plurality of camera datasets, a visual representation of at least a portion of one or more displays as the one or more displays display a virtual sensor target, wherein the one or more displays display the virtual sensor target being oriented according to a virtual pose, wherein the one or more displays are positioned in the analysis environment,
identify the virtual pose of the virtual sensor target as displayed on the one or more displays based on a depiction of the virtual sensor target in the visual representation of at least the portion of the one or more displays that display the virtual sensor target, and
calibrate interpretation of data captured by the camera using at least a transformation that is calibrated based on the visual representation of at least the portion of the one or more displays that display the virtual sensor target and based on the virtual pose of the virtual sensor target as identified.

### Claim 2 (depends on 1)

The system of claim 1, wherein the housing is a vehicle.

### Claim 3 (depends on 2)

The system of claim 2, wherein the vehicle is an automobile.

### Claim 4 (depends on 1)

The system of claim 1, wherein the housing is rotated into a plurality of orientations by a motorized turntable over the course of the calibration time period, wherein the camera captures the plurality of camera datasets over the course of the calibration time period by capturing at least one of the plurality of camera datasets while the housing is at each of the plurality of orientations.

### Claim 5 (depends on 4)

The system of claim 4, further comprising the motorized turntable.

### Claim 6 (depends on 1)

The system of claim 1, wherein the one or more displays are configured to move the virtual sensor target as displayed on the one or more displays along a plurality of display positions on the one or more displays over the course of the calibration time period, wherein the camera captures the plurality of camera datasets over the course of the calibration time period by capturing at least one of the plurality of camera datasets while the virtual sensor target is displayed at each of the plurality of display positions on the one or more displays.

### Claim 7 (depends on 1)

The system of claim 1, wherein the one or more displays in the analysis environment include a display screen wall that displays at least the virtual sensor target.

### Claim 8 (depends on 1)

The system of claim 1, wherein the one or more displays in the analysis environment include a projection surface, wherein a projector projects at least the virtual sensor target onto the projection surface.

### Claim 9 (depends on 1)

The system of claim 1, wherein the one or more displays include a curved display surface.

### Claim 10 (depends on 1)

The system of claim 1, wherein identifying the virtual pose of the virtual sensor target includes receiving the virtual pose of the virtual sensor target from a rendering device that renders the virtual sensor target according to the virtual pose for display on the one or more displays.

### Claim 11 (depends on 1)

The system of claim 1, wherein identifying the virtual pose of the virtual sensor target is based on identifying a perspective distortion of a pattern in the visual representation of at least the portion of the one or more displays that display the virtual sensor target, the virtual sensor target including the pattern.

### Claim 12 (depends on 1)

The system of claim 1, wherein the virtual sensor target includes a pattern, wherein the pattern includes at least one of a checkerboard pattern, an ArUco pattern, a quick response (QR) code pattern, a crosshair pattern, a barcode pattern, an Aztec code pattern, and a dot lattice pattern.

### Claim 13 (depends on 1)

The system of claim 1, wherein the virtual sensor target includes at least one virtual polygonal surface.

### Claim 14 (depends on 1)

The system of claim 1, wherein calibrating interpretation of data captured by the camera includes generating a transformation mapping a position associated with the visual representation of at least the portion of the one or more displays that display the virtual sensor target to a position in the analysis environment.

### Claim 15 (depends on 14)

The system of claim 14, wherein execution of the instructions by the one or more processors causes the one or more processors to further:
receive a post-calibration camera dataset from the camera while the housing is in an external position outside of the analysis environment,
identify a representation of an object within a scene within the post-calibration camera dataset, and
identify a position of the object relative to the external position of the housing by applying the transformation to a position of the representation of the object within the scene.

### Claim 16 (depends on 1)

The system of claim 1, further comprising the one or more displays.

### Claim 17

A method for sensor calibration, the method comprising:
receiving a plurality of camera datasets from a camera coupled to a housing, wherein the camera captures the plurality of camera datasets over a course of a calibration time period while the housing is in an analysis environment;
identifying, within at least a subset of the plurality of camera datasets, a visual representation of at least a portion of one or more displays as the one or more displays display a virtual sensor target, wherein the one or more displays display the virtual sensor target being oriented according to a virtual pose, wherein the one or more displays are positioned in the analysis environment;
identifying the virtual pose of the virtual sensor target as displayed on the one or more displays based on a depiction of the virtual sensor target in the visual representation of at least the portion of the one or more displays that display the virtual sensor target; and
calibrating interpretation of data captured by the camera using at least a transformation that is calibrated based on the visual representation of at least the portion of the one or more displays that display the virtual sensor target and based on the virtual pose of the virtual sensor target as identified.

### Claim 18 (depends on 17)

The method of claim 17, wherein the housing is rotated into a plurality of orientations by a motorized turntable over the course of the calibration time period, wherein the camera captures the plurality of camera datasets over the course of the calibration time period by capturing at least one of the plurality of camera datasets while the housing is at each of the plurality of orientations.

### Claim 19 (depends on 17)

The method of claim 17, wherein the one or more displays are configured to move the virtual sensor target as displayed on the one or more displays along a plurality of display positions on the one or more displays over the course of the calibration time period, wherein the camera captures the plurality of camera datasets over the course of the calibration time period by capturing at least one of the plurality of camera datasets while the virtual sensor target is displayed at each of the plurality of display positions on the one or more displays.

### Claim 20

A non-transitory computer readable storage medium having embodied thereon a program, wherein the program is executable by a processor to perform a method of sensor calibration, the method comprising:
receiving a plurality of camera datasets from a camera coupled to a housing, wherein the camera captures the plurality of camera datasets over a course of a calibration time period while the housing is in an analysis environment;
identifying, within at least a subset of the plurality of camera datasets, a visual representation of at least a portion of one or more displays as the one or more displays display a virtual sensor target, wherein the one or more displays display the virtual sensor target being oriented according to a virtual pose, wherein the one or more displays are positioned in the analysis environment;
identifying the virtual pose of the virtual sensor target as displayed on the one or more displays based on a depiction of the virtual sensor target in the visual representation of at least the portion of the one or more displays that display the virtual sensor target; and
calibrating interpretation of data captured by the camera using at least a transformation that is calibrated based on the visual representation of at least the portion of the one or more displays that display the virtual sensor target and based on the virtual pose of the virtual sensor target as identified.
