# US 12,223,680 B2 · Validation of virtual camera models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,680 B2 - Document ID: US12223680B2 - Status: Granted - Kind: B2 - Application number: 17/552,584 - Filed: 2021-12-16 - Published: 2023-06-22 - Granted: 2025-02-11 - Earliest claimed: 2021-12-16 - Assignees: GM Cruise Holdings LLC - Applicants: GM Cruise Holdings LLC - Inventors: Kaitlyn Williams; Justin DeCell; Glenn Sweeney - Web page: https://patentgrep.com/patent/US12223680B2 ## CPC - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - B60W 50/06: Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot - B60W 60/001: {Planning or execution of driving tasks} - B60W 2420/403: Image sensing, e.g. optical camera - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT ## Abstract The present technology pertains to validating a virtual camera in a simulation environment by utilizing an improved camera model to provide quantitative measurements of the model's performance. A method of validating a virtual camera in a simulation environment comprises presenting at least one reference chart in the simulated environment and capturing images of the reference chart in the simulated environment using a virtual camera. The method further includes interrupting an image pipeline of the virtual camera after at least one simulated process in the image pipeline to extract a RAW image. The method analyzes the RAW image to derive measurements of metrics to characterize the virtual camera. The measured metrics are compared to metrics of a calibrated real-world camera to verify that the metrics are within a threshold delta that is indicative that the virtual camera sufficiently approximates the real-world camera. ## Claims ### Claim 1 A method of validating a simulated camera in a simulation environment comprising: presenting at least one reference chart in the simulated environment; capturing images of the at least one reference chart in the simulated environment using the simulated camera; interrupting an image pipeline of the simulated camera after at least one simulated process in the image pipeline to extract a raw image, wherein the simulated camera is a simulation of a real-world camera and the simulation of the simulated camera includes a simulation of components of the real-world camera and the image pipeline of the real-world camera, wherein extracting at least one raw image from the image signal processor occurs after a particular image processing step; analyzing the raw image to derive measurements of metrics to characterize the simulated camera; and comparing the measurements of metrics to metrics of a calibrated real-world camera to verify that the measurements of metrics are within a threshold delta that is indicative that the simulated camera sufficiently approximates the real-world camera. ### Claim 2 (depends on 1) The method of claim 1, wherein the reference chart is one of an ITUHDR chart, a MacBeth chart, a eSFR chart, or an ITCRC chart. ### Claim 3 (depends on 2) The method of claim 2 wherein comparing the measurements of metrics to the metrics of the calibrated real-world camera to verify that the measurements of metrics are within the threshold delta that is indicative that the simulated camera sufficiently approximates the real-world camera further comprises: comparing the measurements of metrics to published reference values that characterizes the reference chart, wherein the MacBeth chart is an example of a chart with published reference values. ### Claim 4 (depends on 2) The method of claim 2 wherein comparing the measurements of metrics to the metrics of the calibrated real-world camera to verify that the measurements of metrics are within the threshold delta that is indicative that the simulated camera sufficiently approximates the real-world camera further comprises: utilizing the calibrated real-world camera to capture images of the reference chart to obtain reference values; and comparing the measurements of metrics to the obtained reference values that characterizes the reference chart, wherein the ITU HDR chart is an example of a chart for which reference values are obtained. ### Claim 5 (depends on 1) The method of claim 1, wherein the presenting at least one reference chart in the simulated environment further comprises: presenting the at least reference chart under known conditions in simulation for measured lighting, distances, and camera settings. ### Claim 6 A system for validating a simulated camera in a simulation environment, comprising: a storage configured to store instructions; a processor configured to execute the instructions and cause the processor to: present at least one reference chart in the simulated environment; capture images of the at least one reference chart in the simulated environment using the simulated camera; interrupt an image pipeline of the simulated camera after at least one simulated process in the image pipeline to extract a raw image, wherein the simulated camera is a simulation of a real-world camera and the simulation of the simulated camera includes a simulation of components of the real-world camera and the image pipeline of the real-world camera, wherein extracting at least one raw image from the image signal processor occurs after a particular image processing step; analyze the raw image to derive measurements of metrics to characterize the simulated camera; and compare the measurements of metrics to metrics of a calibrated real-world camera to verify that the measurements of metrics are within a threshold delta that is indicative that the simulated camera sufficiently approximates the real-world camera. ### Claim 7 (depends on 6) The system of claim 6, wherein the reference chart is one of an ITUHDR chart, a MacBeth chart, a eSFR chart, or an ITCRC chart. ### Claim 8 (depends on 7) The system of claim 7, wherein the comparing the measurements of metrics to the metrics of a calibrated real-world camera to verify that the measurements of metrics are within the threshold delta that is indicative that the simulated camera sufficiently approximates the real-world camera further includes instructions to cause the processor to: compare the measurements of metrics to published reference values that characterizes the reference chart, wherein the MacBeth chart is an example of a chart with published reference values. ### Claim 9 (depends on 7) The system of claim 7, wherein the processor is configured to execute the instructions and cause the processor to: utilize the calibrated real-world camera to capture images of the reference chart to obtain reference values; and compare the measurements of metrics to the obtained reference values that characterizes the reference chart, wherein the ITU HDR chart is an example of a chart for which reference values are obtained. ### Claim 10 (depends on 6) The system of claim 6, wherein the presenting at least one reference chart in the simulated environment includes the processor executing the instructions to: present the at least reference chart under known conditions in simulation for measured lighting, distances, and camera settings. ### Claim 11 A non-transitory computer readable medium comprising instructions, the instructions, when executed by a computing system, cause the computing system to: present at least one reference chart in the simulated environment; capture images of the at least one reference chart in the simulated environment using a simulated camera; interrupt an image pipeline of the simulated camera after at least one simulated process in the image pipeline to extract a raw image, wherein the simulated camera is a simulation of a real-world camera and the simulation of the simulated camera includes a simulation of components of the real-world camera and the image pipeline of the real-world camera, wherein extracting at least one raw image from the image signal processor occurs after a particular image processing step; analyze the raw image to derive measurements of metrics to characterize the simulated camera; and compare the measurements of metrics to metrics of a calibrated real-world camera to verify that the measurements of metrics are within a threshold delta that is indicative that the simulated camera sufficiently approximates the real-world camera. ### Claim 12 (depends on 11) The computer readable medium of claim 11, the reference chart is one of an ITUHDR chart, a MacBeth chart, a eSFR chart, or an ITCRC chart. ### Claim 13 (depends on 12) The computer readable medium of claim 12, wherein the comparing the measurements of metrics to the metrics of the calibrated real-world camera to verify that the measurements of metrics are within the threshold delta that is indicative that the simulated camera sufficiently approximates the real-world camera, wherein the computer readable medium further comprises instructions that, when executed by the computing system, cause the computing system to: compare the measurements of metrics to published reference values that characterizes the reference chart, wherein the MacBeth chart is an example of a chart with published reference values. ### Claim 14 (depends on 12) The computer readable medium of claim 12, wherein the computer readable medium further comprises instructions that, when executed by the computing system, cause the computing system to: utilize the calibrated real-world camera to capture images of the reference chart to obtain reference values; and compare the measurements of metrics to the obtained reference values that characterizes the reference chart, wherein the ITU HDR chart is an example of a chart for which reference values are obtained.