# US 11,910,099 B2 · Real-time HDR video for vehicle control > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,910,099 B2 - Document ID: US11910099B2 - Status: Granted - Kind: B2 - Application number: 17/840,108 - Filed: 2022-06-14 - Published: 2022-09-29 - Granted: 2024-02-20 - Earliest claimed: 2016-08-09 - Assignees: Contrast, Inc. - Applicants: Contrast, Inc. - Inventors: Willie C. Kiser; Nora Tocci; Michael D. Tocci - Web page: https://patentgrep.com/patent/US11910099B2 ## CPC - H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors - B60R 1/27: providing all-round vision, e.g. using omnidirectional cameras - G02B 27/106: {for splitting or combining a plurality of identical beams or images, e.g. image replication} - G02B 27/1066: {for enhancing image performance, like resolution, pixel numbers, dual magnifications or dynamic range, by tiling, slicing or overlapping fields of view} - G02B 27/144: {using partially transparent surfaces without spectral selectivity (G02B27/147 takes precedence)} - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/582: {of traffic signs} - H04N 9/76: for mixing of colour signals (H04N9/75 takes precedence) - H04N 23/10: for generating image signals from different wavelengths - H04N 23/12: with one sensor only - H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images - H04N 23/57: Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - H04N 23/85: for matrixing - H04N 23/88: for colour balance, e.g. white-balance circuits or colour temperature control - B60R 2300/102: using 360 degree surveillance camera system - B60R 2300/30: characterised by the type of image processing - G05D 1/0246: {using a video camera in combination with image processing means} - G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras - G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The invention provides an autonomous vehicle with a video camera that merges images taken a different light levels by replacing saturated parts of an image with corresponding parts of a lower-light image to stream a video with a dynamic range that extends to include very low-light and very intensely lit parts of a scene. The high dynamic range (HDR) camera streams the HDR video to a HDR system in real time—as the vehicle operates. As pixel values are provided by the camera's image sensors, those values are streamed directly through a pipeline processing operation and on to the HDR system without any requirement to wait and collect entire images, or frames, before using the video information. ## Claims ### Claim 1 An HDR system comprising: an HDR camera comprising a plurality of image sensors coupled to a processing device, wherein the HDR camera is operable to stream pixel values from each of the plurality of image sensors in a frame-independent manner through a pipeline on the processing device, wherein the pipeline includes a synchronization module that synchronizes the pixel values, and a kernel operation that identifies saturated pixel values and a merge module to merge the pixel values to produce a real-time HDR video; and a processing system configured to communicate with the HDR camera and a control system of a vehicle, wherein the processing system is operable to determine, based on the HDR video, a characteristic of a feature in the vehicle's environment and to issue to the control system an instruction that directs a change in the operation of the vehicle based on the characteristic. ### Claim 2 (depends on 1) The system of claim 1, wherein the synchronization module contains line buffers that circulate early-arriving pixel values and release the early-arriving pixel values simultaneous with corresponding later-arriving pixel values. ### Claim 3 (depends on 2) The system of claim 2, wherein the HDR camera captures a 360-degree view around the vehicle. ### Claim 4 (depends on 3) The system of claim 3, wherein the captured 360-degree view is ring-shaped, and further wherein the processing system performs a de-wrapping process to convert the 360- degree view into a rectangular panoramic image. ### Claim 5 (depends on 4) The system of claim 4, further comprising detection and ranging sensors, wherein the processing system is operable to detect an object with the detection and ranging sensor, detect the object with HDR camera, and correlate an image of the object in the HDR video with a detected range of the objection determined via the detection and ranging system. ### Claim 6 (depends on 2) The system of claim 2, wherein the processing system is operable to detect glare in the environment within the 360-degree view and use the HDR camera to capture an HDR image of a portion of the environment affected by the glare. ### Claim 7 (depends on 2) The system of claim 2, wherein the camera further comprises a lens and at least one beamsplitter. ### Claim 8 (depends on 7) The system of claim 7, wherein the plurality of image sensors include at least first sensor that receives images at a first exposure and a second sensor that receives the images at a second exposure, wherein the second exposure is lower than the first exposure. ### Claim 9 (depends on 8) The system of claim 8, wherein the first sensor, the second sensor, the lens and the at least one beamsplitter are arranged to receive an incoming beam of light and split the beam of light into at least a first path that impinges and first sensor and a second path that impinges on the second sensor. ### Claim 10 (depends on 9) The system of claim 9, wherein the beamsplitter directs a majority of the light to the first path and a lesser amount of the light to the second path. ### Claim 11 (depends on 10) The system of claim 10, wherein the first path and the second path impinge on the first sensor and the second sensor, respectively, to generate images that are optically identical but for light level. ### Claim 12 (depends on 11) The system of claim 11, wherein the processing device comprises a field- programmable gate array or an application-specific integrated circuit that includes the pipeline. ### Claim 13 (depends on 12) The system of claim 12, wherein the kernel operation operates on pixel values as they stream from each of the plurality of image sensors by examining, for a given pixel on the HE sensor, values from a neighborhood of pixels surrounding the given pixel, finding saturated values in the neighborhood of pixels, and using information from a corresponding neighborhood on the ME sensor to estimate a value for the given pixel. ### Claim 14 (depends on 13) The system of claim 13, wherein the pipeline includes—in the order in which the pixel values flow: the synchronization; the kernel operation; the merge module; a demosaicing module; and a tone-mapping operator. ### Claim 15 (depends on 14) The system of claim 14, wherein the pipeline further comprises one or more of a color-correction module; an HDR conversion module; and an HDR compression module.