# US 10,873,711 B2 · 3-dimensional hybrid camera and production system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,873,711 B2 - Document ID: US10873711B2 - Status: Granted - Kind: B2 - Application number: 16/272,511 - Filed: 2019-02-11 - Published: 2019-06-06 - Granted: 2020-12-22 - Earliest claimed: 2009-05-20 - Assignees: Continental Advanced Lidar Solutions US, LLC - Applicants: Continental Advanced Lidar Solutions US, LLC - Inventors: Roger Stettner; Brad Short; Patrick Gilliland; Thomas Laux; Laurent Heughebaert - Web page: https://patentgrep.com/patent/US10873711B2 ## CPC - H04N 23/55: Optical parts specially adapted for electronic image sensors; Mounting thereof - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar - H04N 13/189: Recording image signals; Reproducing recorded image signals - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 23/11: for generating image signals from visible and infrared light wavelengths - H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils - H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor - H04N 13/254: in combination with electromagnetic radiation sources for illuminating objects - H04N 25/41: {Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES ## Abstract A three-dimensional imaging system includes a ladar sensor with a first field of view adapted to produce a three-dimensional image. The system also includes a visible light camera with a second field of view overlapping the first field of view and adapted to produce a two-dimensional image output. At least one digital processor is connected to the ladar sensor and the visible light camera and adapted to merge the three-dimensional image output with the two-dimensional image output into a three-dimensional point cloud output. ## Claims ### Claim 1 A three-dimensional imaging system comprising: a ladar sensor having a first field of view, said ladar sensor including a laser transmitter configured to produce a modulated laser light output upon command from a system controller, said laser transmitter providing a zero range reference output indicating the initiation of laser light output, an optical receiver comprised of an array of optical detector elements, each of said detector elements connected to a unit cell electrical circuit, each unit cell electrical circuit also connected to said zero range reference output, each unit cell electrical circuit adapted to measure the range to a reflecting surface in the first field of view and to produce a pixel range output, and a readout integrated circuit connected to each unit cell electrical circuit and adapted to produce a three-dimensional image output comprised of a plurality of the pixel range outputs; a visible light camera having a second field of view overlapping the first field of view and adapted to produce a two-dimensional image output; and at least one digital processor connected via a first communications link to said at least one ladar sensor and connected via a second communications link to said visible light camera; said at least one digital processor adapted to merge the three-dimensional image output with the two-dimensional image output into a three-dimensional point cloud output. ### Claim 2 (depends on 1) The three-dimensional imaging system of claim 1 wherein said first communications link and said second communications link are electrical connections. ### Claim 3 (depends on 1) The three-dimensional imaging system of claim 1 wherein said first communications link and said second communications link have at least one fiber optic waveguide. ### Claim 4 (depends on 1) The three-dimensional imaging system of claim 1 wherein said digital processor merges said three-dimensional output with said two-dimensional output using spatial overlay. ### Claim 5 (depends on 1) The three-dimensional imaging system of claim 1 wherein said digital processor merges said three-dimensional image output with said two-dimensional image output by using a compression algorithm to compress the two-dimensional image output. ### Claim 6 (depends on 1) The three-dimensional imaging system of claim 1 wherein said digital processor merges said three-dimensional image output with said two-dimensional image output using wavelets. ### Claim 7 (depends on 1) The three-dimensional imaging system of claim 1 wherein said two dimensional array of optical detector elements is comprised of avalanche photodiodes. ### Claim 8 A three-dimensional imaging system comprising: a first ladar sensor having a first field of view and first optical axis; a second ladar sensor having a second field of view and second optical axis, the second field of view overlapping the first field of view, and the second optical axis having a converging angle with respect to the first optical axis; said first ladar sensor and second ladar sensor each having a laser transmitter configured to produce a modulated laser light output upon command from a system controller, said laser transmitter providing a zero range reference output indicating the initiation of laser light output, an optical receiver comprised of an array of optical detector elements, a plurality of unit cell electrical circuits, each unit cell electrical circuit connected to one of said optical detector elements and said zero range reference output, each unit cell electrical circuit adapted to measure the range to a reflecting surface in the field of view and to produce a pixel range output, and a readout integrated circuit connected to each unit cell electrical circuit, and adapted to produce a three-dimensional image output comprised of a plurality of the pixel range outputs; a visible light camera having a field of view overlapping at least one of the first field of view and the second field of view, and adapted to produce a two-dimensional image output; and a digital processor connected via a first communications link to said first ladar sensor, connected to said second ladar sensor via a second communications link, and connected via a third communications link to said visible light camera; said digital processor adapted to merge said three-dimensional image outputs with said two-dimensional image output into a three-dimensional point cloud output. ### Claim 9 (depends on 8) The three-dimensional imaging system of claim 8 wherein said first communications link and said second communications link are electrical connections. ### Claim 10 (depends on 8) The three-dimensional imaging system of claim 8 wherein said first communications link and said second communications link have at least one fiber optic waveguide. ### Claim 11 (depends on 8) The three-dimensional imaging system of claim 8 wherein said digital processor merges said three-dimensional image outputs with said two-dimensional image output using spatial overlay. ### Claim 12 (depends on 8) The three-dimensional imaging system of claim 8 wherein said digital processor merges said three-dimensional image outputs with said two-dimensional image output by using a compression algorithm to compress the two-dimensional image output. ### Claim 13 (depends on 8) The three-dimensional imaging system of claim 8 wherein said digital processor merges said three-dimensional image output with said two-dimensional image output using wavelets. ### Claim 14 (depends on 8) The three-dimensional imaging system of claim 8 wherein said array of optical detector elements is comprised of avalanche photodiodes. ### Claim 15 An object recognition system comprising: a ladar sensor having a first field of view, said ladar sensor including a laser transmitter producing a modulated laser light output upon command from a system controller, said laser transmitter providing a zero range reference output indicating the initiation of laser light output, an optical receiver comprised of an array of optical detector elements, a plurality of unit cell electrical circuits, each unit cell electrical circuit connected to one of said optical detector elements and said zero range reference output, each of said unit cell electrical circuits adapted to measure the range to a reflecting surface in the first field of view and to produce a pixel range output, and a readout integrated circuit connected to each unit cell electrical circuit, and adapted to produce a three-dimensional image output comprised of a plurality of said pixel range outputs; a visible light camera having a second field of view overlapping the first field of view and adapted to produce a two-dimensional image output; a digital processor connected via a first communications link to said ladar sensor and connected via a second communications link to said at least one visible light camera; said digital processor adapted to merge said three-dimensional image output with said two-dimensional image output into a three-dimensional point cloud output frame; and and an image processor adapted to search a plurality of three-dimensional point cloud output frames and to identify a number of objects in the first field of view. ### Claim 16 (depends on 15) The three-dimensional imaging system of claim 15 wherein said first communications link and said second communications link are electrical connections. ### Claim 17 (depends on 15) The three-dimensional imaging system of claim 15 wherein said first communications link and said second communications link have at least one fiber optic waveguide. ### Claim 18 (depends on 15) The three dimensional imaging system of claim 15 wherein said digital processor merges said three-dimensional image outputs with said two-dimensional image output using spatial overlay. ### Claim 19 (depends on 15) The three dimensional imaging system of claim 15 wherein said digital processor merges said three-dimensional image outputs with said two-dimensional image output by using a compression algorithm to compress the two-dimensional image output. ### Claim 20 (depends on 15) The three dimensional imaging system of claim 15 wherein said digital processor merges said three-dimensional image output with said two-dimensional image output using wavelets.