# US 10,120,066 B2 · Apparatus for making a distance determination > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,120,066 B2 - Document ID: US10120066B2 - Status: Granted - Kind: B2 - Application number: 15/169,049 - Filed: 2016-05-31 - Published: 2017-06-29 - Granted: 2018-11-06 - Earliest claimed: 2015-12-23 - Assignees: STMicroelectronics (Research & Development) Limited - Applicants: STMicroelectronics (Research & Development) Limited - Inventors: Xiaoyong Yang; Neale Dutton; Kevin Channon - Web page: https://patentgrep.com/patent/US10120066B2 ## CPC - H04N 23/651: {for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera} - G01S 7/4816: {of receivers alone} - G01S 7/483: Details of pulse systems - G01S 17/10: using transmission of interrupted, pulse-modulated waves (determination of distance by phase measurements G01S17/32) - G01S 17/46: Indirect determination of position data - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar - G06T 7/593: from stereo images - H04N 13/128: Adjusting depth or disparity - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/243: using three or more two-dimensional [2D] image sensors - H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor - H04N 13/254: in combination with electromagnetic radiation sources for illuminating objects - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - H04N 23/57: Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - H04N 2013/0092: {Image segmentation from stereoscopic image signals} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An apparatus includes a camera module configured to generate at least one image and a ToF SPAD based range detecting module configured to generate at least one distance determination to an object within a field of view of the camera module. A processor receives the at least one image from the camera module output and receives the at least one distance determination from the ToF SPAD based range detecting module. This data is processed by the processor to determine a depth map. ## Claims ### Claim 1 An apparatus, comprising: a camera module configured to generate a first image and a second image taken from offset positions with partially overlapping fields of view; at least one time of flight (ToF) single photon avalanche diode (SPAD) based range detecting module configured to generate at least one distance determination between the apparatus and an object within the partially overlapping fields of view; and a processor configured to receive the first and second images from the camera module and the at least one distance determination from the ToF SPAD based range detecting module and determine a depth map based on depth information obtained from processing the first and second images from the camera module and the at least one distance determination; wherein the processor is further configured, in response to an ambient light condition, to: determine the depth map substantially based on the at least one distance determination from the ToF SPAD based range detecting module if the ambient light condition indicates a relatively low ambient light; determine the depth map substantially based on the first and second images from the camera module if the ambient light condition indicates a relatively high ambient light; and determine the depth map based on a combination of the at least one distance determination and the first and second images if the ambient light condition indicates neither relatively low nor relatively high ambient light. ### Claim 2 An apparatus, comprising: a camera module configured to generate a first image and a second image taken from offset positions with partially overlapping fields of view; at least one time of flight (ToF) single photon avalanche diode (SPAD) based range detecting module configured to generate at least one distance determination between the apparatus and an object within the partially overlapping fields of view; and a processor configured to receive the first and second images from the camera module and the at least one distance determination from the ToF SPAD based range detecting module and determine a depth map based on depth information obtained from processing the first and second images from the camera module and the at least one distance determination; wherein the processor is further configured to: determine the depth map substantially based on the at least one distance determination from the ToF SPAD based range detecting module if speed of movement of the object is relatively fast; determine the depth map substantially based on the at least one image from the at least one camera module if speed of movement of the object is relatively slow; and determine the depth map based on a combination of the at least one distance determination and the at least one image if speed of movement of the object is neither fast nor slow or is both fast and slow. ### Claim 3 An apparatus, comprising: a camera module configured to generate a first image and a second image taken from offset positions with partially overlapping fields of view; at least one time of flight (ToF) single photon avalanche diode (SPAD) based range detecting module configured to generate at least one distance determination between the apparatus and an object within the partially overlapping fields of view; and a processor configured to receive the first and second images from the camera module and the at least one distance determination from the ToF SPAD based range detecting module and determine a depth map based on depth information obtained from processing the first and second images from the camera module and the at least one distance determination; wherein the processor is further configured to: determine the depth map substantially based on the at least one distance determination from the ToF SPAD based range detecting module if a desired accuracy of object detection is a relatively high accuracy; determine the depth map substantially based on the at least one image from the at least one camera module if the desired accuracy of object detection is a relatively low accuracy; and determine the depth map based on a combination of the at least one distance determination and the at least one image if the desired accuracy of object detection is neither low nor high accuracy. ### Claim 4 An apparatus, comprising: a camera module configured to generate a first image and a second image taken from offset positions with partially overlapping fields of view; at least one time of flight (ToF) single photon avalanche diode (SPAD) based range detecting module configured to generate at least one distance determination between the apparatus and an object within the partially overlapping fields of view; and a processor configured to receive the first and second images from the camera module and the at least one distance determination from the ToF SPAD based range detecting module and determine a depth map based on depth information obtained from processing the first and second images from the camera module and the at least one distance determination; wherein the processor is further configured to: determine the depth map substantially based on the at least one distance determination from the ToF SPAD based range detecting module when a range to the object is a macro or near distance; determine the depth map substantially based on the at least one image from the at least one camera module when the range to the object is a far distance; and determine the depth map based on a combination of the at least one distance determination and the at least one image when the range to the object is neither near or far distance. ### Claim 5 An apparatus, comprising: a camera module configured to generate a first image and a second image taken from offset positions with partially overlapping fields of view; at least one time of flight (ToF) single photon avalanche diode (SPAD) based range detecting module configured to generate at least one distance determination between the apparatus and an object within the partially overlapping fields of view; and a processor configured to receive the first and second images from the camera module and the at least one distance determination from the ToF SPAD based range detecting module and determine a depth map based on depth information obtained from processing the first and second images from the camera module and the at least one distance determination; wherein the processor is further configured to: determine the depth map substantially based on the at least one distance determination from the ToF SPAD based range detecting module when the apparatus is operating in a low power consumption mode; determine the depth map substantially based on the at least one image from the at least one camera module when the apparatus is operating in a high power consumption mode; and determine the depth map based on a combination of the at least one distance determination and the at least one image when the apparatus is operating in neither low nor high power consumption modes. ### Claim 6 (depends on 5) The apparatus as claimed in claim 5, wherein the camera module comprises two cameras separated by a determined distance, wherein the at least one ToF SPAD based range detecting module is configured to be located within a volume defined by the determined distance. ### Claim 7 (depends on 5) The apparatus as claimed in claim 5, wherein the camera module comprises a multi-aperture camera. ### Claim 8 (depends on 5) The apparatus as claimed in claim 5, wherein the processor is configured to: determine at least one point on the depth map which is spatially co-incident with the at least one distance determination from the ToF SPAD based range detecting module; and calibrate the depth map at the at least one point based on the spatially co-incident distance determination. ### Claim 9 (depends on 1) The apparatus as claimed in claim 1, wherein the camera module comprises two cameras separated by a determined distance, wherein the at least one ToF SPAD based range detecting module is configured to be located within a volume defined by the determined distance. ### Claim 10 (depends on 1) The apparatus as claimed in claim 1, wherein the camera module comprises a multi-aperture camera. ### Claim 11 (depends on 1) The apparatus as claimed in claim 1, wherein the processor is configured to: determine at least one point on the depth map which is spatially co-incident with the at least one distance determination from the ToF SPAD based range detecting module; and calibrate the depth map at the at least one point based on the spatially co-incident distance determination. ### Claim 12 (depends on 2) The apparatus as claimed in claim 2, wherein the camera module comprises two cameras separated by a determined distance, wherein the at least one ToF SPAD based range detecting module is configured to be located within a volume defined by the determined distance. ### Claim 13 (depends on 2) The apparatus as claimed in claim 2, wherein the camera module comprises a multi-aperture camera. ### Claim 14 (depends on 2) The apparatus as claimed in claim 2, wherein the processor is configured to; determine at least one point on the depth map which is spatially co-incident with the at least one distance determination from the ToF SPAD based range detecting module; and calibrate the depth map at the at least one point based on the spatially co-incident distance determination. ### Claim 15 (depends on 3) The apparatus as claimed in claim 3, wherein the camera module comprises two cameras separated by a determined distance, wherein the at least one ToF SPAD based range detecting module is configured to be located within a volume defined by the determined distance. ### Claim 16 (depends on 3) The apparatus as claimed in claim 3, wherein the camera module comprises a multi-aperture camera. ### Claim 17 (depends on 3) The apparatus as claimed in claim 3, Wherein the processor is configured to: determine at least one point on the depth map which is spatially co-incident with the at least one distance determination from the ToF SPAD based range detecting module; and calibrate the depth map at the at least one point based on the spatially co-incident distance determination. ### Claim 18 (depends on 4) The apparatus as claimed in claim 4, wherein the camera module comprises two cameras separated by a determined distance, wherein the at least one ToF SPAD based range detecting module is configured to be located within a volume defined by the determined distance. ### Claim 19 (depends on 4) The apparatus as claimed in claim 4, wherein the camera module comprises a multi-aperture camera. ### Claim 20 (depends on 4) The apparatus as claimed in claim 4, wherein the processor is configured to: determine at least one point on the depth map which is spatially co-incident with the at least one distance determination from the ToF SPAD based range detecting module; and calibrate the depth map at the at least one point based on the spatially co-incident distance determination.