# US 2021/0174530 A1 · Method and System for Multiple Stereo Based Depth Estimation and Collision Warning/Avoidance Utilizing the Same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0174530 A1 - Document ID: US20210174530A1 - Status: Published application - Kind: A1 - Application number: 17/178,727 - Filed: 2021-02-18 - Published: 2021-06-10 - Earliest claimed: 2017-12-29 - Assignees: PlusAI, Inc. - Applicants: Plusai Corp - Inventors: Anurag Ganguli; Timothy Patrick Daly, JR.; Hao Zheng; David Wanqian Liu - Web page: https://patentgrep.com/patent/US20210174530A1 ## CPC - G06T 7/593: from stereo images - G06T 7/596: {from three or more stereo images} - B60Q 9/008: {for anti-collision purposes} - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - H04N 13/156: Mixing image signals - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/243: using three or more two-dimensional [2D] image sensors - H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor - H04N 13/293: Generating mixed stereoscopic images; Generating mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T 2207/30261: Obstacle - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The present teaching relates to method, system, medium, and implementation of determining depth information in autonomous driving. Stereo images are first obtained from multiple stereo pairs selected from at least two stereo pairs. The at least two stereo pairs have stereo cameras installed with the same baseline and in the same vertical plane. Left images from the multiple stereo pairs are fused to generate a fused left image and right images from the multiple stereo pairs are fused to generate a fused right image. Disparity is then estimated based on the fused left and right images and depth information can be computed based on the stereo images and the disparity. ## Claims ### Claim 1 A method implemented on a computer having at least one processor, a storage, and a communication platform for determining depth information in autonomous driving, comprising: obtaining a first pair of stereo images from a first pair of stereo cameras deployed on a moving vehicle, wherein the first pair of stereo cameras have a wider field of view and are installed with respect to a baseline in a vertical plane for acquiring images of a scene surrounding the moving vehicle; detecting one or more objects appearing in different regions in the first pair of stereo images corresponding to different areas in the scene; for each of the one or more objects detected in a particular region of a stereo image corresponding to a particular area in the scene, determining a second pair of stereo cameras installed with respect to the baseline in the vertical plane with a narrower field of view that are configured to capture images of the particular area of the scene, receiving a second pair of stereo images from the second pair of stereo cameras, fusing each of the second pair of stereo images with the particular region of a corresponding image of the first pair of stereo images; estimating disparity based on fused stereo images; and computing depth information of each of the one or more objects based on the fused stereo images and the disparity. ### Claim 2 (depends on 1) The method of claim 1, wherein each of the first pair of stereo images acquired by the first pair of stereo cameras have a coarser resolution; and each of the second pair of stereo images acquired by the second pair of stereo cameras has a higher resolution. ### Claim 3 (depends on 2) The method of claim 2, wherein each of the regions corresponding to one of the one or more objects in the fused stereo images has a higher resolution; and remaining regions in the fused stereo images have a coarser resolution. ### Claim 4 (depends on 2) The method of claim 2, wherein the higher resolution for the regions corresponding to the one or more objects enables more accurate estimation of disparity. ### Claim 5 (depends on 4) The method of claim 4, wherein the more accurate estimation of disparity facilitate enhanced estimation of the depth information for each of the one or more objects. ### Claim 6 (depends on 1) The method of claim 1, further comprising: determining an avoidance strategy with respect to each of the one or more objects based on the depth information; and generating a collision warning signal in accordance with the avoidance strategy. ### Claim 7 Machine readable and non-transitory medium having information stored thereon for determining depth information in autonomous driving, wherein the information, when read by the machine, causes the machine to perform the following: obtaining a first pair of stereo images from a first pair of stereo cameras deployed on a moving vehicle, wherein the first pair of stereo cameras have a wider field of view and are installed with respect to a baseline in a vertical plane for acquiring images of a scene surrounding the moving vehicle; detecting one or more objects appearing in different regions in the first pair of stereo images corresponding to different areas in the scene; for each of the one or more objects detected in a particular region of a stereo image corresponding to a particular area in the scene, determining a second pair of stereo cameras installed with respect to the baseline in the vertical plane with a narrower field of view that are configured to capture images of the particular area of the scene, receiving a second pair of stereo images from the second pair of stereo cameras, fusing each of the second pair of stereo images with the particular region of a corresponding image of the first pair of stereo images; estimating disparity based on fused stereo images; and computing depth information of each of the one or more objects based on the fused stereo images and the disparity. ### Claim 8 (depends on 7) The medium of claim 7, wherein each of the first pair of stereo images acquired by the first pair of stereo cameras have a coarser resolution; and each of the second pair of stereo images acquired by the second pair of stereo cameras has a higher resolution. ### Claim 9 (depends on 8) The medium of claim 8, wherein each of the regions corresponding to one of the one or more objects in the fused stereo images has a higher resolution; and remaining regions in the fused stereo images have a coarser resolution. ### Claim 10 (depends on 8) The medium of claim 8, wherein the higher resolution for the regions corresponding to the one or more objects enables more accurate estimation of disparity. ### Claim 11 (depends on 10) The medium of claim 10, wherein the more accurate estimation of disparity facilitate enhanced estimation of the depth information for each of the one or more objects. ### Claim 12 (depends on 7) The medium of claim 7, wherein the information, when read by the machine, further causes the machine to perform the following: determining an avoidance strategy with respect to each of the one or more objects based on the depth information; and generating a collision warning signal in accordance with the avoidance strategy. ### Claim 13 A system for determining depth information in autonomous driving, comprising: a first pair of stereo cameras configured for obtaining a first pair of stereo images, wherein the first pair of stereo cameras are deployed on a moving vehicle and installed with respect to a baseline in a vertical plane for acquiring images of a scene surrounding the moving vehicle with a wider field of view; an object recognition unit configured for detecting one or more objects appearing in different regions in the first pair of stereo images corresponding to different areas in the scene; an adaptive image acquisition unit configured for, with respect to each of the one or more objects detected in a particular region of a stereo image corresponding to a particular area in the scene, selecting a second pair of stereo cameras, wherein the second pair of stereo cameras are installed with respect to the baseline in the vertical plane having a narrower field of view and configured for capturing a second pair of stereo images of the particular area of the scene; an image fusion unit configured for fusing each of the second pair of stereo images with the particular region of a corresponding image of the first pair of stereo images; a stereo based disparity estimator configured for estimating disparity based on fused stereo images; and a multi-stereo based depth estimator configured for computing depth information of each of the one or more objects based on the fused stereo images and the disparity. ### Claim 14 (depends on 13) The system of claim 13, wherein each of the first pair of stereo images acquired by the first pair of stereo cameras have a coarser resolution; and each of the second pair of stereo images acquired by the second pair of stereo cameras has a higher resolution. ### Claim 15 (depends on 14) The system of claim 14, wherein each of the regions corresponding to one of the one or more objects in the fused stereo images has a higher resolution; and remaining regions in the fused stereo images have a coarser resolution. ### Claim 16 (depends on 14) The system of claim 14, wherein the higher resolution for the regions corresponding to the one or more objects enables more accurate estimation of disparity. ### Claim 17 (depends on 16) The system of claim 16, wherein the more accurate estimation of disparity facilitate enhanced estimation of the depth information for each of the one or more objects. ### Claim 18 (depends on 13) The system of claim 13, wherein the depth information determined with respect to each of the one or more objects is used for at least one of: determining an avoidance strategy based on the depth information; and generating a collision warning signal in accordance with the avoidance strategy.