# US 12,183,023 B2 · Methods and apparatus for depth estimation using stereo cameras in a vehicle system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,023 B2 - Document ID: US12183023B2 - Status: Granted - Kind: B2 - Application number: 18/132,311 - Filed: 2023-04-07 - Published: 2023-12-07 - Granted: 2024-12-31 - Earliest claimed: 2022-06-02 - Assignees: PlusAI, Inc. - Applicants: PlusAI, Inc. - Inventors: Mianwei Zhou - Web page: https://patentgrep.com/patent/US12183023B2 ## CPC - G06T 7/593: from stereo images - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/759: {Region-based matching} - G06V 10/764: using classification, e.g. of video objects - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - B60W 60/0015: {specially adapted for safety} - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2554/20: Static objects - B60W 2554/402: Type - G06T 2207/10012: Stereo images - G06T 2207/20081: Training; Learning - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06V 2201/08: Detecting or categorising vehicles - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT ## Abstract A method comprises: receiving, at a processor, a first image from a first camera from a stereo camera pair and a second image from a second camera from the stereo camera pair. The method also includes determining, at the processor using a machine learning model, a first set of objects in the first image. The processor determines an object type. The processor identifies a second set of objects in the second image associated with the first plurality of objects. The method also includes calculating, at the processor, a set of disparity values between the first image and the second image based on (1) an object from the first set of objects, (2) an object from the second set of objects and associated with the object from the first set of objects, and (3) an object type of the object from the first set of objects. ## Claims ### Claim 1 A computer-implemented method comprising: generating, by a computing system, a first bounding box to represent a first region of interest in a first image; searching, by the computing system, for a plurality of second bounding boxes in a second image that represent second regions of interest in the second image that correspond to the first region of interest in the first image; determining, by the computing system, a second bounding box of the plurality of second bounding boxes in the second image that matches the first bounding box in the first image; and providing, by the computing system, depth information for an object in the first region of interest based on disparity values associated with the second bounding box that matches the first bounding box, the disparity values between a plurality of second sets of pixels sampled from the second regions of interest in the second image and a first set of pixels sampled from the first region of interest in the first image. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, the first region of interest based on semantic segmentation of the first image; and determining, by the computing system, the second regions of interest based on semantic segmentation of the second image. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, semantic information associated with the first region of interest; and selecting, by the computing system, the first set of pixels sampled from the first region of interest and the second sets of pixels sampled from the second regions of interest based on the semantic information. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein, the first bounding box includes at least one of: a portion of a road, a portion of a shoulder area, a portion of a ground, a portion of a wall, a portion of a sky, and a portion of a background, and wherein the plurality of second bounding boxes include at least one of: portions of the road, portions of the shoulder area, portions of the ground, portions of the wall, portions of the sky, and portions of the background. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, further comprising: identifying, by the computing system, a set of objects in the first image; selecting, by the computing system, an object of the set of objects; and determining, by the computing system, a size, a shape, and a location of the first region of interest in the image based on the object of the set of objects. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the plurality of second bounding boxes include a plurality of objects in the second image, and wherein the determining the second bounding box of the plurality of bounding boxes is based on a match of an object of the plurality of objects in the second image with an object included in the first bounding box. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, further comprising: determining, by the computing system, an object type of an object in the first image; and determining, by the computing system, a computational power at which to calculate the disparity values based on the object type. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, further comprising: providing, by the computing system, a signal that represents the depth information to a driving controller. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the first bounding box and the plurality of second bounding boxes are cuboid shapes, horizontal lines, or vertical lines. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the first image is from a first camera of a stereo camera pair, the second image is from a second camera of the stereo camera pair, and the depth information is relative to the stereo camera pair. ### Claim 11 A system comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising: generating a first bounding box to represent a first region of interest in a first image; searching for a plurality of second bounding boxes in a second image that represent second regions of interest in the second image that correspond to the first region of interest in the first image; determining a second bounding box of the plurality of second bounding boxes in the second image that matches the first bounding box in the first image; and providing depth information for an object in the first region of interest based on disparity values associated with the second bounding box that matches the first bounding box, the disparity values between a plurality of second sets of pixels sampled from the second regions of interest in the second image and a first set of pixels sampled from the first region of interest in the first image. ### Claim 12 (depends on 11) The system of claim 11, the operations further comprising: determining the first region of interest based on semantic segmentation of the first image; and determining the second regions of interest based on semantic segmentation of the second image. ### Claim 13 (depends on 11) The system of claim 11, the operations further comprising: determining semantic information associated with the first region of interest; and selecting the first set of pixels sampled from the first region of interest and the second sets of pixels sampled from the second regions of interest based on the semantic information. ### Claim 14 (depends on 11) The system of claim 11, wherein, the first bounding box includes at least one of: a portion of a road, a portion of a shoulder area, a portion of a ground, a portion of a wall, a portion of a sky, and a portion of a background, and wherein the plurality of second bounding boxes includes at least one of: portions of the road, portions of the shoulder area, portions of the ground, portions of the wall, portions of the sky, and portions of the background. ### Claim 15 (depends on 11) The system of claim 11, the operations further comprising: identifying a set of objects in the first image; selecting an object of the set of objects; and determining a size, a shape, and a location of the first region of interest in the image based on the object of the set of objects. ### Claim 16 A non-transitory computer-readable storage medium including instructions that, when executed by at least on processor of a computing system, cause the computing system to perform operations comprising: generating a first bounding box to represent a first region of interest in a first image; searching for a plurality of second bounding boxes in a second image that represent second regions of interest in the second image that correspond to the first region of interest in the first image; determining a second bounding box of the plurality of second bounding boxes in the second image that matches the first bounding box in the first image; and providing depth information for an object in the first region of interest based on disparity values associated with the second bounding box that matches the first bounding box, the disparity values between a plurality of second sets of pixels sampled from the second regions of interest in the second image and a first set of pixels sampled from the first region of interest in the first image. ### Claim 17 (depends on 16) The non-transitory computer-readable storage medium of claim 16, the operations further comprising: determining the first region of interest based on semantic segmentation of the first image; and determining the second regions of interest based on semantic segmentation of the second image. ### Claim 18 (depends on 16) The non-transitory computer-readable storage medium of claim 16, the operations further comprising: determining semantic information associated with the first region of interest; and selecting the first set of pixels sampled from the first region of interest and the second sets of pixels sampled from the second regions of interest based on the semantic information. ### Claim 19 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein, the first bounding box includes at least one of: a portion of a road, a portion of a shoulder area, a portion of a ground, a portion of a wall, a portion of a sky, and a portion of a background, and wherein the plurality of second bounding boxes includes at least one of: portions of the road, portions of the shoulder area, portions of the ground, portions of the wall, portions of the sky, and portions of the background. ### Claim 20 (depends on 16) The non-transitory computer-readable storage medium of claim 16, the operations further comprising: identifying a set of objects in the first image; selecting an object of the set of objects; and determining a size, a shape, and a location of the first region of interest in the image based on the object of the set of objects.