# US 10,872,459 B2 · Scene recognition using volumetric substitution of real world objects > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,459 B2 - Document ID: US10872459B2 - Status: Granted - Kind: B2 - Application number: 16/267,582 - Filed: 2019-02-05 - Published: 2020-08-06 - Granted: 2020-12-22 - Earliest claimed: 2019-02-05 - Assignees: X Development LLC - Applicants: X Development LLC - Inventors: Nicholas John Foster; Matthew Sibigtroth - Web page: https://patentgrep.com/patent/US10872459B2 ## CPC - G06T 7/75: {involving models} - G06T 15/08: Volume rendering - G01C 21/1656: {with passive imaging devices, e.g. cameras} - G05D 1/0246: {using a video camera in combination with image processing means} - G05D 1/0276: {using signals provided by a source external to the vehicle (involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058)} - G06T 7/50: Depth or shape recovery - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 20/20: in augmented reality scenes - G06V 20/64: Three-dimensional [3D] objects - G05B 19/042: using digital processors (G05B19/05 takes precedence) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract Techniques are provided to estimate of location or position of objects that are depicted in an image of a scene. Some implementations include obtaining an image of a scene; identifying an object within the image of the scene; obtaining a three-dimensional model that corresponds to the object that was identified within the image of the scene, the three-dimensional model being obtained from the database of three-dimensional models; determining, based on data from the three-dimensional model, an estimated depth of the object within the scene; generating or updating a three-dimensional representation of the scene based at least on the estimated depth of the object within the scene; and providing the three-dimensional representation of the scene, including at least a portion of the three-dimensional representation of the scene that was generated or updated based on the three-dimensional model of the object, to the scene analyzer. ## Claims ### Claim 1 A computer-implemented method comprising: obtaining, by one or more sensors of a control system that includes (i) the one or more sensors, (ii) a three-dimensional scene generator, (iii) a database of three-dimensional models, (iv) an object location estimator, and (v) a scene analyzer, a two-dimensional (2D) image of a scene through which a robot is to navigate, wherein the 2D image that lacks depth information about objects within the scene; identifying, by the control system and based on matching (i) a portion of the 2D image of the scene that lacks depth information about objects within the scene with (ii) a stored, 2D image, an object within the image of the scene that is pre-associated with the stored, 2D image that matches the portion of the 2D image of the scene; after the object within the image of the scene has been identified based on matching (i) the portion of the 2D image of the scene that lacks depth information with (ii) the stored, 2D image, obtaining, by the control system, a three-dimensional model that corresponds to the object, the three-dimensional model being obtained from the database of three-dimensional models; determining, by the object location estimator and based on data from the three-dimensional model, an estimated distance between the one or more sensors and the object; generating or updating, by the three-dimensional scene generator, a three-dimensional representation of the scene through which the robot is to navigate based at least on (i) the estimated distance between the one or more sensors and the object, and (ii) dimensions of the object that are indicated by the three-dimensional model; providing, by the control system, the three-dimensional representation of the scene to the scene analyzer; and generating, for the robot to navigate and based at least on the three-dimensional representation of the scene, a navigation path through the scene that avoids contact between the robot and the object. ### Claim 2 (depends on 1) The method of claim 1, wherein the estimated distance is determined by using calibration information of the one or more sensors, as well as specifications of the object as indicated in the three-dimensional model, the calibration information being used to correlate size and position of the object within a field of view of the one or more sensors. ### Claim 3 (depends on 1) The method of claim 1, wherein, in the image, the object is at least partially obscured by another object. ### Claim 4 (depends on 1) The method of claim 1, wherein, the object is oblique to the one or more sensors. ### Claim 5 (depends on 1) The method of claim 1, wherein the data from the three-dimensional model includes an orientation of the object as depicted in the image. ### Claim 6 (depends on 1) The method of claim 1, wherein the data from the three-dimensional model indicates metadata of the object, including a dimension of a portion of the object that is not discernable from the image. ### Claim 7 (depends on 1) The method of claim 1, wherein the object is a first object, the three-dimensional model is a first three-dimensional model, and the method further comprises identifying, by the control system, a second object within the image of the scene; obtaining, by the control system, a second three-dimensional model that corresponds to the second object, the second three-dimensional model being obtained from the database of three-dimensional models; determining, by the object location estimator and based on data from the second three-dimensional model, an estimated depth of the second object within the scene; and determining a distance between the first object and the second object based on the estimated depth of the first object within the scene and the estimated depth of the second object within the scene, wherein the three-dimensional representation of the scene is generated or updated based on the distance between the first object and the second object. ### Claim 8 (depends on 1) The method of claim 1, further comprising identifying, in the image, a planar surface that is in contact with the object; and determining a depth of the planar surface based on the estimated depth of the object. ### Claim 9 (depends on 1) The method of claim 1, further comprising: determining that the image captures a confined area; and determining one or more dimensions of the confined area based on the estimated depth of the object. ### Claim 10 A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising: obtaining, by one or more sensors, a two-dimensional (2D) image of a scene through which a robot is to navigate, wherein the 2D image that lacks depth information about objects within the scene; identifying, based on matching (i) a portion of the 2D image of the scene that lacks depth information about objects within the scene with (ii) a stored, 2D image, an object within the image of the scene that is pre-associated with the stored, 2D image that matches the portion of the 2D image of the scene; after the object within the image of the scene has been identified based on matching (i) the portion of the 2D image of the scene that lacks depth information with (ii) the stored, 2D image, obtaining a three-dimensional model that corresponds to the object, the three-dimensional model being obtained from the database of three-dimensional models; determining, based on data from the three-dimensional model, an estimated distance between the one or more sensors and the object; generating or updating a three-dimensional representation of the scene through which the robot is to navigate based at least on (i) the estimated distance between the one or more sensors and the object, and (ii) dimensions of the object that are indicated by the three-dimensional model; providing the three-dimensional representation of the scene to a scene analyzer; and generating, for the robot to navigate and based at least on the three-dimensional representation of the scene, a navigation path through the scene that avoids contact between the robot and the object. ### Claim 11 (depends on 10) The non-transitory computer-readable storage medium of claim 10, wherein the estimated depth of the object is determined by using calibration information of the one or more sensors, as well as specifications of the object as indicated in the three-dimensional model, the calibration information being used to correlate size and position of the object within a field of view of the one or more sensors. ### Claim 12 (depends on 10) The non-transitory computer-readable storage medium of claim 10, wherein, in the image, the object is at least partially obscured by another object. ### Claim 13 (depends on 10) The non-transitory computer-readable storage medium of claim 10, wherein, the object is oblique to the one or more sensors. ### Claim 14 (depends on 10) The non-transitory computer-readable storage medium of claim 10, wherein the data from the three-dimensional model indicates metadata of the object, including a dimension of a portion of the object that is not discernable from the image. ### Claim 15 A system, comprising: a computing device; and a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations, the operations comprising: obtaining, by one or more sensors, a two-dimensional (2D) image of a scene through which a robot is to navigate, wherein the 2D image that lacks depth information about objects within the scene; identifying, based on matching (i) a portion of the 2D image of the scene that lacks depth information about objects within the scene with (ii) a stored, 2D image, an object within the image of the scene that is pre-associated with the stored, 2D image that matches the portion of the 2D image of the scene; after the object within the image of the scene has been identified based on matching (i) the portion of the 2D image of the scene that lacks depth information with (ii) the stored, 2D image, obtaining a three-dimensional model that corresponds to the object, the three-dimensional model being obtained from the database of three-dimensional models; determining, based on data from the three-dimensional model, an estimated distance between the one or more sensors and the object; generating or updating a three-dimensional representation of the scene through which the robot is to navigate based at least on (i) the estimated distance between the one or more sensors and the object, and (ii) dimensions of the object that are indicated by the three-dimensional model; providing the three-dimensional representation of the scene to a scene analyzer; and generating, for the robot to navigate and based at least on the three-dimensional representation of the scene, a navigation path through the scene that avoids contact between the robot and the object. ### Claim 16 (depends on 15) The system of claim 15, wherein the estimated depth of the object is determined by using calibration information of the one or more sensors, as well as specifications of the object as indicated in the three-dimensional model, the calibration information being used to correlate size and position of the object within a field of view of the one or more sensors. ### Claim 17 (depends on 15) The system of claim 15, wherein, in the image, the object is at least partially obscured by another object. ### Claim 18 (depends on 15) The system of claim 15, wherein, the object is oblique to the one or more sensors. ### Claim 19 (depends on 15) The system of claim 15, wherein the data from the three-dimensional model indicates metadata of the object, including a dimension of a portion of the object that is not discernable from the image.