# US 10,484,659 B2 · Large-scale environmental mapping in real-time by a robotic system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,484,659 B2 - Document ID: US10484659B2 - Status: Granted - Kind: B2 - Application number: 15/693,092 - Filed: 2017-08-31 - Published: 2019-02-28 - Granted: 2019-11-19 - Earliest claimed: 2017-08-31 - Assignees: Disney Enterprises, Inc.; ETH Zurich - Applicants: Disney Enterprises, Inc. - Inventors: Paul Beardsley; Anurag Sai Vempati; Juan Nieto; Igor Gilitschenski - Web page: https://patentgrep.com/patent/US10484659B2 ## CPC - H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence) - G01C 15/002: {Active optical surveying means (optical plumbing G01C15/105)} - G01C 21/20: Instruments for performing navigational calculations (G01C21/24, G01C21/26 take precedence) - G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00) - G05D 1/0011: {associated with a remote control arrangement} - G05D 1/0202: {specially adapted to aircraft} - G05D 1/0206: {specially adapted to water vehicles} - G05D 1/0248: {in combination with a laser (lasers per se H01S)} - G05D 1/027: {comprising intertial navigation means, e.g. azimuth detector (inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165)} - G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)} - G06T 15/08: Volume rendering - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 17/05: Geographic models - H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor - H04N 13/257: Colour aspects - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - H04N 13/296: Synchronisation thereof; Control thereof - G06T 15/005: {General purpose rendering architectures} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract According to one implementation, a robotic system for performing large-scale environmental mapping in real-time includes a mobile reconnaissance unit having a color sensor, a depth sensor, and a graphics processing unit (GPU) with a GPU memory, and a navigation unit communicatively coupled to the mobile reconnaissance unit and having a central processing unit (CPU) with a CPU memory. The robotic system begins a three-dimensional (3D) scan of an environment of the mobile reconnaissance unit using the color sensor and the depth sensor, and generates mapping data for populating a volumetric representation of the environment. The robotic system continues the 3D scan of the environment using the color sensor and the depth sensor, updates the mapping data, and partitions the volumetric representation between the GPU memory and the CPU memory based on a memory allocation criteria. ## Claims ### Claim 1 A robotic system comprising: a plurality of mobile reconnaissance units including a mobile reconnaissance unit having a color sensor, a depth sensor, and a graphics processing unit (GPU) with a GPU memory; a navigation unit communicatively coupled to the plurality of mobile reconnaissance units, the navigation unit having a central processing unit (CPU) with a CPU memory, wherein the navigation unit is a remote base station for all of the plurality of mobile reconnaissance units; the robotic system configured to: begin a three-dimensional (3D) scan of an environment of the mobile reconnaissance unit using the color sensor and the depth sensor; generate a mapping data for populating a volumetric representation of the environment produced using the GPU; continue the 3D scan of the environment using the color sensor and the depth sensor; update the mapping data based on the continued 3D scan; and partition the volumetric representation between the GPU memory and the CPU memory based on a memory allocation criteria; wherein the volumetric representation of the environment is stored partially in the GPU memory of the mobile reconnaissance unit and partially in the CPU memory of the navigation unit. ### Claim 2 (depends on 1) The robotic system of claim 1, wherein the navigation unit further comprises a display screen, and wherein the robotic system is further configured to display a visual representation of the environment of the mobile reconnaissance unit on the display screen. ### Claim 3 (depends on 1) The robotic system of claim 1, wherein the navigation unit is integrated with the mobile reconnaissance unit. ### Claim 4 (depends on 1) The robotic system of claim 1, wherein the mobile reconnaissance unit comprises an aerial vehicle. ### Claim 5 (depends on 1) The robotic system of claim 1, wherein the mobile reconnaissance unit comprises one of a wheeled vehicle, a legged vehicle, and a continuous track propulsion vehicle. ### Claim 6 (depends on 1) The robotic system of claim 1, wherein the mobile reconnaissance unit comprises a submersible vehicle. ### Claim 7 A robotic system comprising: a mobile reconnaissance unit having a color sensor, a depth sensor, and a graphics processing unit (GPU) with a GPU memory; a navigation unit communicatively coupled to the mobile reconnaissance unit, the navigation unit having a central processing unit (CPU) with a CPU memory; the robotic system configured to: begin a three-dimensional (3D) scan of an environment of the mobile reconnaissance unit using the color sensor and the depth sensor; generate a mapping data for populating a volumetric representation of the environment produced using the GPU; continue the 3D scan of the environment using the color sensor and the depth sensor; update the mapping data based on the continued 3D scan; and partition the volumetric representation between the GPU memory and the CPU memory based on a memory allocation criteria; wherein the volumetric representation of the environment is stored partially in the GPU memory of the mobile reconnaissance unit and partially in the CPU memory of the navigation unit, wherein the mobile reconnaissance unit further comprises an inertial sensor, and wherein the robotic system is further configured to: detect an abrupt movement of the mobile reconnaissance unit during the second 3D scan, using the inertial sensor; generate a perturbation data corresponding to the abrupt movement; and correct the mapping data to compensate for the abrupt movement using the perturbation data. ### Claim 8 (depends on 7) The robotic system of claim 7, wherein the navigation unit is a remote base station for controlling the mobile reconnaissance unit. ### Claim 9 (depends on 7) The robotic system of claim 7, wherein the mobile reconnaissance unit is one of a plurality of mobile reconnaissance units communicatively coupled to the navigation unit, and wherein the navigation unit is a remote base station for all of the plurality of mobile reconnaissance units. ### Claim 10 A robotic system comprising: a mobile reconnaissance unit having a color sensor, a depth sensor, and a graphics processing unit (GPU) with a GPU memory; a navigation unit communicatively coupled to the mobile reconnaissance unit, the navigation unit having a central processing unit (CPU) with a CPU memory; the robotic system configured to: begin a three-dimensional (3D) scan of an environment of the mobile reconnaissance unit using the color sensor and the depth sensor; generate a mapping data for populating a volumetric representation of the environment produced using the GPU; continue the 3D scan of the environment using the color sensor and the depth sensor; update the mapping data based on the continued 3D scan; and partition the volumetric representation between the GPU memory and the CPU memory based on a memory allocation criteria; wherein the volumetric representation of the environment is stored partially in the GPU memory of the mobile reconnaissance unit and partially in the CPU memory of the navigation unit, and wherein the color sensor is an RGB camera, and the RGB camera and the depth sensor form an integrated RGB-D sensor of the mobile reconnaissance unit. ### Claim 11 A method for use by a robotic system comprising a plurality of mobile reconnaissance units including a mobile reconnaissance unit having a color sensor, a depth sensor, and a graphics processing unit (GPU) with a GPU memory, and further including a navigation unit communicatively coupled to the plurality of mobile reconnaissance units and having a central processing unit (CPU) with a CPU memory, the navigation unit being a remote base station for all of the plurality of mobile reconnaissance units, the method comprising: beginning a three-dimensional (3D) scan of an environment of the mobile reconnaissance unit using the color sensor and the depth sensor; generating a mapping data for populating a volumetric representation of the environment produced using the GPU; continuing the 3D scan of the environment using the color sensor and the depth sensor; updating the mapping data based on the continued 3D scan; and partitioning the volumetric representation between the GPU memory and the CPU memory based on a memory allocation criteria; wherein the volumetric representation of the environment is stored partially in the GPU memory of the mobile reconnaissance unit and partially in the CPU memory of the navigation unit. ### Claim 12 (depends on 11) The method of claim 11, wherein the navigation unit further includes a display screen, and wherein the method further comprises displaying a visual representation of the environment of the mobile reconnaissance unit on the display screen. ### Claim 13 (depends on 11) The method of claim 11, wherein the navigation unit is integrated with the mobile reconnaissance unit. ### Claim 14 (depends on 11) The method of claim 11, wherein the mobile reconnaissance unit comprises an aerial vehicle. ### Claim 15 (depends on 11) The method of claim 11, wherein the mobile reconnaissance unit comprises one of a wheeled vehicle, a legged vehicle, and a continuous track propulsion vehicle. ### Claim 16 (depends on 11) The method of claim 11, wherein the mobile reconnaissance unit comprises a submersible vehicle. ### Claim 17 A method for use by a robotic system including a mobile reconnaissance unit having a color sensor, a depth sensor, and a graphics processing unit (GPU) with a GPU memory, and further including a navigation unit communicatively coupled to the mobile reconnaissance unit and having a central processing unit (CPU) with a CPU memory, the method comprising: beginning a three-dimensional (3D) scan of an environment of the mobile reconnaissance unit using the color sensor and the depth sensor; generating a mapping data for populating a volumetric representation of the environment produced using the GPU; continuing the 3D scan of the environment using the color sensor and the depth sensor; updating the mapping data based on the continued 3D scan; and partitioning the volumetric representation between the GPU memory and the CPU memory based on a memory allocation criteria; wherein the volumetric representation of the environment is stored partially in the GPU memory of the mobile reconnaissance unit and partially in the CPU memory of the navigation unit, wherein the mobile reconnaissance unit further includes an inertial sensor, and wherein the method further comprises: detecting a abrupt movement of the mobile reconnaissance unit during the second 3D scan, using the inertial sensor; generating a perturbation data corresponding to the abrupt movement; and correcting the mapping data to compensate for the abrupt movement using the perturbation data. ### Claim 18 (depends on 17) The method of claim 17, wherein the navigation unit is a remote base station for controlling the mobile reconnaissance unit. ### Claim 19 (depends on 17) The method of claim 17, wherein the mobile reconnaissance unit is one of a plurality of mobile reconnaissance units communicatively coupled to the navigation unit, and wherein the navigation unit is a remote base station for all of the plurality of mobile reconnaissance units. ### Claim 20 A method for use by a robotic system including a mobile reconnaissance unit having a color sensor, a depth sensor, and a graphics processing unit (GPU) with a GPU memory, and further including a navigation unit communicatively coupled to the mobile reconnaissance unit and having a central processing unit (CPU) with a CPU memory, the method comprising: beginning a three-dimensional (3D) scan of an environment of the mobile reconnaissance unit using the color sensor and the depth sensor; generating a mapping data for populating a volumetric representation of the environment produced using the GPU; continuing the 3D scan of the environment using the color sensor and the depth sensor; updating the mapping data based on the continued 3D scan; and partitioning the volumetric representation between the GPU memory and the CPU memory based on a memory allocation criteria; wherein the volumetric representation of the environment is stored partially in the GPU memory of the mobile reconnaissance unit and partially in the CPU memory of the navigation unit, and wherein the color sensor is an RGB camera, and the RGB camera and the depth sensor form an integrated RGB-D sensor of the mobile reconnaissance unit.