# US 10,122,995 B2 · Systems and methods for generating and displaying a 3D model of items in a warehouse > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,122,995 B2 - Document ID: US10122995B2 - Status: Granted - Kind: B2 - Application number: 15/273,465 - Filed: 2016-09-22 - Published: 2018-03-22 - Granted: 2018-11-06 - Earliest claimed: 2016-09-22 - Assignees: X Development LLC - Applicants: X Development LLC - Inventors: Ethan Rublee; John Zevenbergen - Web page: https://patentgrep.com/patent/US10122995B2 ## CPC - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06F 18/22: Matching criteria, e.g. proximity measures - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 19/003: {Navigation within 3D models or images} - G06V 20/20: in augmented reality scenes - H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - G06T 2200/08: involving all processing steps from image acquisition to 3D model generation - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2215/16: Using real world measurements to influence rendering - G06T 2219/2021: Shape modification - Y10S 901/01: Mobile robot - Y10S 901/47: Optical - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS ## Abstract An example method involves receiving, from at least one camera located in an environment, a plurality of images captured during a first time interval. The method also involves selecting one or more of the plurality of images having a movable platform supporting one or more objects. The method further involves generating a three-dimensional model of the movable platform supporting the one or more objects. The method yet further involves updating the three-dimensional model based on one or more images captured during a second time interval. The method still further involves presenting the three-dimensional model via a display of a user interface, and providing an option to view a history of the three-dimensional model such that the three-dimensional model remains in a fixed position on the display during a viewing of the history. ## Claims ### Claim 1 A method comprising: receiving, from a camera located in an environment, a plurality of images captured during a first time interval; selecting an image from the plurality of images, wherein the selected image includes a movable platform supporting an object; generating, based on the selected image, a three-dimensional model of the movable platform supporting the object; updating the three-dimensional model based on an additional image captured during a second time interval, wherein the second time interval begins later than the first time interval, and wherein (i) a first state of the three-dimensional model corresponding to the first time interval and (ii) a second state of the three-dimensional model corresponding to the second time interval are stored in the three-dimensional model to represent a history thereof; presenting the three-dimensional model using a user interface provided on a display; and providing an option, by the user interface, to view the history of the three-dimensional model, the history including the first time interval and the second time interval, such that, when the three-dimensional model changes between the first state and the second state during a viewing of the history, the three-dimensional model remains in a fixed position on the display. ### Claim 2 (depends on 1) The method of claim 1, wherein the selected image depicts the movable platform supporting the object when the movable platform is being transported within the environment. ### Claim 3 (depends on 1) The method of claim 1, wherein selecting the image from the plurality of images comprises: determining a target visual identifier corresponding to the movable platform; determining an image of the plurality of images that depicts the target visual identifier; and selecting the determined image. ### Claim 4 (depends on 1) The method of claim 1, wherein selecting the image from the plurality of images comprises: determining a target non-visual identifier corresponding to the movable platform; determining, based on the target non-visual identifier, an image of the plurality of images; and selecting the determined image. ### Claim 5 (depends on 1) The method of claim 1, further comprising determining an expected location of the movable platform, wherein selecting the image from the plurality of images comprises selecting the image based on the expected location of the movable platform. ### Claim 6 (depends on 5) The method of claim 5, wherein the expected location of the movable platform is determined based on a task assigned to a robotic device transporting the movable platform. ### Claim 7 (depends on 1) The method of claim 1, wherein the image captured during the first time interval and the additional image captured during the second time interval are captured by cameras that have non-overlapping fields of view. ### Claim 8 (depends on 1) The method of claim 1, wherein two images are selected from the plurality of images such that the selected images include two sides of the movable platform. ### Claim 9 (depends on 1) The method of claim 1, wherein updating the three-dimensional model based on the additional image captured during the second time interval comprises: receiving, from the camera, the additional image captured during the second time interval; determining that the received additional image includes a first side of the movable platform, corresponding to a first side of the three-dimensional model; and updating the first side of the three-dimensional model. ### Claim 10 (depends on 1) The method of claim 1, further comprising: determining, for a particular point in time, an expected shape of the movable platform supporting the object; determining, based on the three-dimensional model, for the particular point in time, an actual shape of the movable platform supporting the object; determining that the expected shape and the actual shape lack a threshold extent of similarity; and responsive to determining that the actual shape and the expected shape lack a threshold extent of similarity, presenting, by the user interface, an indication of the particular point in time. ### Claim 11 (depends on 1) The method of claim 1, further comprising: updating the three-dimensional model based on a third image captured during a third time interval that begins later than the second time interval, wherein a third state of the three-dimensional model corresponding to the third time interval is stored in the three-dimensional model to represent the history thereof, wherein providing the option to view the history of the three-dimensional model comprises presenting the three-dimensional model on the display such that the display scrolls through showing the first, second, and third states of the three-dimensional model during the first, second, and third time intervals, respectively. ### Claim 12 (depends on 1) The method of claim 1, wherein providing the option to view the history of the three-dimensional model comprises presenting the three-dimensional model on the display, wherein the three-dimensional model includes an item in a volume surrounding the movable platform, such that, when the three-dimensional model changes between the first state and the second state during the viewing of the history, the item is displayed moving relative to the movable platform and the movable platform is displayed in the fixed position on the display. ### Claim 13 (depends on 1) The method of claim 1, further comprising: updating the three-dimensional model at fixed time intervals; and providing, on the display, an indication that the three-dimensional model has changed by more than a threshold amount between adjacent time intervals. ### Claim 14 A system comprising: a movable platform configured to support an object; a camera located in an environment; and a computing system, wherein the computing system is configured to: receive, from the camera, a plurality of images captured during a first time interval; select an image from the plurality of images, wherein the selected image includes the movable platform; generate, based on the selected image, a three-dimensional model of the movable platform supporting the object; update the three-dimensional model based on an additional image captured during a second time interval, wherein the second time interval begins later than the first time interval, and wherein (i) a first state of the three-dimensional model corresponding to the first time interval and (ii) a second state of the three-dimensional model corresponding to the second time interval are stored in the three-dimensional model to represent a history thereof; present the three-dimensional model using a user interface provided on a display of the computing system; and provide an option, by the user interface, to view the history of the three-dimensional model, the history including the first time interval and the second time interval, such that, when the three-dimensional model changes between the first state and the second state during a viewing of the history, the three-dimensional model remains in a fixed position on the display during a viewing of the history. ### Claim 15 (depends on 14) The system of claim 14, wherein the computing system is further configured to determine an expected location of the movable platform, wherein the computing system is configured to select the image from the plurality of images based on the expected location of the movable platform. ### Claim 16 (depends on 15) The system of claim 15, further comprising: a robotic device configured to transport the movable platform within the environment, wherein the computing system is further configured to transmit an instruction to the robotic device to transport the movable platform, and wherein the expected location of the movable platform is determined based on the instruction transmitted to the robotic device. ### Claim 17 (depends on 14) The system of claim 14, wherein the computing system is further configured to: determine, for a particular point in time, an expected shape of the movable platform supporting the object; determine, based on the three-dimensional model, for the particular point in time, an actual shape of the movable platform supporting the object; determine that the expected shape and the actual shape lack a threshold extent of similarity; and responsive to determining that the actual shape and the expected shape lack a threshold extent of similarity, present, via the user interface, an indication of the particular point in time. ### Claim 18 (depends on 14) The system of claim 14, wherein the computing system is further configured to: update the three-dimensional model based on a third image captured during a third time interval that begins later than the second time interval, wherein a third state of the three-dimensional model corresponding to the third time interval is stored in the three-dimensional model to represent the history thereof, wherein providing the option to view the history of the three-dimensional model comprises presenting the three-dimensional model on the display such that the display scrolls through showing the first, second, and third states of the three-dimensional model during the first, second, and third time intervals respectively. ### Claim 19 (depends on 14) The system of claim 14, wherein the three-dimensional model includes an item in a volume surrounding the movable platform, and wherein the computing system is configured to provide the option to view the history of the three-dimensional model by presenting the three-dimensional model on the display such that, when the three-dimensional model changes between the first state and the second state during the viewing of the history, the item is displayed moving relative to the movable platform and the movable platform is displayed in the fixed position on the display. ### Claim 20 (depends on 14) The system of claim 14, wherein the computing system is further configured to: update the three-dimensional model at fixed time intervals; and provide, on the display, an indication that the three-dimensional model has changed by more than a threshold amount between adjacent time intervals. ### Claim 21 A non-transitory computer-readable medium having stored thereon program instructions that when executed cause a computing device to perform a set of acts comprising: receiving, from a camera located in an environment, a plurality of images captured during a first time interval; selecting an image from the plurality of images, wherein the selected image includes a movable platform supporting an object; generating, based on the selected image, a three-dimensional model of the movable platform supporting the object; updating the three-dimensional model based on an additional image captured during a second time interval, wherein the second time interval begins later than the first time interval, and wherein (i) a first state of the three-dimensional model corresponding to the first time interval and (ii) a second state of the three-dimensional model corresponding to the second time interval are stored in the three-dimensional model to represent a history thereof; presenting the three-dimensional model using a user interface provided on a display of the computing device; and providing an option, by the user interface, to view the history of the three-dimensional model, the history including the first time interval and the second time interval, such that, when the three-dimensional model changes between the first state and the second state during a viewing of the history, the three-dimensional model remains in a fixed position on the display.