# US 12,759,153 B2 · Systems, methods, and computer program products for implementing object permanence in a simulated environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,759,153 B2 - Document ID: US12759153B2 - Status: Granted - Kind: B2 - Application number: 18/225,790 - Filed: 2023-07-25 - Published: 2024-02-01 - Granted: 2026-10-13 - Earliest claimed: 2022-07-27 - Assignees: Sanctuary Cognitive Systems Corporation - Applicants: Sanctuary Cognitive Systems Corporation - Inventors: Jonathan McCully Moore; Emma Amelia Fajeau - Web page: https://patentgrep.com/patent/US12759153B2 ## CPC - B25J 9/1671: {characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems} - B25J 9/1664: {characterised by motion, path, trajectory planning} - G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)} - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 17/05: Geographic models - G06T 2215/16: Using real world measurements to influence rendering - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems, methods, and computer program products for managing simulated environments are described. A simulated environment is accessed which represents a physical environment, and representations of objects are included, maintained, or removed in the simulated environment based on whether objects are represented in image data of the physical environment, and based on whether objects are occluded by other objects in the image data of the physical environment. Future occlusion can also be predicted based on motion paths of objects. ## Claims ### Claim 1 A computer program product comprising a non-transitory processor-readable storage medium storing processor-executable instructions and/or data that, when executed by at least one processor of a robot system comprising a robot body in a physical environment, cause the robot system to implement object permanence in a simulated environment that comprises an environment model representing the physical environment in which the robot body operates, wherein the robot system uses the simulated environment to operate within the physical environment, and wherein the processor executable instructions and/or data that, when executed by the at least one processor, cause the robot system to implement object permanence in the simulated environment, cause the robot system to: access, by the at least one processor, the simulated environment; capture, by at least one image sensor carried by the robot body, first image data representing the physical environment at a first time, the first image data further representing a first object at a first position; include, in the environment model, a first representation of the first object based on the first image data; capture, by the at least one image sensor, second image data representing the physical environment at a second time after the first time; determine, by the at least one processor, whether the first object is represented in the second image data; if the first object is represented in the second image data, maintain representation of the first object in the environment model; if the first object is not represented in the second image data: determine, by the at least one processor, whether the first object is occluded from the second image data by at least one member of the robot body at the second time; if the first object is occluded from the second image data by the at least one member of the robot body at the second time: maintain representation of the first object in the environment model; and if the first object is not occluded from the second image data by the at least one member of the robot body at the second time: remove representation of the first object from the environment model; wherein the processor-executable instructions and/or data which cause the robot system to determine whether the first object is occluded from the second image data by the at least one member of the robot body at the second time cause the robot system to: identify, at a third time before the second time, a motion path of the at least one member of the robot body; determine, by the at least one processor before the second time, a predicted position of the at least one member of the robot body at the second time based on the identified motion path; and determine whether the predicted position of the at least one member of the robot body is between the first position of the first object at the first time and a position of the image sensor. ### Claim 2 (depends on 1) The computer program product of claim 1 wherein the processor-executable instructions and/or data which cause the robot system to include, in the environment model, a first representation of the first object based on the first image data cause the robot system to: populate, by the at least one processor, the environment model with the first representation of the first object based on the first image data. ### Claim 3 (depends on 1) The computer program product of claim 1 wherein the processor-executable instructions and/or data which cause the robot system to include, in the environment model, a first representation of the first object based on the first image data cause the robot system to: determine, by the at least one processor, whether the environment model includes the first representation of the first object; if the environment model includes the first representation of the first object, preserve the first representation of the first object as included in the environment model; and if the environment model does not include the first representation of the first object, populate the environment model with the first representation of the first object based on the first image data. ### Claim 4 (depends on 1) The computer program product of claim 1 wherein the processor-executable instructions and/or data which cause the robot system to determine whether the first object is occluded from the second image data by the at least one member of the robot body at the second time cause the robot system to: identify a second position of the at least one member of the robot body at the second time; and determine, by the at least one processor, whether the second position of the at least one member of the robot body is between the first position of the first object at the first time and a position of the image sensor at the second time. ### Claim 5 (depends on 1) The computer program product of claim 1 wherein the at least one member of the robot body includes at least one end-effector of the robot body. ### Claim 6 (depends on 5) The computer program product of claim 5 wherein the at least one end-effector of the robot body includes at least one hand member of the robot body. ### Claim 7 (depends on 1) The computer program product of claim 1 wherein the processor-executable instructions and/or data which cause the robot system to identify the motion path of the at least one member of the robot body cause the robot system to: identify the motion path of the at least one member of the robot body by simulating, by the at least one processor, a trajectory of the at least one member of the robot body based on speed and direction of travel of the at least one member of the robot body. ### Claim 8 (depends on 7) The computer program product of claim 7, wherein the robot body further carries at least one haptic sensor, and the speed and direction of travel of the at least one member of the robot body are identified based on haptic data captured by the at least one haptic sensor. ### Claim 9 (depends on 7) The computer program product of claim 7 wherein the speed and direction of travel of the at least one member of the robot body are identified based on image data from the at least one image sensor. ### Claim 10 (depends on 1) The computer program product of claim 1 wherein the processor-executable instructions and/or data which cause the robot system to identify the motion path of the at least one member of the robot body cause the robot system to: identify the motion path of the at least one member of the robot body based on at least one motion instruction upon which movement of the at least one member of the robot body is based. ### Claim 11 (depends on 1) The computer program product of claim 1, wherein the at least one processor is carried by the robot body. ### Claim 12 (depends on 1) The computer program product of claim 1, wherein the at least one non-transitory processor-readable storage medium is carried by the robot body, and the simulated environment including the environment model is stored on the at least one non-transitory processor-readable storage medium carried by the robot body. ### Claim 13 (depends on 1) The computer program product of claim 1, wherein: the robot system further includes at least one device remote from the robot body; the at least one non-transitory processor-readable storage medium includes at least one first non-transitory processor-readable storage medium at the remote device; and the simulated environment including the environment model is stored on the at least one first non-transitory processor-readable storage medium at the remote device. ### Claim 14 (depends on 1) The computer program product of claim 1, wherein the processor-executable instructions and/or data which cause the robot system to determine, by the at least one processor, whether the first object is represented in the second image data cause the robot system to: determine that the first object is not represented in the second image data.