# US 12,217,478 B2 · Utilizing prediction models of an environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,478 B2 - Document ID: US12217478B2 - Status: Granted - Kind: B2 - Application number: 17/807,648 - Filed: 2022-06-17 - Published: 2023-01-12 - Granted: 2025-02-04 - Earliest claimed: 2016-06-09 - Assignees: Shmuel Ur Innovation LTD. - Applicants: Shmuel Ur Innovation Ltd. - Inventors: Shmuel Ur; Vlad Dabija; David Hirshberg - Web page: https://patentgrep.com/patent/US12217478B2 ## CPC - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - B25J 9/1612: {characterised by the hand, wrist, grip control} - B25J 9/1669: {characterised by special application, e.g. multi-arm co-operation, assembly, grasping} - B25J 9/1697: {Vision controlled systems} - G06T 7/12: Edge-based segmentation - G06T 7/75: {involving models} - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G05B 2219/39536: Planning of hand motion, grasping - G05B 2219/40425: Sensing, vision based motion planning - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A method, system and product for utilizing prediction models of an environment. In one embodiment, using a model of an environment and based on a first scene of the environment, a predicted second scene of the environment is predicted. An observed second scene is obtained and compared to the predicted second scene. Based on the comparison between the predicted second scene and the observed second scene, an action is performed. ## Claims ### Claim 1 A system comprising: one or more sensors configured to capture scenes of an environment, the environment comprises one or more physical objects; a processor configured to: obtain a computer-representation of a first scene of the environment from said one or more sensors; predict, using a model of the environment and based on the computer-representation of the first scene of the environment, a predicted second scene of the environment, the predicted second scene comprises predicted pixel values; obtain a computer-representation of a second observed second scene of the environment from said one or more sensors, the observed second scene comprises observed pixel values; determine a difference between the predicted second scene and the observed second scene based on different values of the predicted pixel values compared to the observed pixel values; and create a compressed computer-representation of the observed second scene, wherein the compressed computer-representation includes the determined difference between the predicted second scene and the observed second scene, thereby the computer-representation of the observed second scene can be reconstructed by predicting, using the model and the computer-representation of the first scene, the predicted second scene, and by applying the determined difference between the predicted second scene and the computer representation of the observed second scene. ### Claim 2 (depends on 1) The system of claim 1, wherein the observed second scene occurs later in time than the first scene. ### Claim 3 (depends on 1) The system of claim 1, further comprising a storage device for storing the compressed computer-representation of the observed second scene, wherein the processor is further configured to store the compressed computer-representation of the observed second scene in the storage device, and to uncompress the compressed computer-representation of the observed second scene by applying the determined difference between the predicted second scene and the computer-representation of the observed second scene on the predicted second scene. ### Claim 4 (depends on 1) The system of claim 1, further comprising: a computing device and a communication module configured to transmit the computer-representation of the first scene and the compressed computer-representation of the observed second scene to the computing device; wherein said computing device is configured to uncompress the compressed computer-representation of the observed second scene to obtain the computer-representation of the observed second scene by: predicting, using the model of the environment, a predicted scene of the environment based on the computer-representation of the first scene and by modifying the predicted scene using the determined difference between the predicted second scene and the observed second scene to reconstruct the computer-representation of the observed second scene. ### Claim 5 (depends on 1) The system of claim 1, wherein the computer-representation of the first scene is obtained from a first portion of the one or more sensors; wherein the computer-representation of the observed second scene is obtained from a second portion of the one or more sensors, and wherein the first portion and the second portion of the one or more sensors are different. ### Claim 6 (depends on 5) The system of claim 5, wherein scenes of the environment observed by the first portion of the one or more sensors and scenes of the environment observed by the second portion of one or more sensors demonstrate different views of a same location in the environment. ### Claim 7 (depends on 6) The system of claim 6, wherein the first scene and the observed second scene occur at a same time. ### Claim 8 (depends on 1) The system of claim 1, wherein the one or more sensors include one or more cameras, wherein scenes obtained from the one or more sensors include images of the environment, wherein the compressed computer-representation is a compressed representation of an image of the observed second scene. ### Claim 9 (depends on 1) The system of claim 1, wherein said processor is configured to utilize a machine-learning-based predictor in order to predict the predicted second scene, wherein the machine-learning based predictor is trained based on scenes of the environment that were captured by the one or more sensors. ### Claim 10 (depends on 1) The system of claim 1, wherein the environment is a production environment, wherein the one or more physical objects are articles manufactured in the production environment. ### Claim 11 A method comprising: obtaining a computer-representation of a first scene of an environment from one or more sensors, the environment comprises one or more physical objects, the one or more sensors are configured to capture scenes of the environment; predicting, using a model of the environment and based on the computer-representation of the first scene of the environment, a predicted second scene of the environment, the predicted second scene comprises predicted pixel values; obtaining a computer-representation of an observed second scene of the environment from the one or more sensors, the observed second scene comprises observed pixel values; determining a difference between the predicted second scene and the observed second scene based on different values of the predicted pixel values compared to the observed pixel values; and creating a compressed computer-representation of the observed second scene, wherein the compressed computer-representation includes the determined difference between the predicted second scene and the observed second scene, thereby the computer-representation of the observed second scene can be reconstructed by predicting, using the model and the computer representation of the first scene, the predicted second scene, and by applying the determined difference between the predicted second scene and the computer-representation of the observed second scene. ### Claim 12 (depends on 11) The method of claim 11, wherein the compressed computer-representation of the observed second scene includes information useful for transforming the predicted second scene into the observed second scene. ### Claim 13 (depends on 11) The method of claim 11, wherein a device can reconstruct the computer-representation of the observed second scene by predicting, using the model and based on the computer-representation of the first scene, the predicted second scene, and by manipulating the predicted second scene using the determined difference between the predicted second scene and the observed second scene from the compressed computer-representation. ### Claim 14 (depends on 11) The method of claim 11 further comprises storing the compressed computer-representation in a storage device. ### Claim 15 (depends on 11) The method of claim 11 further comprises transmitting the computer-representation of the first scene and the compressed computer-representation to a remote device, the remote device is configured to reconstruct the computer-representation of the observed second scene by: predicting, using the model of the environment, a predicted scene of the environment based on the computer-representation of the first scene and by modifying the predicted scene using the determined difference between the predicted second scene and the observed second scene to reconstruct the computer-representation of the observed second scene. ### Claim 16 (depends on 11) The method of claim 11, wherein said predicting is performed using a deterministic machine-learning based predictor, whereby compression and uncompression are performed while relying on identical predictions by the deterministic machine-learning based predictor given identical input, whereby the difference between the predicted second scene and the observed second scene is consistent for different predictions performed using the deterministic machine-learning based predictor. ### Claim 17 A method for uncompressing scene information from a compressed computer-representation of an observed scene, the method comprising: obtaining a computer-representation of a base scene, wherein the base scene and the observed scene are scenes captured by one or more sensors or portion thereof, the one or more sensors are configured to capture scenes of an environment, and the environment comprises one or more physical objects; predicting, using a model of the environment and based on the computer representation of the base scene of the environment, a predicted scene of the environment, the predicted scene comprises predicted pixel values; and obtaining from the compressed computer-representation a determined difference between the predicted scene and the observed scene, the determined difference is based on different values of the predicted pixel values and observed pixel values of the computer-representation of the observed scene; modifying the predicted scene of the environment based on the determined difference between the predicted scene and the observed scene to reconstruct the computer-representation of the observed scene. ### Claim 18 (depends on 17) The method of claim 17, wherein the compressed computer-representation of the observed scene was generated using the predicted scene of the environment, which was obtained using the model of the environment and the computer-representation of the base scene. ### Claim 19 (depends on 17) The method of claim 17 further comprises: receiving transmission of the compressed computer-representation over a network; wherein said obtaining of the computer-representation of the base scene includes receiving transmission of the computer-representation of the base scene over the network, thereby reducing utilized bandwidth for the transmission of the computer-representation of the observed scene. ### Claim 20 (depends on 1) The system of claim 1, wherein the compressed computer-representation excludes parts of the observed second scene that match the predicted second scene. ### Claim 21 (depends on 1) The system of claim 1, wherein the compressed computer-representation consists only of the determined difference between the predicted second scene and the observed second scene.