# US 2019/0168384 A1 · System, Method and Product for Utilizing Prediction Models of an Environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0168384 A1 - Document ID: US20190168384A1 - Status: Published application - Kind: A1 - Application number: 16/272,655 - Filed: 2019-02-11 - Published: 2019-06-06 - Earliest claimed: 2016-06-09 - Assignees: Shmuel Ur Innovation LTD. - Applicants: Shmuel Ur Innovation Ltd. - Inventors: Shmuel Ur; Vlad Dabija; David Hirshberg - Web page: https://patentgrep.com/patent/US20190168384A1 ## CPC - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - B25J 9/1612: {characterised by the hand, wrist, grip control} - B25J 9/1669: {characterised by special application, e.g. multi-arm co-operation, assembly, grasping} - B25J 9/1697: {Vision controlled systems} - G06T 7/12: Edge-based segmentation - G06T 7/75: {involving models} - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G05B 2219/39536: Planning of hand motion, grasping - G05B 2219/40425: Sensing, vision based motion planning - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract A first method comprising: predicting a scene of an environment using a model of the environment and based on a first scene of the environment obtained from sensors observing scenes of the environment; comparing the predicted scene with an observed scene from the sensors; and performing an action based on differences determined between the predicted scene and the observed scene. A second method comprising applying a vibration stimuli on an object via a computer-controlled component; obtaining a plurality of images depicting the object from a same viewpoint, captured during the application of the vibration stimuli. The second method further comprising comparing the plurality of images to detect changes occurring in response to the application of the vibration stimuli, which changes are attributed to a change of a location of a boundary of the object; and determining the boundary of the object based on the comparison. ## Claims ### Claim at position 1 1-28. (canceled) ### Claim 29 A method comprising: obtaining an observed first scene of an environment from one or more sensors, wherein the one or more sensors observe scenes of the environment, wherein the environment comprises one or more physical objects; determining a stimuli force to be applied on the one or more physical objects; applying, using one or more computer-controlled components, the stimuli force to the one or more physical objects of the environment; obtaining an observed second scene of the environment under the stimuli force from the one or more sensors; and performing an action based on the observed first scene and the observed second scene. ### Claim 30 (depends on 29) The method of claim 29, wherein the observed second scene captures induced movement of objects within the environment produced by the stimuli force. ### Claim 31 (depends on 29) The method of claim 29, wherein the stimuli force is a vibration force. ### Claim 32 (depends on 31) The method of claim 31, wherein the vibration force is an ultrasonic vibration force. ### Claim 33 (depends on 31) The method of claim 31, wherein a frequency of the vibration force is in a range between 10-100 Hertz. ### Claim 34 (depends on 31) The method of claim 31, wherein a frequency of the vibration force is in a range between 20 KHz-1500 GHz. ### Claim 35 (depends on 31) The method of claim 31, wherein the observed second scene captures induced vibration of objects within the environment produced by the vibration stimuli force. ### Claim 36 (depends on 31) The method of claim 31, wherein the one or more sensors comprise at least one of subsonic transducer, sonic transducer and ultrasonic transducer. ### Claim 37 (depends on 31) The method of claim 31, wherein the one or more sensors comprise a microphone. ### Claim 38 (depends on 31) The method of claim 31, wherein a frequency of the vibration force is below a sampling frequency of the one or more sensors. ### Claim 39 (depends on 31) The method of claim 31, wherein a frequency of the vibration force is above a sampling frequency of the one or more sensors, wherein the frequency is correlated with the sampling frequency. ### Claim 40 (depends on 29) The method of claim 29, wherein the computer-controlled component is a robotic device. ### Claim 41 (depends on 29) The method of claim 29, wherein the computer-controlled component is a conveyor belt or a vibrating plate. ### Claim 42 (depends on 29) The method of claim 29, wherein said one or more sensors are one or more cameras, wherein movements or vibration are detectable in pixels associated with the edge of moving or vibrating objects in images captured by the one or more cameras. ### Claim 43 (depends on 29) The method of claim 29, wherein the one or more physical objects comprise a pile of physical objects. ### Claim 44 (depends on 43) The method of claim 43, wherein said performed action comprises determining an order of picking up objects from the pile. ### Claim 45 (depends on 43) The method of claim 43, wherein said performed action comprises picking objects from the pile. ### Claim 46 (depends on 43) The method of claim 43, wherein said performed action comprises updating a model of the pile. ### Claim 47 (depends on 46) The method of claim 46 further comprises determining, based on said comparing, a mechanical coupling between two or more objects of the pile of physical objects that was not modeled in the model of the pile, wherein said updating the model comprises introducing a modeling of the mechanical coupling to the model. ### Claim 48 (depends on 47) The method of claim 47 further comprises selecting the object upon which the stimuli force is to be applied, wherein said selecting comprises selecting the object based on a determination that applying stimuli force on the object is expected to provide information useful for validating or refining the model of the pile. ### Claim 49 (depends on 29) The method of claim 29, wherein an intensity of the stimuli force is below a maximal threshold, wherein the maximal threshold is an intensity capable of permanently changing a structure of the pile. ### Claim 50 (depends on 29) The method of claim 29, wherein an intensity of the stimuli force is above a minimal threshold, wherein the minimal threshold is an intensity capable of moving the physical object in a manner detectable by the one or more sensors. ### Claim 51 A method comprising: applying a vibration stimuli on an object, wherein said applying is performed via a computer-controlled component that is capable of applying the vibration stimuli; obtaining a plurality of images of the object, wherein the plurality of images depicting the object from a same viewpoint, wherein at least a portion of the plurality of images are captured during the application of the vibration stimuli; comparing the plurality of images to detect changes occurring in response to the application of the vibration stimuli, which changes are attributed to a change of a location of a boundary of the object; and determining the boundary of the object based on said comparing. ### Claim 52 (depends on 51) The method of claim 51, wherein the computer-controlled component is a robotic device. ### Claim 53 (depends on 51) The method of claim 51 further comprises performing motion deletion, wherein said performing motion deletion comprises: detecting at least one change in the plurality of images that are attributed to a cause different than the application of the vibration stimuli; and disregarding the at least one change upon performing said determining the boundary of the object. ### Claim 54 (depends on 51) The method of claim 51, wherein the plurality of images are obtained from a camera, wherein said applying is performed so as the object is vibrated in a manner that is synchronized with a capture timing of the camera; whereby effects of motion occurring irrespective of the vibration stimuli are distinguishable from effects of the vibration stimuli, wherein said determining the boundary is performed based on the effects of the vibration stimuli and disregarding the effects of the motion occurring irrespective of the vibration stimuli. ### Claim 55 (depends on 51) The method of claim 51 further comprising: having an initial model of an environment comprising the object and at least one element, wherein the initial model comprises an initial boundary of the object; wherein said comparing comprises detecting changes occurring in response to the application of the vibration stimuli on the object, which changes are attributed to the at least one element; wherein said determining the boundary of the object comprises expanding the initial boundary to cover the at least one element; and updating the initial model to provide an updated model, wherein the updated model comprises the boundary of the object. ### Claim 56 (depends on 51) The method of claim 51, wherein said determining the boundary comprises: identifying, in each image of the plurality of images, a contour associated with an estimated boundary of the object, wherein each contour comprising a plurality of contour points; determining the boundary based on an average computation of each point of the plurality of contour points, wherein the average computation of each point is computed based on points in different images of the plurality of images. ### Claim 57 A system comprising: one or more sensors, configured to observe scenes of an environment, wherein the environment comprises one or more physical objects; means to apply stimuli force; a computer configured to determine a stimuli force to be applied on the one or more physical objects using said means to apply stimuli force; wherein said computer is further configured to obtain an observed first scene of the environment from said one or more sensors prior to the stimuli force being applied, and an observed second scene of the environment from said one or more sensors under the stimuli force from the one or more sensors; and wherein said computer is further configured to perform an action based on the observed first scene and the observed second scene. ### Claim 58 (depends on 57) The system of claim 57, wherein the stimuli force is a vibration force. ### Claim 59 (depends on 57) The system of claim 57, wherein the computer comprising a boundary detection module for detecting, based on the observed first scene and the observed second scene, a boundary of the one or more physical objects. ### Claim 60 (depends on 57) The system of claim 57, wherein the computer retaining a modeling of the one or more physical objects, wherein the action comprises updating the modeling.