System and method for constructing a 3D scene model from an image

A method for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of:

• (a) receiving, by a processor, the current image of the scene;
• (b) initializing, by the processor, the 3D scene model to the prior 3D scene model;
• and
• (c) modifying the 3D scene model to be consistent with the image, by:
• (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data by:
• (A) rendering each 3D object model p of the 3D scene model, resulting in a rendered image for p;
• and
• (B) finding locations in the image such that a first difference between the value of the image data at a location and the value of the rendered image for 3D object model p at the location is less than or equal to a tolerance and for all other 3D object models q, a second difference between the value of the image data at the location and the value of the rendered image for 3D object model q at the location is not less than the first difference, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model of the 3D scene model being unassociated data;
• (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• and
• (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model;
• (d) computing, by the processor, a trajectory that avoids the 3D object models of the 3D scene model;
• (e) directing, by the processor, the robotic manipulator to follow the computed trajectory in the scene;
• and
• (f) following, by the robotic manipulator, the trajectory directed by the processor.

A method for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of:

• (a) receiving, by a processor, the current image of the scene;
• (b) initializing, by the processor, the 3D scene model to the prior 3D scene model;
• and
• (c) modifying the 3D scene model to be consistent with the image, by:
• (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data;
• (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• and
• (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent by:
• (A) rendering the 3D scene model, resulting in a synthetic rendering;
• (B) using the synthetic rendering to compute synthetic data associated with 3D object models;
• (C) comparing the associated data with the synthetic data associated with objects to find 3D object models in the 3D scene model that do not appear in the image;
• and
• (D) determining that a given 3D object model q in the 3D scene model that does not appear in the image is absent from the scene by finding that there is no set of other 3D object models that may be computed from the image that occludes the 3D object model q;
• and removing the absent 3D object models from the 3D scene model;
• (d) computing, by the processor, a trajectory that avoids the 3D object models of the 3D scene model;
• (e) directing, by the processor, the robotic manipulator to follow the computed trajectory in the scene;
• and
• (f) following, by the robotic manipulator, the trajectory directed by the processor.

A method for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of:

• (a) receiving, by a processor, the current image of the scene;
• (b) initializing, by the processor, the 3D scene model to the prior 3D scene model;
• and
• (c) modifying the 3D scene model to be consistent with the image, by:
• (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data, by:
• (A) processing the image to detect image features;
• (B) processing each 3D object model in the 3D scene model to detect model features that would be visible;
• (C) associating the image features to visible model features of a given 3D object model p when they match at the same or nearby locations and have no better match with visible model features from any other 3D object model q of the 3D scene model;
• (D) taking the image features associated with model features of 3D object model p as data associated with 3D object model p;
• (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• and
• (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model;
• (d) computing, by the processor, a trajectory that avoids the 3D object models of the 3D scene model;
• (e) directing, by the processor, the robotic manipulator to follow the computed trajectory in the scene;
• and
• (f) following, by the robotic manipulator, the trajectory directed by the processor.

A method for grasping an object in a scene by a robotic manipulator, based upon and a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of:

• (a) receiving, by a processor, the current image of the scene, wherein the image of the scene is a range image;
• (b) initializing, by the processor, the 3D scene model to the prior 3D scene model, wherein each object p of the 3D scene model has a signed distance function;
• and
• (c) modifying the 3D scene model to be consistent with the image, by:
• (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data by, for each 3D object model p, finding locations in the image such that the value of the signed distance function of the 3D object model p applied to the location is less than or equal to a specified tolerance, and the value of the distance function for all other 3D object models q applied to the location is not less than the value of p's distance function, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model p of the 3D scene model being unassociated data;
• (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• and
• (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model;
• (d) computing, by the processor, a grasp of one of the 3D object models of the 3D scene model;
• (e) directing, by the processor, the robotic manipulator to grasp the corresponding object in the scene using the computed grasp;
• and
• (f) grasping, by the robotic manipulator, the object directed by the processor using the grasp directed by the processor.

A method for grasping an object in a scene by a robotic manipulator, based upon and a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of:

• (a) receiving, by a processor, the current image of the scene;
• (b) initializing, by the processor, the 3D scene model to the prior 3D scene model;
• and
• (c) modifying the 3D scene model to be consistent with the image, by:
• (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data by:
• (A) rendering each 3D object model p of the 3D scene model, resulting in a rendered image for p;
• and
• (B) finding locations in the image such that a first difference between the value of the image data at a location and the value of the rendered image for 3D object model p at the location is less than or equal to a tolerance and for all other 3D object models q, a second difference between the value of the image data at the location and the value of the rendered image for 3D object model q at the location is not less than the first difference, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model of the 3D scene model being unassociated data;
• (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• and
• (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model;
• (d) computing, by the processor, a grasp of one of the 3D object models of the 3D scene model;
• (e) directing, by the processor, the robotic manipulator to grasp the corresponding object in the scene using the computed grasp;
• and
• (f) grasping, by the robotic manipulator, the object directed by the processor using the grasp directed by the processor.

A method for grasping an object in a scene by a robotic manipulator, based upon and a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of:

• (a) receiving, by a processor, the current image of the scene;
• (b) initializing, by the processor, the 3D scene model to the prior 3D scene model;
• and
• (c) modifying the 3D scene model to be consistent with the image, by:
• (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data;
• (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• and
• (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent by:
• (A) rendering the 3D scene model, resulting in a synthetic rendering;
• (B) using the synthetic rendering to compute synthetic data associated with 3D object models;
• (C) comparing the associated data with the synthetic data associated with objects to find 3D object models in the 3D scene model that do not appear in the image;
• and
• (D) determining that a given 3D object model q in the 3D scene model that does not appear in the image is absent from the scene by finding that there is no set of other 3D object models that may be computed from the image that occludes the 3D object model q;
• and removing the absent 3D object models from the 3D scene model;
• (d) computing, by the processor, a grasp of one of the 3D object models of the 3D scene model;
• (e) directing, by the processor, the robotic manipulator to grasp the corresponding object in the scene using the computed grasp;
• and
• (f) grasping, by the robotic manipulator, the object directed by the processor using the grasp directed by the processor.

A method for grasping an object in a scene by a robotic manipulator, based upon and a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the method comprising the steps of:

• (a) receiving, by a processor, the current image of the scene;
• (b) initializing, by the processor, the 3D scene model to the prior 3D scene model;
• and
• (c) modifying the 3D scene model to be consistent with the image, by:
• (i) comparing, by the processor, data of the image with 3D object models of the 3D scene model, resulting in associated data and unassociated data, by:
• (A) processing the image to detect image features;
• (B) processing each 3D object model in the 3D scene model to detect model features that would be visible;
• (C) associating the image features to visible model features of a given 3D object model p when they match at the same or nearby locations and have no better match with visible model features from any other 3D object model q of the 3D scene model;
• (D) taking the image features associated with model features of 3D object model p as data associated with 3D object model p;
• (ii) using, by the processor, the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• and
• (iii) using, by the processor, the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model;
• (d) computing, by the processor, a grasp of one of the 3D object models of the 3D scene model;
• (e) directing, by the processor, the robotic manipulator to grasp the corresponding object in the scene using the computed grasp;
• and
• (f) grasping, by the robotic manipulator, the object directed by the processor using the grasp directed by the processor.

A system for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising:

• (a) a processor configured to:
• (i) receive the current image of the scene, wherein the image of the scene is a range image;
• (ii) initialize the 3D scene model to the prior 3D scene model wherein each object p of the 3D scene model has a signed distance function;
• and
• (iii) modify the 3D scene model to be consistent with the image, by:
• (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data by, for each 3D object model p, finding locations in the image such that the value of the signed distance function of the 3D object model p applied to the location is less than or equal to a specified tolerance, and the value of the distance function for all other 3D object models q applied to the location is not less than the value of p's distance function, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model p of the 3D scene model being unassociated data;
• (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model;
• (iv) compute a trajectory that avoids the 3D object models of the 3D scene model;
• and
• (v) direct the robotic manipulator to follow the computed trajectory in the scene;
• and
• (b) the robotic manipulator configured to follow a trajectory as directed by the processor.

A system for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising:

• (a) a processor configured to:
• (i) receive the current image of the scene;
• (ii) initialize the 3D scene model to the prior 3D scene model;
• and
• (iii) modify the 3D scene model to be consistent with the image, by:
• (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data by:
• (1) rendering each 3D object model p of the 3D scene model, resulting in a rendered image for p;
• and
• (2) finding locations in the image such that a first difference between the value of the image data at a location and the value of the rendered image for 3D object model p at the location is less than or equal to a tolerance and for all other 3D object models q, a second difference between the value of the image data at the location and the value of the rendered image for 3D object model q at the location is not less than the first difference, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model of the 3D scene model being unassociated data;
• (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model;
• (iv) compute a trajectory that avoids the 3D object models of the 3D scene model;
• and
• (v) direct the robotic manipulator to follow the computed trajectory in the scene;
• and
• (b) the robotic manipulator configured to follow a trajectory as directed by the processor.

A system for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising:

• (a) a processor configured to:
• (i) receive the current image of the scene;
• (ii) initialize the 3D scene model to the prior 3D scene model;
• and
• (iii) modify the 3D scene model to be consistent with the image, by:
• (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data;
• (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent by:
• (1) rendering the 3D scene model, resulting in a synthetic rendering;
• (2) using the synthetic rendering to compute synthetic data associated with 3D object models;
• (3) comparing the associated data with the synthetic data associated with objects to find 3D object models in the 3D scene model that do not appear in the image;
• and
• (4) determining that a given 3D object model q in the 3D scene model that does not appear in the image is absent from the scene by finding that there is no set of other 3D object models that may be computed from the image that occludes the 3D object model q;
• and removing the absent 3D object models from the 3D scene model;
• (iv) compute a trajectory that avoids the 3D object models of the 3D scene model;
• and
• (v) direct the robotic manipulator to follow the computed trajectory in the scene;
• and
• (b) the robotic manipulator configured to follow a trajectory as directed by the processor.

A system for moving a robotic manipulator in free space to avoid objects in a scene, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising:

• (a) a processor configured to:
• (i) receive the current image of the scene;
• (ii) initialize the 3D scene model to the prior 3D scene model;
• and
• (iii) modify the 3D scene model to be consistent with the image, by:
• (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data, by:
• (1) processing the image to detect image features;
• (2) processing each 3D object model in the 3D scene model to detect model features that would be visible;
• (3) associating the image features to visible model features of a given 3D object model p when they match at the same or nearby locations and have no better match with visible model features from any other 3D object model q of the 3D scene model;
• (4) taking the image features associated with model features of 3D object model p as data associated with 3D object model p;
• (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model;
• (iv) compute a trajectory that avoids the 3D object models of the 3D scene model;
• and
• (v) direct the robotic manipulator to follow the computed trajectory in the scene;
• and
• (b) the robotic manipulator configured to follow a trajectory as directed by the processor.

A system for grasping an object in a scene by a robotic manipulator, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising:

• (a) a processor configured to:
• (i) receive the current image of the scene, wherein the image of the scene is a range image;
• (ii) initialize the 3D scene model to the prior 3D scene model, wherein each object p of the 3D scene model has a signed distance function;
• and
• (iii) modify the 3D scene model to be consistent with the image, by:
• (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data by, for each 3D object model p, finding locations in the image such that the value of the signed distance function of the 3D object model p applied to the location is less than or equal to a specified tolerance, and the value of the distance function for all other 3D object models q applied to the location is not less than the value of p's distance function, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model p of the 3D scene model being unassociated data;
• (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model;
• (iv) compute a grasp of one of the 3D object models of the 3D scene model;
• and
• (v) direct the robotic manipulator to grasp the corresponding object in the scene using the computed grasp;
• and
• (b) the robotic manipulator configured to grasp an object as directed by the processor.

A system for grasping an object in a scene by a robotic manipulator, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising:

• (a) a processor configured to:
• (i) receive the current image of the scene;
• (ii) initialize the 3D scene model to the prior 3D scene model;
• and
• (iii) modify the 3D scene model to be consistent with the image, by:
• (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data by:
• (1) rendering each 3D object model p of the 3D scene model, resulting in a rendered image for p;
• and
• (2) finding locations in the image such that a first difference between the value of the image data at a location and the value of the rendered image for 3D object model p at the location is less than or equal to a tolerance and for all other 3D object models q, a second difference between the value of the image data at the location and the value of the rendered image for 3D object model q at the location is not less than the first difference, the data points of these locations being associated data of the 3D object model p, and the data points of any locations not associated with any 3D object model of the 3D scene model being unassociated data;
• (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model;
• (iv) compute a grasp of one of the 3D object models of the 3D scene model;
• and
• (v) direct the robotic manipulator to grasp the corresponding object in the scene using the computed grasp;
• and
• (b) the robotic manipulator configured to grasp an object as directed by the processor.

A system for grasping an object in a scene by a robotic manipulator, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising:

• (a) a processor configured to:
• (i) receive the current image of the scene;
• (ii) initialize the 3D scene model to the prior 3D scene model;
• and
• (iii) modify the 3D scene model to be consistent with the image, by:
• (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data;
• (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model by:
• (1) rendering the 3D scene model, resulting in a synthetic rendering;
• (2) using the synthetic rendering to compute synthetic data associated with 3D object models;
• (3) comparing the associated data with the synthetic data associated with objects to find 3D object models in the 3D scene model that do not appear in the image;
• and
• (4) determining that a given 3D object model q in the 3D scene model that does not appear in the image is absent from the scene by finding that there is no set of other 3D object models that may be computed from the image that occludes the 3D object model q;
• (iv) compute a grasp of one of the 3D object models of the 3D scene model;
• and
• (v) direct the robotic manipulator to grasp the corresponding object in the scene using the computed grasp;
• and
• (b) the robotic manipulator configured to grasp an object as directed by the processor.

A system for grasping an object in a scene by a robotic manipulator, based upon a prior 3D scene model corresponding to the scene in a prior state, the prior 3D scene model comprised of 3D object models corresponding to objects of the scene in the prior state, the system comprising:

• (a) a processor configured to:
• (i) receive the current image of the scene;
• (ii) initialize the 3D scene model to the prior 3D scene model;
• and
• (iii) modify the 3D scene model to be consistent with the image, by:
• (A) comparing data of the image with the 3D object models of the 3D scene model, resulting in associated data and unassociated data by:
• (1) processing the image to detect image features;
• (2) processing each 3D object model in the 3D scene model to detect model features that would be visible;
• (3) associating the image features to visible model features of a given 3D object model p when they match at the same or nearby locations and have no better match with visible model features from any other 3D object model q of the 3D scene model;
• (4) taking the image features associated with model features of 3D object model p as data associated with 3D object model p;
• (B) using the unassociated data to compute new 3D object models and adding the new 3D object models to the 3D scene model;
• (C) using the associated data to detect 3D object models in the prior 3D scene model that are absent and removing the absent 3D object models from the 3D scene model;
• (iv) compute a grasp of one of the 3D object models of the 3D scene model;
• and
• (v) direct the robotic manipulator to grasp the corresponding object in the scene using the computed grasp;
• and
• (b) the robotic manipulator configured to grasp an object as directed by the processor.