Back to results
US 12,611,770 B2Granted
Category-level manipulation from visual demonstration
AssigneeIntrinsic Innovation LLC
InventorsWenzhao Lian, Bowen Wen, Stefan Schaal
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for robotic control using demonstrations to learn category-level manipulation task. One of the methods includes obtaining a collection of object models for a plurality of different types of objects belonging to a same object category and training a category-level representation in a category-level space from the collection of object models. A category-level trajectory is generated the demonstration data of a demonstration object. For a new object in the object category, a trajectory projection is generated in the category-level space, which is used to cause a robot to perform the robotic manipulation task on the new object.
Claims19
A method performed by one or more computers, the method comprising:
- obtaining demonstration data representing a trajectory of a demonstration object while a manipulation task is performed on the demonstration object;
- generating a category-level trajectory from the demonstration data, including generating a sequence of poses of the demonstration object relative to a reference point in an environment of the demonstration object;
- generating, for each of the sequence of poses, a respective corresponding attention heatmap and an anchor point defining an origin of a category-level coordinate system;
- receiving data representing a new object belonging to the object category;
- generating a trajectory projection of the category-level trajectory according to the representation of the new instance of the object category in the category-level space by training a neural network to learn a mapping between a partial point cloud representation of the new object and a category-level representation of object models of the object category in the category-level space, wherein the neural network generates a location of the new object in the category-level space from the partial point cloud representation and wherein generating the trajectory projection further comprises, after learning the mapping:
- transferring the attention heatmap to the partial point cloud representation of the new object;
- and
- determining, from the attention heatmap, an anchor point for the new object to align the coordinate frame of the new object with the category-level coordinate system for the trajectory projection;
- using the trajectory projection to cause a robot to perform the manipulation task using the new object belonging to the object category.
6 more elements
A system comprising:
- one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
- obtaining demonstration data representing a trajectory of a demonstration object while a manipulation task is performed on the demonstration object;
- generating a category-level trajectory from the demonstration data, including generating a sequence of poses of the demonstration object relative to a reference point in the environment of the demonstration object;
- generating, for each of the sequence of poses, a respective corresponding attention heatmap and an anchor point defining an origin of a category-level coordinate system;
- receiving data representing a new object belonging to the object category;
- generating a trajectory projection of the category-level trajectory according to the representation of the new instance of the object category in the category-level space by training a neural network to learn a mapping between a partial point cloud representation of the new object and a category-level representation of object models of the object category in the category-level space, wherein the neural network generates a location of the new object in the category-level space from the partial point cloud representation and wherein generating the trajectory projection further comprises, after learning the mapping:
- transferring the attention heatmap to the partial point cloud representation of the new object;
- and
- determining, from the attention heatmap, an anchor point for the new object to align the coordinate frame of the new object with the category-level coordinate system for the trajectory projection;
- using the trajectory projection to cause a robot to perform the manipulation task using the new object belonging to the object category.
7 more elements
One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
- obtaining demonstration data representing a trajectory of a demonstration object while a manipulation task is performed on the demonstration object;
- generating a category-level trajectory from the demonstration data, including generating a sequence of poses of the demonstration object relative to a reference point in the environment of the demonstration object;
- generating, for each of the sequence of poses, a respective corresponding attention heatmap and an anchor point defining an origin of a category-level coordinate system;
- receiving data representing a new object belonging to the object category;
- generating a trajectory projection of the category-level trajectory according to the representation of the new instance of the object category in the category-level space by training a neural network to learn a mapping between a partial point cloud representation of the new object and a category-level representation of object models of the object category in the category-level space, wherein the neural network generates a location of the new object in the category-level space from the partial point cloud representation and wherein generating the trajectory projection further comprises, after learning the mapping:
- transferring the attention heatmap to the partial point cloud representation of the new object;
- and
- determining, from the attention heatmap, an anchor point for the new object to align the coordinate frame of the new object with the category-level coordinate system for the trajectory projection;
- using the trajectory projection to cause a robot to perform the manipulation task using the new object belonging to the object category.
6 more elements
Dependencies are inferred from the claim text.