System and method for multimodal object-centric representation learning
AssigneeHonda Motor Co., Ltd.
InventorsAlireza Rezazadeh, Nawid Jamali, Soshi Iba
Life of this patent
Estimating the term…
Abstract
A system and method for multimodal object-centric representation learning that include receiving data associated with an image and a depth map of an object. The system and method also include determining an object-surface point cloud based on the image and the depth map. The system and method additionally include determining multi-resolution receptive fields based on the object-surface point cloud. The system and method further include passing the multi-resolution receptive fields through convolutional encoders to learn an object centric representation of the object.
Claims21
A computer-implemented method for multimodal object-centric representation learning comprising:
- receiving data associated with an image and a depth map of an object;
- determining an object-surface point cloud based on the image and the depth map;
- determining multi-resolution receptive fields based on the object-surface point cloud, including projecting object-surface points of the object-surface point cloud into the image to register a 2d object point in image coordinates of the object for each object-surface point of the object-surface point cloud and encapsulating the 2d object point with a plurality of bounding boxes of different sizes as the multi-resolution receptive fields;
- and
- passing the multi-resolution receptive fields through convolutional encoders to learn an object centric representation of the object, wherein the object centric representation is utilized to electronically control at least one electronic component to complete at least one object specific task associated with the object.
A system for multimodal object-centric representation learning comprising:
- a memory storing instructions when executed by a processor cause the processor to:
- receive data associated with an image and a depth map of an object;
- determine an object-surface point cloud based on the image and the depth map;
- determine multi-resolution receptive fields based on the object-surface point cloud, including projecting object-surface points of the object-surface point cloud into the image to register a 2d object point in image coordinates of the object for each object-surface point of the object-surface point cloud;
- and
- pass the multi-resolution receptive fields through convolutional encoders to learn an object centric representation of the object and encapsulate the 2d object point with a plurality of bounding boxes of different sizes as the multi-resolution receptive fields, wherein the object centric representation is utilized to electronically control at least one electronic component to complete at least one object specific task associated with the object.
A non-transitory computer readable storage medium storing instructions that when executed by a computer, which includes a processor performs a method, the method comprising:
- receiving data associated with an image and a depth map of an object;
- determining an object-surface point cloud based on the image and the depth map;
- determining multi-resolution receptive fields based on the object-surface point cloud, including projecting object-surface points of the object-surface point cloud into the image to register a 2d object point in image coordinates of the object for each object-surface point of the object-surface point cloud;
- and
- passing the multi-resolution receptive fields through convolutional encoders to learn an object centric representation of the object and encapsulating the 2d object point with a plurality of bounding boxes of different sizes as the multi-resolution receptive fields, wherein the object centric representation is utilized to electronically control at least one electronic component to complete at least one object specific task associated with the object.
Dependencies are inferred from the claim text.