Three-Dimensional Bounding Box From Two-Dimensional Image and Point Cloud Data
AssigneeZoox, Inc.
InventorsDanfei Xu, Dragomir Dimitrov Anguelov, Ashesh Jain
Life of this patent
Estimating the term…
Abstract
A three-dimensional bounding box is determined from a two-dimensional image and a point cloud. A feature vector associated with the image and a feature vector associated with the point cloud may be passed through a neural network to determine parameters of the three-dimensional bounding box. Feature vectors associated with each of the points in the point cloud may also be determined and considered to produce estimates of the three-dimensional bounding box on a per-point basis.
Claims20
A system for estimating a three-dimensional bounding box, the system including a non-transitory computer readable medium containing instructions that, when executed by one or more processors, cause the system to:
- receive an image captured from an image capture device;
- detect an object in the image;
- crop the image to form a cropped image including the object;
- receive point cloud data associated with the object;
- determine a first feature vector associated with the point cloud data, the first feature vector comprising a geometric feature vector;
- determine a second feature vector associated with the cropped image, the second feature vector comprising an appearance feature vector;
- pass the first feature vector and the second feature vector into a neural network;
- and
- receive, from the neural network, coordinates descriptive of a three-dimensional bounding box associated with the object.
A computer-implemented method for estimating a three-dimensional bounding box of an object in an environment, the computer-implemented method comprising:
- receiving an image of the environment from an image capture device;
- receiving point cloud data associated with the environment, the point cloud data comprising a plurality of points;
- detecting an object in the image;
- cropping the image to form a cropped image comprising an image of the object;
- inputting the cropped image into a first neural network;
- inputting the point cloud into a second neural network;
- extracting from the first neural network an appearance feature vector associated with the cropped image;
- extracting from the second neural network a global geometric feature vector associated with the point cloud data;
- extracting from the second neural network a plurality of per-point geometric feature vectors, individual of the per-point geometric feature vectors being associated with individual of the plurality of points;
- inputting the appearance feature vector, the global geometric feature vector, and the plurality of per-point geometric feature vectors into a third neural network;
- and
- receiving from the third neural network information associated with a three-dimensional bounding box of the object.
A system for estimating a three-dimensional bounding box, the system comprising:
- an autonomous vehicle;
- an image capture device associated with the autonomous vehicle and configured to capture images in an environment of the autonomous vehicle;
- a sensor associated with the autonomous vehicle and configured to output point cloud data corresponding to the environment;
- one or more processors;
- and
- non-transitory computer readable medium containing instructions that, when executed by the one or more processors, cause the system to:
- receive an image captured by the image capture device;
- detect an object in the image;
- crop the image to form a cropped image including the object;
- receive the point cloud data;
- determine a first feature vector associated with the point cloud data;
- determine a second feature vector associated with the cropped image;
- pass the first feature vector and the second feature vector into a neural network;
- and
- receive from the neural network coordinates associated with a three-dimensional bounding box for the object.
Dependencies are inferred from the claim text.