US 11,379,688 B2Granted

Systems and methods for keypoint detection with convolutional neural networks

AssigneePacksize LLC

InventorsPaolo Di Febbo, Carlo Dal Mutto, Kinh Tieu

Life of this patent

Estimating the term…

Loading patent…

Abstract

A keypoint detection system includes: a camera system including at least one camera; and a processor and memory, the processor and memory being configured to: receive an image captured by the camera system; compute a plurality of keypoints in the image using a convolutional neural network including: a first layer implementing a first convolutional kernel; a second layer implementing a second convolutional kernel; an output layer; and a plurality of connections between the first layer and the second layer and between the second layer and the output layer, each of the connections having a corresponding weight stored in the memory; and output the plurality of keypoints of the image computed by the convolutional neural network.

Claims30

Dependent claims

A system comprising:

a camera system comprising at least one camera;
and
a processor and memory, the processor and memory being configured to:
receive an image captured by the camera system;
compute a plurality of keypoints in the image, using a convolutional neural network comprising:
a first layer implementing a first convolutional kernel,
a second layer implementing a second convolutional kernel,
an output layer, and
a plurality of connections between the first layer and the second layer and between the second layer and the output layer, each of the connections having a corresponding weight stored in the memory, wherein the weights are computed by a training system comprising a processor and memory, the memory of the training system having instructions stored thereon that, when executed by the processor of the training system, cause the training system to:
generate training data comprising a plurality of training images and training detected keypoints of the training images,
transform the training detected keypoints to a response space,
uniformly sample patches from the response space,
identify patches of the training images corresponding to the sampled patches to generate a first training set for training the convolutional neural network, and
compute a plurality of first stage learned neural network parameters from the first training set using backpropagation, and
further train the convolutional neural network using additional patches from the response space and patches of the training images corresponding to the sampled response space patches;
and
output the plurality of keypoints of the image computed by the convolutional neural network.

14 more elements

A system comprising:

a camera system comprising at least one camera;
and
a processor and memory, the processor and memory being configured to:
receive an image captured by the camera system;
compute a plurality of keypoints in the image using a convolutional neural network comprising:
a first layer implementing a first convolutional kernel;
a second layer implementing a second convolutional kernel;
an output layer;
and
a plurality of connections between the first layer and the second layer and between the second layer and the output layer, each of the connections having a corresponding weight stored in the memory;
and
output the plurality of keypoints of the image computed by the convolutional neural network,
wherein the camera system comprises:
a first infrared camera having a first optical axis and a field of view;
a second infrared camera having a second optical axis substantially parallel to the first optical axis and a field of view substantially overlapping the field of view of the first infrared camera;
and
an infrared illuminator configured to project light in a direction within the field of view of the first infrared camera and the field of view of the second infrared camera,
wherein the image is a point cloud computed from infrared images captured by the first infrared camera and the second infrared camera,
wherein the camera system is arranged to place a conveyor belt in the field of view of the first infrared camera and the field of view of the second infrared camera, and
wherein the processor is configured to:
control the camera system to capture images of objects on the conveyor belt;
compute a plurality of point clouds from the images of the objects;
identify keypoints of the point clouds using the convolutional neural network;
and
merge two or more of the point clouds by identifying corresponding ones of the keypoints and rigidly transforming at least one of the point clouds to align the corresponding ones of the keypoints.

22 more elements

A method for computing keypoints of an image, comprising:

controlling, by a processor, a camera system comprising at least one camera to capture an image, wherein the image is a point cloud computed infrared images captured by a camera system comprising:
a first infrared camera having a first optical axis and a field of view,
a second infrared camera having a second optical axis and field of view, the first optical axis being substantially parallel to the second optical axis, and the field of view of the first infrared camera substantially overlapping the field of view of the second infrared camera, and
an infrared illuminator configured to project light in a direction within the field of view of the first camera and the field of view of second infrared camera;
controlling the camera system to capture images of objects on a conveyor belt;
computing a plurality of point clouds from the images of the objects;
identifying keypoints of the point clouds using a convolutional neural network;
merging two or more of the point clouds by identifying corresponding ones of the keypoints;
supplying, by the processor, the image to the convolutional neural network to compute a plurality of keypoints,
the convolutional neural network comprising:
a first layer implementing a first convolutional kernel;
a second layer implementing a second convolutional kernel;
an output layer;
and
a plurality of connections between the first layer and the second layer and between the second layer and the output layer, each of the connections having a corresponding weight stored in a memory;
and
outputting the plurality of keypoints of the image computed by the convolutional neural network.

14 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/924,162Filed 16 Mar 2018 · us provisional application
Priority: 16 Mar 2017
Published: US 2018/0268256 A120 Sep 2018
Granted: 5 Jul 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification28 codes

G06V 10/82
Using pattern recognition or machine learning: using neural networks
G06F 18/214
Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation: generating training…
G06F 18/2414
Distances to cluster centroïds: smoothing the distance, e.g. radial basis function networks [RBFN]
G06N 3/045
Architecture, e.g. interconnection topology: combinations of networks