Semi-supervised keypoint based models

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a neural network to predict keypoints of unseen objects using a training dataset comprising labeled and unlabeled training data, the operations comprising:

• receiving the training dataset comprising a plurality of training samples, each training sample comprising a set of synchronized images of one or more objects from a respective scene, wherein each image in the set is synchronously taken by a respective camera from a different point of view, and wherein a subset of the set of synchronized images is labeled with ground-truth keypoints and the remaining images in the set are unlabeled;
• and
• for each of one or more training samples in the plurality of training samples:
• for each labeled image in the training sample and for each of the ground-truth keypoints in the labeled image:
• constructing a respective ground-truth heat-map of the ground-truth keypoint in the labeled image, and
• generating, using the neural network with the current values of the parameters, a respective predicted heat-map of the ground-truth keypoint in the labeled image, and
• training the neural network on the training sample by updating current values of parameters of the neural network to minimize a loss function,
• wherein the loss function is a combination of a supervised loss function and an unsupervised loss function, wherein the supervised loss function minimizes a difference between the ground-truth heat-maps and the predicted heat-maps generated by the neural network for the labeled images in accordance with the current values of the parameters of the neural network, and wherein the unsupervised loss function minimizes multi-view inconsistency among respective predicted keypoints generated by the neural network for the unlabeled images taken from different points of view of a same scene in accordance with the current values of the parameters, and
• wherein the neural network is configured to receive an input image and to generate, for each of a plurality of possible keypoints of one or more objects in the input image, a respective heat-map for the possible keypoint including a respective score for each of a plurality of locations in the input image, the respective score representing a likelihood that the possible keypoint is located at the location.

One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a neural network to predict keypoints of unseen objects using a training dataset comprising labeled and unlabeled training data, the operations comprising:

• receiving the training dataset comprising a plurality of training samples, each training sample comprising a set of synchronized images of one or more objects from a respective scene, wherein each image in the set is synchronously taken by a respective camera from a different point of view, and wherein a subset of the set of synchronized images is labeled with ground-truth keypoints and the remaining images in the set are unlabeled;
• and
• for each of one or more training samples in the plurality of training samples:
• for each labeled image in the training sample and for each of the ground-truth keypoints in the labeled image:
• constructing a respective ground-truth heat-map of the ground-truth keypoint in the labeled image, and
• generating, using the neural network with the current values of the parameters, a respective predicted heat-map of the ground-truth keypoint in the labeled image, and
• training the neural network on the training sample by updating current values of parameters of the neural network to minimize a loss function,
• wherein the loss function is a combination of a supervised loss function and an unsupervised loss function, wherein the supervised loss function minimizes a difference between the ground-truth heat-maps and the predicted heat-maps generated by the neural network for the labeled images in accordance with the current values of the parameters of the neural network, and wherein the unsupervised loss function minimizes multi-view inconsistency among respective predicted keypoints generated by the neural network for the unlabeled images taken from different points of view of a same scene in accordance with the current values of the parameters, and
• wherein the neural network is configured to receive an input image and to generate, for each of a plurality of possible keypoints of one or more objects in the input image, a respective heat-map for the possible keypoint including a respective score for each of a plurality of locations in the input image, the respective score representing a likelihood that the possible keypoint is located at the location.