Deep neural network pose estimation system
AssigneeMagic Leap, Inc.
InventorsDanying Hu, Daniel DeTone, Tomasz Jan Malisiewicz
Life of this patent
Estimating the term…
Abstract
A deep neural network provides real-time pose estimation by combining two custom deep neural networks, a location classifier and an ID classifier, with a pose estimation algorithm to achieve a 6D0F location of a fiducial marker. The locations may be further refined into subpixel coordinates using another deep neural network. The networks may be trained using a combination of auto-labeled videos of the target marker, synthetic subpixel corner data, and/or extreme data augmentation. The deep neural network provides improved pose estimations particularly in challenging low-light, high-motion, and/or high-blur scenarios.
Claims20
A system comprising:
- a mixed reality device comprising a wearable headset and a handheld controller;
- a hardware computer processor;
- a non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by the hardware computer processor to cause the system to perform operations comprising:
- accessing, from an imaging device, an image comprising a plurality of fiducial markers on a substrate;
- applying, using a two-headed convolutional neural network, a plurality of stages of encoding to the image to generate an encoded image with reduced dimensionality;
- applying a first neural network head of the two-headed convolutional neural network to the encoded image to determine two-dimensional locations of fiducial markers in the encoded image;
- applying a second neural network head of the two-headed convolutional neural network to the encoded image to determine identifiers associated with respective fiducial markers detected in the encoded image;
- determining, using a subpixel refinement component, a subpixel corner location based on the two-dimensional locations;
- and
- determining a pose of the imaging device based at least on the subpixel corner location and the determined identifiers of the fiducial markers.
A system for determining pose of a camera, the system comprising:
- an encoder comprising one or more processors and performing spatial downsampling of an image acquired from an augmented reality headset to generate a spatially downsampled image;
- a two-headed convolutional neural network comprising:
- a first head that detects two-dimensional locations of fiducial boards in the spatially downsampled image;
- a second head that determines identifiers encoded at the two-dimensional locations of fiducial boards in the spatially downsampled image;
- a subpixel refinement component that receives as input image patches of the spatially downsampled image and, for each image patch, provides as output a subpixel corner location determined based on the two-dimensional locations of fiducial boards in the spatially downsampled image;
- and
- a pose estimation component that determines a pose augmented reality headset based at least on the two-dimensional locations and the subpixel corner locations.
Dependencies are inferred from the claim text.