Head-mounted display for virtual and mixed reality with inside-out positional, user body and environment tracking
AssigneeApple Inc.
InventorsSimon Fortin-Deschênes, Vincent Chapdelaine-Couture, Yan Côté, Anthony Ghannoum
Life of this patent
Estimating the term…
Abstract
A Head-Mounted Display system together with associated techniques for performing accurate and automatic inside-out positional, user body and environment tracking for virtual or mixed reality are disclosed. The system uses computer vision methods and data fusion from multiple sensors to achieve real-time tracking. High frame rate and low latency is achieved by performing part of the processing on the HMD itself.
Claims21
A method comprising:
- at a head-mounted device (HMD) including non-transitory memory, one or more processors, and a communications interface for communicating with first and second RGB camera sensors, first and second mono camera sensors, and a display;
- obtaining, via the first and second RGB camera sensors, pass-through stereo view images of a physical environment;
- obtaining, via the first and second mono camera sensors, stereo images;
- obtaining a dense depth map associated with the physical environment;
- performing embedded tracking based on the pass-through stereo view images from the first and second RGB camera sensors, the stereo images from the first and second mono camera sensors, and the dense depth map;
- generating rendered graphics associated with virtual content based on the embedded tracking;
- generating a display image by mixing the rendered graphics with the pass-through stereo view images from the first and second RGB camera sensors based on the dense depth map;
- and
- displaying, via the display, the display image.
A head-mounted device (HMD) comprising:
- a communications interface for communicating with first and second RGB stereo-cameras, first and second mono camera sensors, and a display;
- one or more processors;
- and
- a non-transitory memory storing one or more programs, which, when executed by the one or more processors, cause the HMD to:
- obtain via the first and second RGB camera sensors, pass-through stereo view images of a physical environment;
- obtain, via the first and second mono camera sensors, stereo images;
- obtain a dense depth map associated with the physical environment;
- perform embedded tracking based on the pass-through stereo view images from the first and second RGB camera sensors, the stereo images from the first and second mono camera sensors, and the dense depth map;
- generate rendered graphics associated with virtual content based on the embedded tracking;
- generate processed images by performing one or more image processing operations on the pass-through stereo view images from the first and second RGB camera sensors and the stereo images from the first and second mono camera sensors;
- generate a display image by mixing the rendered graphics with processed images based on the dense depth map;
- and
- display, via the display, the display image.
A non-transitory computer-readable medium having instructions encoded thereon, which, when executed by one or more processors of a head-mounted device (HMD) including a communications interface for communicating with first and second RGB stereo-cameras, first and second mono camera sensors, and a display, cause the HMD to:
- obtain via the first and second RGB camera sensors, pass-through stereo view images of a physical environment;
- obtain, via the first and second mono camera sensors, stereo images;
- obtain a dense depth map associated with the physical environment;
- perform embedded tracking based on the pass-through stereo view images from the first and second RGB camera sensors, the stereo images from the first and second mono camera sensors, and the dense depth map;
- generate rendered graphics associated with virtual content based on the embedded tracking;
- generate a display image by mixing the rendered graphics with the pass-through stereo view images from the first and second RGB camera sensors based on the dense depth map;
- and
- display, via the display, the display image.
Dependencies are inferred from the claim text.