Augmented reality display device with deep learning sensors
AssigneeMagic Leap, Inc.
InventorsAndrew Rabinovich, Tomasz Jan Malisiewicz, Daniel DeTone
Life of this patent
Estimating the term…
Abstract
A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).
Claims21
(canceled)
A system comprising:
- a plurality of sensors for capturing different types of sensor data;
- non-transitory memory configured to store executable instructions, and a neural network which receives the sensor data as input, wherein the neural network comprises:
- an input layer for receiving input of the neural network,
- a plurality of intermediate layers including a first intermediate layer of the plurality of intermediate layers coupled to the input layer, and
- a plurality of head components including a head output node coupled to a last intermediate layer of the plurality of intermediate layers through a plurality of head component layers;
- a hardware processor in communication with the plurality of sensors and the non-transitory memory, wherein the hardware processor is programmed by the executable instructions to:
- receive the different types of sensor data from the plurality of sensors;
- and
- determine output associated with the neural network.
A method implemented by a display system of one or more processors, the display system comprising a plurality of sensors configured to capture different types of sensor data, and wherein the method comprises:
- receiving sensor data from the plurality of sensors;
- determining, based on the sensor data, output associated with a neural network, wherein the neural network comprises:
- an input layer for receiving input of the neural network,
- a plurality of intermediate layers including a first intermediate layer of the plurality of intermediate layers coupled to the input layer, and
- a plurality of head components including a head output node coupled to a last intermediate layer of the plurality of intermediate layers through a plurality of head component layers;
- and
- causing display, via the display system, of at least a portion of the determined output.
Non-transitory computer storage media storing instructions for execution by a display of one or more processors, the display system comprising a plurality of sensors configured to capture different types of sensor data, and wherein the instructions cause the one or more processors to perform operations comprising:
- receiving sensor data from the plurality of sensors;
- determining, based on the sensor data, output associated with a neural network, wherein the neural network comprises:
- an input layer for receiving input of the neural network,
- a plurality of intermediate layers including a first intermediate layer of the plurality of intermediate layers coupled to the input layer, and
- a plurality of head components including a head output node coupled to a last intermediate layer of the plurality of intermediate layers through a plurality of head component layers;
- and
- causing display, via the display system, of at least a portion of the determined output.
Dependencies are inferred from the claim text.