Hand-based light estimation for extended reality
AssigneeSnap Inc.
InventorsDavid Mandl, Alexander Plopski
Life of this patent
Estimating the term…
Abstract
Examples described herein relate to hand-based light estimation for extended reality (XR). An image sensor of an XR device is used to obtain an image of a hand in a real-world environment. At least part of the image is processed to detect a pose of the hand. One of a plurality of machine learning models is selected based on the detected pose. At least part of the image is processed via the machine learning model to obtain estimated illumination parameter values associated with the hand. The estimated illumination parameter values are used to render virtual content to be presented by the XR device.
Claims20
A method comprising:
- obtaining, via an image sensor of an extended reality (XR) device, an image of a hand in a real-world environment;
- processing at least part of the image to detect a pose of the hand;
- selecting, based on the detected pose, a machine learning model from among a plurality of machine learning models;
- processing at least part of the image via the machine learning model to obtain estimated illumination parameter values associated with the hand;
- performing, by the XR device, hand tracking to determine at least one of a position or a zone of the hand within the real-world environment;
- applying the estimated illumination parameter values as a local light probe associated with the at least one of the position or the zone of the hand within the real-world environment;
- and
- using the estimated illumination parameter values to render virtual content to be presented by the XR device.
A system comprising:
- at least one processor;
- and
- at least one memory storing instructions that, when executed by the at least one processor, configure the system to perform operations comprising:
- obtaining, via an image sensor of an extended reality (XR) device, an image of a hand in a real-world environment;
- processing at least part of the image to detect a pose of the hand;
- selecting, based on the detected pose, a machine learning model from among a plurality of machine learning models;
- processing at least part of the image via the machine learning model to obtain estimated illumination parameter values associated with the hand;
- performing, by the XR device, hand tracking to determine at least one of a position or a zone of the hand within the real-world environment;
- applying the estimated illumination parameter values as a local light probe associated with the at least one of the position or the zone of the hand within the real-world environment;
- and
- using the estimated illumination parameter values to render virtual content to be presented by the XR device.
A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by at least one processor, cause the at least one processor to perform operations comprising:
- obtaining, via an image sensor of an extended reality (XR) device, an image of a hand in a real-world environment;
- processing at least part of the image to detect a pose of the hand;
- selecting, based on the detected pose, a machine learning model from among a plurality of machine learning models;
- processing at least part of the image via the machine learning model to obtain estimated illumination parameter values associated with the hand;
- performing, by the XR device, hand tracking to determine at least one of a position or a zone of the hand within the real-world environment;
- applying the estimated illumination parameter values as a local light probe associated with the at least one of the position or the zone of the hand within the real-world environment;
- and
- using the estimated illumination parameter values to render virtual content to be presented by the XR device.
Dependencies are inferred from the claim text.