Hand-based light estimation for extended reality
AssigneeSnap Inc.
InventorsDavid Mandl, Alexander Plopski
Life of this patent
Estimating the term…
Abstract
Examples described herein relate to hand-based light estimation for extended reality (XR). An image sensor of an XR device is used to obtain an image of a hand in a real-world environment. At least part of the image is processed to detect a pose of the hand. One of a plurality of machine learning models is selected based on the detected pose. At least part of the image is processed via the machine learning model to obtain estimated illumination parameter values associated with the hand. The estimated illumination parameter values are used to render virtual content to be presented by the XR device.
Claims20
A method comprising:
- detecting a pose of a hand in an image obtained via an image sensor of an extended reality (XR) device;
- selecting, based on the detected pose, a machine learning model from among a plurality of machine learning models;
- processing at least part of the image via the machine learning model to determine spherical harmonics data associated with the hand;
- and
- applying the spherical harmonics data in a light estimation associated with at least one of a position or a zone of the hand within a real-world environment.
A system comprising:
- at least one processor;
- and
- at least one memory storing instructions that, when executed by the at least one processor, configure the system to perform operations comprising:
- detecting a pose of a hand in an image obtained via an image sensor of an extended reality (XR) device;
- selecting, based on the detected pose, a machine learning model from among a plurality of machine learning models;
- processing at least part of the image via the machine learning model to determine spherical harmonics data associated with the hand;
- and
- applying the spherical harmonics data in a light estimation associated with at least one of a position or a zone of the hand within a real-world environment.
A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by at least one processor, cause the at least one processor to perform operations comprising:
- detecting a pose of a hand in an image obtained via an image sensor of an extended reality (XR) device;
- selecting, based on the detected pose, a machine learning model from among a plurality of machine learning models;
- processing at least part of the image via the machine learning model to determine spherical harmonics data associated with the hand;
- and
- applying the spherical harmonics data in a light estimation associated with at least one of a position or a zone of the hand within a real-world environment.
Dependencies are inferred from the claim text.