# US 12,602,863 B2 · Hand-based light estimation for extended reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,863 B2 - Document ID: US12602863B2 - Status: Granted - Kind: B2 - Application number: 18/604,321 - Filed: 2024-03-13 - Published: 2025-09-18 - Granted: 2026-04-14 - Earliest claimed: 2024-03-13 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: David Mandl; Alexander Plopski - Web page: https://patentgrep.com/patent/US12602863B2 ## CPC - G06T 15/506: {Illumination models} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06T 7/11: Region-based segmentation - G06T 7/73: using feature-based methods - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) - G06V 10/82: using neural networks - G06V 10/87: {using selection of the recognition techniques, e.g. of a classifier in a multiple classifier system} - G06V 40/11: {Hand-related biometrics; Hand pose recognition} - G06V 40/113: {Recognition of static hand signs} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30196: Human being; Person - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Examples described herein relate to hand-based light estimation for extended reality (XR). An image sensor of an XR device is used to obtain an image of a hand in a real-world environment. At least part of the image is processed to detect a pose of the hand. One of a plurality of machine learning models is selected based on the detected pose. At least part of the image is processed via the machine learning model to obtain estimated illumination parameter values associated with the hand. The estimated illumination parameter values are used to render virtual content to be presented by the XR device. ## Claims ### Claim 1 A method comprising: obtaining, via an image sensor of an extended reality (XR) device, an image of a hand in a real-world environment; processing at least part of the image to detect a pose of the hand; selecting, based on the detected pose, a machine learning model from among a plurality of machine learning models; processing at least part of the image via the machine learning model to obtain estimated illumination parameter values associated with the hand; performing, by the XR device, hand tracking to determine at least one of a position or a zone of the hand within the real-world environment; applying the estimated illumination parameter values as a local light probe associated with the at least one of the position or the zone of the hand within the real-world environment; and using the estimated illumination parameter values to render virtual content to be presented by the XR device. ### Claim 2 (depends on 1) The method of claim 1, wherein each of the plurality of machine learning models is uniquely associated with a respective hand gesture of a plurality of hand gestures, and the selection of the machine learning model based on the detected pose comprises: identifying, from among the plurality of hand gestures, a hand gesture corresponding to the detected pose; and selecting the machine learning model that is associated with the hand gesture. ### Claim 3 (depends on 2) The method of claim 2, wherein each of the plurality of machine learning models is trained to estimate illumination parameter values from images depicting the respective hand gesture associated with the machine learning model. ### Claim 4 (depends on 1) The method of claim 1, wherein the local light probe is stored to supplement at least one other set of illumination parameter values that is not specific to the hand. ### Claim 5 (depends on 1) The method of claim 1, wherein the XR device is a head-mounted XR device, and the hand is a hand of a user of the XR device. ### Claim 6 (depends on 1) The method of claim 1, wherein the estimated illumination parameter values include values indicative of incident illumination on the hand. ### Claim 7 (depends on 1) The method of claim 1, wherein the estimated illumination parameter values include values for a set of spherical harmonics coefficients representing one or more of intensity or direction of light. ### Claim 8 (depends on 1) The method of claim 1, wherein the using of the estimated illumination parameter values to render the virtual content comprises using the estimated illumination parameter values to generate an irradiance map. ### Claim 9 (depends on 1) The method of claim 1, wherein the virtual content comprises a virtual object, and the rendering of the virtual object comprises: generating a three-dimensional model of the virtual object, the three-dimensional model having predetermined properties; and using the estimated illumination parameter values and the predetermined properties of the three-dimensional model to obtain rendering characteristics for the virtual object. ### Claim 10 (depends on 1) The method of claim 1, wherein the image is captured by the image sensor and includes the hand and other features in the real-world environment, the method further comprising: performing segmentation to separate the hand from the other features, wherein the at least part of the image that is processed via the machine learning model includes the hand and excludes the other features. ### Claim 11 (depends on 1) The method of claim 1, wherein model parameters of the machine learning model are stored locally on the XR device, and the processing of the at least part of the image via the machine learning model is performed at the XR device. ### Claim 12 (depends on 1) The method of claim 1, the machine learning model comprises a Convolutional Neural Network (CNN). ### Claim 13 (depends on 1) The method of claim 1, further comprising: causing presentation of the virtual content, via a display arrangement of the XR device, as overlaid on the real-world environment. ### Claim 14 (depends on 1) The method of claim 1, wherein each of the plurality of machine learning models is trained on a respective training dataset comprising training images of hands in varying lighting conditions. ### Claim 15 (depends on 14) The method of claim 14, wherein each of the plurality of machine learning models is uniquely associated with a respective hand gesture of a plurality of hand gestures, and wherein the training dataset of each of the plurality of machine learning models comprises training images of hands depicting the respective hand gesture associated with the machine learning model. ### Claim 16 (depends on 14) The method of claim 14, wherein each of the plurality of machine learning models is trained using supervised learning, and each respective training dataset further comprises, for each training image, corresponding illumination parameter values representing lighting conditions associated with a hand depicted in the training image. ### Claim 17 (depends on 14) The method of claim 14, wherein at least a subset of the training images are synthetic images generated using a graphics engine that simulates the varying lighting conditions. ### Claim 18 A system comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, configure the system to perform operations comprising: obtaining, via an image sensor of an extended reality (XR) device, an image of a hand in a real-world environment; processing at least part of the image to detect a pose of the hand; selecting, based on the detected pose, a machine learning model from among a plurality of machine learning models; processing at least part of the image via the machine learning model to obtain estimated illumination parameter values associated with the hand; performing, by the XR device, hand tracking to determine at least one of a position or a zone of the hand within the real-world environment; applying the estimated illumination parameter values as a local light probe associated with the at least one of the position or the zone of the hand within the real-world environment; and using the estimated illumination parameter values to render virtual content to be presented by the XR device. ### Claim 19 A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by at least one processor, cause the at least one processor to perform operations comprising: obtaining, via an image sensor of an extended reality (XR) device, an image of a hand in a real-world environment; processing at least part of the image to detect a pose of the hand; selecting, based on the detected pose, a machine learning model from among a plurality of machine learning models; processing at least part of the image via the machine learning model to obtain estimated illumination parameter values associated with the hand; performing, by the XR device, hand tracking to determine at least one of a position or a zone of the hand within the real-world environment; applying the estimated illumination parameter values as a local light probe associated with the at least one of the position or the zone of the hand within the real-world environment; and using the estimated illumination parameter values to render virtual content to be presented by the XR device. ### Claim 20 (depends on 18) The system of claim 18, wherein the estimated illumination parameter values include values for a set of spherical harmonics coefficients representing one or more of intensity or direction of light.