Generating and modifying representations of hands in an artificial reality environment

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

• receive an image of a real environment captured using a camera worn by a user, the image comprising a hand of the user;
• determine a pose of the hand based on the image and a light direction of a light source;
• generate, based on a three-dimensional model of the hand having the determined pose, a first two-dimensional surface representing the hand as viewed from a first viewpoint of the user;
• project, according to the light direction, the three-dimensional model of the hand having the determined pose onto one or more virtual objects;
• generate a second two-dimensional surface representing a shadow of the hand on the one or more virtual objects as viewed from the first viewpoint of the user;
• position the first two-dimensional surface representing the hand, the second two-dimensional surface representing the shadow of the hand on the one or more virtual objects, and one or more virtual-object representations of the one or more virtual objects in a three-dimensional space;
• determine that a first portion of the first two-dimensional surface representing the hand and a second portion of the second two-dimensional surface representing the shadow of the hand are visible from a second viewpoint in the three-dimensional space;
• and
• generate an output image, wherein (1) a first set of image pixels of the output image corresponding to the portion of the first portion of the first two-dimensional surface that is visible is configured to cause a display to turn off a set of corresponding display pixels, and (2) a second set of image pixels of the output image is generated using the second portion of the second two-dimensional surface and one or more associated transparency levels.

A system comprising:

• one or more processors;
• and
• one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
• receive an image of a real environment captured using a camera worn by a user, the image comprising a hand of the user;
• determine a pose of the hand based on the image and a light direction of a light source;
• generate, based on a three-dimensional model of the hand having the determined pose, a first two-dimensional surface representing the hand as viewed from a first viewpoint of the user;
• project, according to the light direction, the three-dimensional model of the hand having the determined pose onto one or more virtual objects;
• generate a second two-dimensional surface representing a shadow of the hand on the one or more virtual objects as viewed from the first viewpoint of the user;
• position the first two-dimensional surface representing the hand, the second two-dimensional surface representing the shadow of the hand on the one or more virtual objects, and one or more virtual-object representations of the one or more virtual objects in a three-dimensional space;
• determine that a first portion of the first two-dimensional surface representing the hand and a second portion of the second two-dimensional surface representing the shadow of the hand are visible from a second viewpoint in the three-dimensional space;
• and
• generate an output image, wherein (1) a first set of image pixels of the output image corresponding to the portion of the first portion of the first two-dimensional surface that is visible is configured to cause a display to turn off a set of corresponding display pixels, and (2) a second set of image pixels of the output image is generated using the second portion of the second two-dimensional surface and one or more associated transparency levels.