Fast hand meshing for dynamic occlusion
AssigneeMagic Leap, Inc.
InventorDavid Geoffrey Molyneaux
Life of this patent
Estimating the term…
Abstract
An augmented reality/mixed reality system that provides an immersive user experience. That experience is provided with increased speed of update with low computational resources for occlusion data representing a user's hand that can be used to occlude virtual objects. The hand occlusion data are generated by computing a hand mesh from live depth sensor data. A small amount of data is processed by masking live depth data with a model in which a hand is simply represented by multiple segments identified from key-points. The hand occlusion data may be generated by predicting changes in a hand pose between a time of capture of the depth data and the time when the hand mesh will be used for occlusion processing. The hand mesh may be distorted to represent the hand in the predicted pose.
Claims21
A method of operating a computing system to reconstruct a hand for dynamically occluding a virtual object, the method comprising:
- receiving a query from an application rendering a virtual object in a scene for data related to a hand in the scene;
- capturing information of the scene from a device worn by a user, the device comprising one or more sensors, the information of the scene comprising depth information indicating distances between the device worn by the user and physical objects in the scene;
- detecting whether the physical objects in the scene comprises a hand;
- when the hand is detected, computing a model of the hand based, at least in part, on the information of the scene;
- masking, with the model of the hand, the depth information indicating the distances between the device worn by the user and the physical objects in the scene;
- computing a hand mesh based on the depth information masked to the model of the hand, the computing comprising updating the hand mesh in real time as relative locations between the device and the hand change;
- and
- supplying the hand mesh to the application such that the application renders portions of the virtual object not occluded by the hand mesh.
An electronic system portable by a user comprising:
- a device worn by the user, wherein the device comprises a display configured to render a virtual object and comprises one or more sensors configured to capture a headpose of the user wearing the device and information of a scene comprising one or more physical objects, the information of the scene comprising depth information indicating distances between the device and the one or more physical objects;
- a hand meshing component configured to execute computer executable instructions to detect a hand in the scene and compute a hand mesh of the detected hand and update the hand mesh in real time as the head pose changes and/or the hand moves;
- and
- an application configured to execute computer executable instructions to render the virtual object in the scene, wherein the application receives, from the hand meshing component, the hand mesh and portions of the virtual object that is occluded by the hand.
A method of operating an AR system to render a virtual object in a scene including physical objects, wherein the AR system comprises at least one sensor and at least one processor, and the method comprises:
- capturing information of a scene with the at least one sensor, the information of the scene comprising depth information indicating distances to physical objects in the scene;
- with the at least one processor:
- processing the captured information to detect a hand in the scene and compute points on the hand;
- selecting, based on proximity to the computed points on the hand, a subset of the depth information;
- and
- computing a representation of the hand based on the selected depth information, wherein the representation of the hand indicates surfaces of the hand.
Dependencies are inferred from the claim text.