Back to results
US 12,613,573 B2Granted
Determining body pose from environmental data
AssigneeSamsung Electronics Co., Ltd.
InventorsShijia Zhang, Wenjun Jiang, Xun Chen
Life of this patent
Estimating the term…
Abstract
In one embodiment, a method includes accessing video captured by one or more cameras of a head-mounted device (HMD) worn by a user and determining, from the accessed video, multiple environmental feature maps and multiple corresponding environmental embeddings representing an environment of the user captured in the accessed video. The method further includes fusing, by multiple trained environmental transformer models, the environmental embeddings to create a fused environmental embedding; determining a head pose of the user coincident with the accessed video; and predicting, by a trained decoder and based on the fused environmental embedding and the determined head pose of the user, a body pose of the user.
Claims20
A method comprising:
- accessing egocentric video captured by one or more cameras of a head-mounted device (HMD) worn by a user;
- determining, from the accessed video, a plurality of environmental feature maps and a corresponding plurality of environmental embeddings representing an environment of the user captured in the accessed video, wherein the plurality of environmental feature maps and the corresponding plurality of environmental embeddings comprise one or more of:
- a static-scene feature map and corresponding static-scene embedding obtained from an image of the video;
- a dynamic-scene feature map and corresponding dynamic-scene embedding representing changes among images in a sequence of images of the video;
- or
- an interactee feature map and corresponding interactee embedding obtained from an image of another user in the video;
- fusing, by a plurality of trained environmental transformer models, the plurality of environmental embeddings to create a fused environmental embedding;
- determining a head pose of the user coincident with the accessed video;
- and
- predicting, by a trained decoder and based on the fused environmental embedding and the determined head pose of the user, a body pose of the user from the egocentric video.
7 more elements
One or more non-transitory computer readable storage media storing instructions that are operable when executed to:
- access egocentric video captured by one or more cameras of a head-mounted device (HMD) worn by a user;
- determine, from the accessed video, a plurality of environmental feature maps and a corresponding plurality of environmental embeddings representing an environment of the user captured in the accessed video, wherein the plurality of environmental feature maps and the corresponding plurality of environmental embeddings comprise one or more of:
- a static-scene feature map and corresponding static-scene embedding obtained from an image of the video;
- a dynamic-scene feature map and corresponding dynamic-scene embedding representing changes among images in a sequence of images of the video;
- or
- an interactee feature map and corresponding interactee embedding obtained from an image of another user in the video;
- fuse, by a plurality of trained environmental transformer models, the plurality of environmental embeddings to create a fused environmental embedding;
- determine a head pose of the user coincident with the accessed video;
- and
- predict, by a trained decoder and based on the fused environmental embedding and the determined head pose of the user, a body pose of the user from the egocentric video.
7 more elements
An apparatus comprising:
- a head-mounted device (HMD) configured to be worn by a user and comprising one or more cameras configured to capture video of an environment of the user;
- and
- a computing device comprising one or more non-transitory computer readable storage media storing instructions;
- and one or more processors coupled to the one or more non-transitory computer readable storage media and operable to execute the instructions to:
- access egocentric video captured by the one or more cameras of the (HMD);
- determine, from the accessed video, a plurality of environmental feature maps and a corresponding plurality of environmental embeddings representing an environment of the user captured in the accessed video, wherein the plurality of environmental feature maps and the corresponding plurality of environmental embeddings comprise one or more of:
- a static-scene feature map and corresponding static-scene embedding obtained from an image of the video;
- a dynamic-scene feature map and corresponding dynamic-scene embedding representing changes among images in a sequence of images of the video;
- or
- an interactee feature map and corresponding interactee embedding obtained from an image of another user in the video;
- fuse, by a plurality of trained environmental transformer models, the plurality of environmental embeddings to create a fused environmental embedding;
- determine a head pose of the user coincident with the accessed video;
- and
- predict, by a trained decoder and based on the fused environmental embedding and the determined head pose of the user, a body pose of the user from the egocentric video.
11 more elements
Dependencies are inferred from the claim text.