US 2025/0165060 A1 · Determining Body Pose From Environmental Data

Abstract

In one embodiment, a method includes accessing video captured by one or more cameras of a head-mounted device (HMD) worn by a user and determining, from the accessed video, multiple environmental feature maps and multiple corresponding environmental embeddings representing an environment of the user captured in the accessed video. The method further includes fusing, by multiple trained environmental transformer models, the environmental embeddings to create a fused environmental embedding; determining a head pose of the user coincident with the accessed video; and predicting, by a trained decoder and based on the fused environmental embedding and the determined head pose of the user, a body pose of the user.

Claims20

Dependent claims

A method comprising:

accessing video captured by one or more cameras of a head-mounted device (HMD) worn by a user;
determining, from the accessed video, a plurality of environmental feature maps and a corresponding plurality of environmental embeddings representing an environment of the user captured in the accessed video;
fusing, by a plurality of trained environmental transformer models, the plurality of environmental embeddings to create a fused environmental embedding;
determining a head pose of the user coincident with the accessed video;
and
predicting, by a trained decoder and based on the fused environmental embedding and the determined head pose of the user, a body pose of the user.

3 more elements

One or more non-transitory computer readable storage media storing instructions that are operable when executed to:

access video captured by one or more cameras of a head-mounted device (HMD) worn by a user;
determine, from the accessed video, a plurality of environmental feature maps and a corresponding plurality of environmental embeddings representing an environment of the user captured in the accessed video;
fuse, by a plurality of trained environmental transformer models, the plurality of environmental embeddings to create a fused environmental embedding;
determine a head pose of the user coincident with the accessed video;
and
predict, by a trained decoder and based on the fused environmental embedding and the determined head pose of the user, a body pose of the user.

3 more elements

An apparatus comprising:

a head-mounted device (HMD) configured to be worn by a user and comprising one or more cameras configured to capture video of an environment of the user;
and
a computing device comprising one or more non-transitory computer readable storage media storing instructions;
and one or more processors coupled to the one or more non-transitory computer readable storage media and operable to execute the instructions to:
access video captured by the one or more cameras of the (HMD);
determine, from the accessed video, a plurality of environmental feature maps and a corresponding plurality of environmental embeddings representing an environment of the user captured in the accessed video;
fuse, by a plurality of trained environmental transformer models, the plurality of environmental embeddings to create a fused environmental embedding;
determine a head pose of the user coincident with the accessed video;
and
predict, by a trained decoder and based on the fused environmental embedding and the determined head pose of the user, a body pose of the user.

7 more elements

Dependencies are inferred from the claim text.

Determining Body Pose From Environmental Data

Life of this patent

Determining Body Pose From Environmental Data

Life of this patent

Abstract

Claims20