# US 12,216,815 B2 · Systems and methods for predicting lower body poses > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,216,815 B2 - Document ID: US12216815B2 - Status: Granted - Kind: B2 - Application number: 18/454,698 - Filed: 2023-08-23 - Published: 2023-12-14 - Granted: 2025-02-04 - Earliest claimed: 2020-09-17 - Assignees: Meta Platforms Technologies, LLC - Applicants: Meta Platforms Technologies, LLC - Inventors: Benjamin Antoine Georges Lefaudeux; Samuel Alan Johnson; Carsten Sebastian Stoll; Kishore Venkateshan - Web page: https://patentgrep.com/patent/US12216815B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06N 5/04: Inference or reasoning models - G06N 20/00: Machine learning - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 2207/20081: Training; Learning - G06T 2207/30196: Human being; Person - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A computing system may receive sensor data from one or more sensors coupled to a user. Based on sensor data, the computing system may generate an upper body pose that corresponds to a first portion of a body of the user, which comprises a head and an arm of the user. The computing system may determine contextual information associated with the user. The computing system may generate a lower body pose corresponding to a second portion of the body of the user comprising a leg of the user based on the upper body pose and the contextual information. The computing system may generate a full body pose of the user based on the first upper body pose and the lower body pose. ## Claims ### Claim 1 A method comprising, by a computing system: receiving sensor data captured by one or more sensors coupled to a user; generating, based on the sensor data, an upper body pose that corresponds to a first portion of a body of the user, the first portion of the body comprising a head and an arm of the user; determining contextual information associated with the user; generating, based on the upper body pose and the contextual information, a lower body pose that corresponds to a second portion of the body of the user, the second portion of the body comprising a leg of the user; and generating a full body pose of the user based on the upper body pose and the lower body pose. ### Claim 2 (depends on 1) The method of claim 1, wherein generating the lower body pose is further based on a machine-learning model, and wherein the machine-learning model is trained using a second machine-learning model trained to determine whether a given full body pose is likely generated using the machine-learning model. ### Claim 3 (depends on 2) The method of claim 2, wherein the machine-learning model is optimized during training to cause a second machine-learning model to incorrectly determine that a given full body pose, generated using the machine-learning model, is unlikely generated using the machine-learning model. ### Claim 4 (depends on 1) The method of claim 1, wherein the one or more sensors are associated with a head-mounted device worn by the user. ### Claim 5 (depends on 1) The method of claim 1, wherein generating the upper body pose comprises: determining, based on the sensor data, a plurality of poses corresponding to a plurality of predetermined body parts of the user; and inferring the upper body pose based on the plurality of poses. ### Claim 6 (depends on 5) The method of claim 5, wherein the upper body pose further comprises a wrist pose corresponding to a wrist of the user, wherein the wrist pose is determined based on one or more images captured by a head-mounted device worn by the user, the one or more images depicting (1) the wrist of the user or (2) a device held by the user. ### Claim 7 (depends on 1) The method of claim 1, wherein generating the lower body pose comprises processing the upper body pose and the contextual information using a machine-learning model. ### Claim 8 (depends on 7) The method of claim 7, wherein the contextual information comprises an application the user is interacting with when the upper body pose is determined. ### Claim 9 (depends on 7) The method of claim 7, wherein the contextual information indicates the user is standing or sitting at a first time when the sensor data was captured. ### Claim 10 (depends on 1) The method of claim 1, further comprising generating, based on the full body pose, an avatar of the user. ### Claim 11 (depends on 1) The method of claim 1, wherein the lower body pose is generated by further processing one or more physical constraints associated with the upper body pose using a machine-learning model. ### Claim 12 (depends on 11) The method of claim 11, wherein the one or more physical constraints comprise a physical limitation on a range of motion of one or more joints of the user. ### Claim 13 One or more computer-readable non-transitory non-volatile storage media comprising software that is operable when executed by a server to: receive sensor data captured by one or more sensors coupled to a user; generate, based on the sensor data, an upper body pose that corresponds to a first portion of a body of the user, the first portion of the body comprising a head and an arm of the user; determine contextual information associated with the user; generate, based on the upper body pose and the contextual information, a lower body pose that corresponds to a second portion of the body of the user, the second portion of the body comprising a leg of the user; and generate a full body pose of the user based on the upper body pose and the lower body pose. ### Claim 14 (depends on 13) The media of claim 13, wherein generating the upper body pose comprises: determining, based on the sensor data, a plurality of poses corresponding to a plurality of predetermined body parts of the user; and inferring the upper body pose based on the plurality of poses. ### Claim 15 (depends on 13) The media of claim 13, wherein generating the lower body pose comprises processing the upper body pose and the contextual information using a machine-learning model. ### Claim 16 (depends on 15) The media of claim 15, wherein the contextual information comprises an application the user is interacting with when the upper body pose is determined. ### Claim 17 (depends on 13) The media of claim 13, wherein the software is further operable when executed to generate, based on the full body pose, an avatar of the user. ### Claim 18 (depends on 13) The media of claim 13, wherein the lower body pose is generated by further processing one or more physical constraints associated with the upper body pose using a machine-learning model. ### Claim 19 A system comprising: one or more processors; and one or more computer-readable non-transitory non-volatile storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to: receive sensor data captured by one or more sensors coupled to a user; generate, based on the sensor data, an upper body pose that corresponds to a first portion of a body of the user, the first portion of the body comprising a head and an arm of the user; determine contextual information associated with the user; generate, based on the upper body pose and the contextual information, a lower body pose that corresponds to a second portion of the body of the user, the second portion of the body comprising a leg of the user; and generate a full body pose of the user based on the upper body pose and the lower body pose. ### Claim 20 (depends on 19) The system of claim 19, wherein generating the upper body pose comprises: determining, based on the sensor data, a plurality of poses corresponding to a plurality of predetermined body parts of the user; and inferring the upper body pose based on the plurality of poses.