# US 11,308,698 B2 · Using deep learning to determine gaze > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,698 B2 - Document ID: US11308698B2 - Status: Granted - Kind: B2 - Application number: 16/705,221 - Filed: 2019-12-05 - Published: 2021-06-10 - Granted: 2022-04-19 - Earliest claimed: 2019-12-05 - Assignees: Facebook Technologies, LLC - Applicants: Facebook Technologies, LLC - Inventors: Kara Emery; Marina Zannoli; Lei Xiao - Web page: https://patentgrep.com/patent/US11308698B2 ## CPC - G06F 3/012: {Head tracking input arrangements} - G02B 27/017: {Head mounted} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06N 20/00: Machine learning - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract In one embodiment, a computing system may generate and display a virtual reality environment to a user. The computing system may determine a head pose of the user based on headset tracking data associated with a headset worn by the user. The computing system may determine a hand pose of the user based on hand tracking data associated with a device held or worn by a hand of the user. The computing system may access scene information associated with the displayed virtual reality environment. The computing system may determine a predicted focal point of the user within the virtual reality environment by processing the head pose, the hand pose, and the scene information using a machine-learning model. ## Claims ### Claim 1 A method comprising, by a computing system: generating and displaying a rendered environment to a user, wherein the rendered environment is associated with an application that is related to a category; determining a head pose of the user based on headset tracking data associated with a headset worn by the user; determining a hand pose of the user based on hand tracking data; accessing scene information associated with the displayed rendered environment, wherein the scene information comprises the category related to the application; and determining a predicted focal point of the user within the rendered environment by processing the head pose, the hand pose, and the scene information using a machine-learning model. ### Claim 2 (depends on 1) The method of claim 1, further comprising: adjusting an image presented to the user by the computing system based on the predicted focal point of the user within the rendered environment. ### Claim 3 (depends on 1) The method of claim 1, wherein the head pose is determined with respect to the rendered environment. ### Claim 4 (depends on 1) The method of claim 1, wherein the hand pose is determined with respect to one of the rendered environment or the headset worn by the user. ### Claim 5 (depends on 1) The method of claim 1, wherein determining the hand pose of the user comprises: identifying a hand of the user based on one or more cameras coupled to the computing system analyzing a plurality of images comprising the hand of the user. ### Claim 6 (depends on 1) The method of claim 1, wherein the hand tracking data is associated with a device held or worn by a hand of the user. ### Claim 7 (depends on 6) The method of claim 6, further comprising: determining an action performed by the device held or worn by the hand of the user, and wherein determining the predicted focal point further comprises processing the action performed by the device held or worn by the hand of the user using the machine-learning model. ### Claim 8 (depends on 1) The method of claim 1, wherein the scene information includes color and depth data. ### Claim 9 (depends on 1) The method of claim 1, wherein the scene information comprises semantic information of one or more elements within the rendered environment. ### Claim 10 (depends on 1) The method of claim 1, wherein the predicted focal point is a three-dimensional coordinate within the rendered environment. ### Claim 11 (depends on 1) The method of claim 1, further comprising: generating a confidence map of one or more locations for the predicted focal point using the machine-learning model, wherein the confidence map assigns a probability of the predicted focal point being at each of the one or more locations, and wherein determining the predicted focal point comprises utilizing the confidence map. ### Claim 12 (depends on 1) The method of claim 1, wherein the predicted focal point is determined without eye tracking sensors. ### Claim 13 (depends on 1) The method of claim 1, wherein the rendered environment comprises one or more of an augmented reality environment, a virtual reality environment, or a mixed reality environment. ### Claim 14 One or more computer-readable non-transitory storage media embodying software that is operable when executed to: generate and display a rendered environment to a user, wherein the rendered environment is associated with an application that is related to a category; determine a head pose of the user based on headset tracking data associated with a headset worn by the user; determine a hand pose of the user based on hand tracking data associated with a device held or worn by a hand of the user; access scene information associated with the displayed rendered environment, wherein the scene information comprises the category related to the application; and determine a predicted focal point of the user within the rendered environment by processing the head pose, the hand pose, and the scene information using a machine-learning model. ### Claim 15 A system comprising: one or more processors; and a non-transitory memory coupled to the one or more processors comprising instructions executable by the one or more processors, the one or more processors operable when executing the instructions to: generate and display a rendered environment to a user, wherein the rendered environment is associated with an application that is related to a category; determine a head pose of the user based on headset tracking data associated with a headset worn by the user; determine a hand pose of the user based on hand tracking data associated with a device held or worn by a hand of the user; access scene information associated with the displayed rendered environment, wherein the scene information comprises the category related to the application; and determine a predicted focal point of the user within the rendered environment by processing the head pose, the hand pose, and the scene information using a machine-learning model. ### Claim 16 (depends on 15) The system of claim 15, wherein the one or more processors are further operable when executing the instructions to: adjust an image presented to the user by the computing system based on the predicted focal point of the user within the rendered environment. ### Claim 17 (depends on 15) The system of claim 15, wherein the head pose is determined with respect to the rendered environment. ### Claim 18 (depends on 15) The system of claim 15, wherein the hand pose is determined with respect to one of the rendered environment or the headset worn by the user. ### Claim 19 (depends on 15) The system of claim 15, wherein the one or more processors are further operable when executing the instructions to: identify a hand of the user based on one or more cameras coupled to the computing system analyzing a plurality of images comprising the hand of the user. ### Claim 20 (depends on 15) The system of claim 15, wherein the hand tracking data is associated with a device held or worn by a hand of the user.