# US 12,223,117 B1 · Spatiotemporal smoothing for improved hand tracking > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,117 B1 - Document ID: US12223117B1 - Status: Granted - Kind: B1 - Application number: 18/371,111 - Filed: 2023-09-21 - Granted: 2025-02-11 - Earliest claimed: 2022-09-23 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Emmanuel Piuze-Phaneuf; Ali Ercan; Julian K. Shutzberg; Paul A. Lacey - Web page: https://patentgrep.com/patent/US12223117B1 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06T 5/70: Denoising; Smoothing - G06T 7/73: using feature-based methods - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30196: Human being; Person - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract In some implementations, a method includes: obtaining uncorrected hand tracking data; obtaining a depth map associated with a physical environment; identifying a position of a portion of the finger within the physical environment based on the depth map and the uncorrected hand tracking data; performing spatial depth smoothing on a region of the depth map adjacent to the position of the portion of the finger; and generating corrected hand tracking data by performing point of view (POV) correction on the uncorrected hand tracking data based on the spatially depth smoothed region of the depth map adjacent to the portion of the finger. ## Claims ### Claim 1 A method comprising: at a computing system including non-transitory memory and one or more processors, wherein the computing system is communicatively coupled to a display device and one or more input devices via a communication interface: obtaining uncorrected hand tracking data; obtaining a depth map associated with a physical environment; identifying a position of a portion of the finger within the physical environment based on the depth map and the uncorrected hand tracking data; performing spatial depth smoothing on a region of the depth map adjacent to the position of the portion of the finger; and generating corrected hand tracking data by performing point of view (POV) correction on the uncorrected hand tracking data based on the spatially depth smoothed region of the depth map adjacent to the portion of the finger. ### Claim 2 (depends on 1) The method of claim 1, wherein the portion of the finger corresponds to one of a fingertip, a particular knuckle, or a centroid of a finger. ### Claim 3 (depends on 1) The method of claim 1, wherein the position of the portion of the finger is identified by projecting the portion of a finger into a depth space associated with the depth map. ### Claim 4 (depends on 1) The method of claim 1, wherein the region of the depth map adjacent to the portion of the finger corresponds to an N×M pixel area centered on a projection of the portion of the finger. ### Claim 5 (depends on 1) The method of claim 1, wherein the region of the depth map to the portion of the finger corresponds to a predefined radius centered on a projection of the portion of the finger. ### Claim 6 (depends on 1) The method of claim 1, wherein performing POV correction on the uncorrected hand tracking data based on the spatially depth smoothed region adjacent to the portion of the finger includes performing POV correction on each joint within the uncorrected hand tracking data based on the spatially depth smoothed region adjacent to the portion of the finger. ### Claim 7 (depends on 1) The method of claim 1, wherein performing POV correction on the uncorrected hand tracking data based on the spatially depth smoothed region adjacent to the portion of the finger includes: obtaining a first set of two-dimensional coordinates of a hand of the user in the physical environment; and transforming the first set of two-dimensional coordinates into a second set of two-dimensional coordinates based on the spatially depth smoothed region adjacent to the portion of the finger. ### Claim 8 (depends on 7) The method of claim 7, wherein the first set of two-dimensional coordinates are generated by projecting first three-dimensional coordinates onto an image plane. ### Claim 9 (depends on 7) The method of claim 7, wherein the first set of two-dimensional coordinates includes a left set and a right set used for triangulation. ### Claim 10 (depends on 7) The method of claim 7, further comprising: presenting, via the display device, hand locations at the second set of two-dimensional coordinates. ### Claim 11 (depends on 7) The method of claim 7, wherein a second set of three-dimensional coordinates is generated from the second set of two-dimensional coordinates. ### Claim 12 (depends on 11) The method of claim 11, further comprising: determining a user interaction based on the hand location at the second set of three-dimensional coordinates interacting with a virtual object at the three-dimensional coordinates. ### Claim 13 (depends on 1) The method of claim 1, further comprising: rendering a user interaction with a virtual object based on the corrected hand tracking data; and presenting, via the display device, the rendered user interaction with the virtual object based on the corrected hand tracking data. ### Claim 14 A computing system comprising: one or more processors; a non-transitory memory; an interface for communicating with a display device and one or more input devices; and one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the computing system to: obtain uncorrected hand tracking data; obtain a depth map associated with a physical environment; identify a position of a portion of the finger within the physical environment based on the depth map and the uncorrected hand tracking data; perform spatial depth smoothing on a region of the depth map adjacent to the position of the portion of the finger; and generate corrected hand tracking data by performing point of view (POV) correction on the uncorrected hand tracking data based on the spatially depth smoothed region of the depth map adjacent to the portion of the finger. ### Claim 15 (depends on 14) The computing system of claim 14, wherein the region of the depth map adjacent to the portion of the finger corresponds to an N×M pixel area centered on a projection of the portion of the finger. ### Claim 16 (depends on 14) The computing system of claim 14, wherein the region of the depth map to the portion of the finger corresponds to a predefined radius centered on a projection of the portion of the finger. ### Claim 17 (depends on 14) The computing system of claim 14, wherein performing POV correction on the uncorrected hand tracking data based on the spatially depth smoothed region adjacent to the portion of the finger includes performing POV correction on each joint within the uncorrected hand tracking data based on the spatially depth smoothed region adjacent to the portion of the finger. ### Claim 18 (depends on 14) The computing system of claim 14, wherein the one or more programs further cause the computing system to: render a user interaction with a virtual object based on the corrected hand tracking data; and present, via the display device, the rendered user interaction with the virtual object based on the corrected hand tracking data. ### Claim 19 A non-transitory memory storing one or more programs, which, when executed by one or more processors of a computing system with an interface for communicating with a display device and one or more input devices, cause the computing system to: obtain uncorrected hand tracking data; obtain a depth map associated with a physical environment; identify a position of a portion of the finger within the physical environment based on the depth map and the uncorrected hand tracking data; perform spatial depth smoothing on a region of the depth map adjacent to the position of the portion of the finger; and generate corrected hand tracking data by performing point of view (POV) correction on the uncorrected hand tracking data based on the spatially depth smoothed region of the depth map adjacent to the portion of the finger. ### Claim 20 (depends on 19) The non-transitory memory of claim 19, wherein the region of the depth map adjacent to the portion of the finger corresponds to an N×M pixel area centered on a projection of the portion of the finger. ### Claim 21 (depends on 19) The non-transitory memory of claim 19, wherein the region of the depth map to the portion of the finger corresponds to a predefined radius centered on a projection of the portion of the finger. ### Claim 22 (depends on 19) The non-transitory memory of claim 19, wherein performing POV correction on the uncorrected hand tracking data based on the spatially depth smoothed region adjacent to the portion of the finger includes performing POV correction on each joint within the uncorrected hand tracking data based on the spatially depth smoothed region adjacent to the portion of the finger. ### Claim 23 (depends on 19) The non-transitory memory of claim 19, wherein the one or more programs further cause the computing system to: render a user interaction with a virtual object based on the corrected hand tracking data; and present, via the display device, the rendered user interaction with the virtual object based on the corrected hand tracking data.