# US 12,118,200 B1 · Fuzzy hit testing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,200 B1 - Document ID: US12118200B1 - Status: Granted - Kind: B1 - Application number: 18/375,260 - Filed: 2023-09-29 - Granted: 2024-10-15 - Earliest claimed: 2023-06-02 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Julian K Shutzberg; David M Teitelbaum; Chase B Lortie; David J Meyer; Lorenzo Soto Doblado - Web page: https://patentgrep.com/patent/US12118200B1 ## CPC - G06F 3/0487: using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34) - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials - G06F 2203/04802: 3D-info-object: information is displayed on the internal or external surface of a three dimensional manipulable object, e.g. on the faces of a cube that can be rotated by the user - G06F 2203/04804: Transparency, e.g. transparent or translucent windows - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Various implementations disclosed herein include devices, systems, and methods that associate user activities with user interface (UI) elements within a 3D environment. Some implementations do so in ways that account for inaccuracies (i.e., fuzziness) in sensor-based detection of the user activities, e.g., inaccuracy in sensor data-based gaze tracking or sensor data-based hand/joint position. Some implementations use a sampling technique to associate user activity in a 3D environment with an appropriate portion of a UI positioned within a 3D environment. For example, a sampling technique may be used to identify sample locations within the 3D environment (e.g., sample locations around a gaze direction) to evaluate and ultimately select from to associate with a user activity, e.g., associating a particular UI button with a gaze direction. ## Claims ### Claim 1 A method comprising: at an electronic device having a processor: receiving, at an input support process, data corresponding to user activity in a 3D coordinate system corresponding to a 3D environment in which a plurality of user interface (UI) targets of a user interface are positioned; generating, at the input support process, a plurality of sample locations in the 3D coordinate system based on the data corresponding to the user activity; identifying, at the input support process, a subset of the plurality of UI targets within the 3D coordinate system for evaluation using a UI target selection criterion, the subset identified based on the plurality of sample locations; selecting, at the input support process, a UI target of the identified subset to associate with the user activity based on the evaluation of subset of the plurality of UI targets using the UI target selection criterion; and interpreting, at the input support process, the user activity as input associated with the selected UI target based on the selected UI target being associated with the user activity. ### Claim 2 (depends on 1) The method of claim 1, wherein selecting the UI target to associate with the user activity comprises: for each UI target of the identified subset, determining a point on the respective UI target based on the user activity; and prioritizing the UI targets of the subset based on the point computed for each respective UI target. ### Claim 3 (depends on 2) The method of claim 2, wherein determining the point on each of the respective UI targets comprises: determining a closest opaque point to a sample location associated with the user activity; and determining a distance of the closest opaque point of each of the respective UI targets to the sample location associated with the user activity. ### Claim 4 (depends on 2) The method of claim 2, wherein selecting the UI target to associate with the user activity is based on determining that a closest opaque point within the UI target is within an angular distance threshold of a sample location associated with the user activity. ### Claim 5 (depends on 2) The method of claim 2, wherein the UI target to associate with the user activity is selected based on: determining that closest opaque points within multiple UI targets are within an angular distance threshold of a sample location associated with the user activity; and selecting the UI target from the multiple UI targets based on a policy that ranks UI targets based on element type, UI layers, UI geometry, or hysteresis logic. ### Claim 6 (depends on 1) The method of claim 1, wherein the UI targets of the subset comprise: 3D virtual objects; and 2D elements defined by one or more applications. ### Claim 7 (depends on 1) The method of claim 1, wherein identifying the subset comprises: receiving, at the input support process, data corresponding to positioning of UI elements of an application within the 3D coordinate system, the data corresponding to the positioning of the UI element based at least in part on data provided by the application; and identifying the subset by identifying intersections of the plurality of gaze sample locations with the UI elements of the application positioned within the 3D coordinate system. ### Claim 8 (depends on 7) The method of claim 7, wherein the UI elements of the application occupy a two-dimensional (2D) region and the method further comprises, based on selecting the UI target to associate with the user activity, identifying a point within the 2D region to the application such that the application can recognize an action to associate with the UI element using a 2D app action recognition process. ### Claim 9 (depends on 7) The method of claim 7, wherein the data provided by the application comprises a layered tree structure defining the positional and containment relationships of the UI elements relative to one another on a two-dimensional (2D) coordinate system. ### Claim 10 (depends on 7) The method of claim 7, wherein the data provided by the application identifies external effects for some of the UI elements, wherein an external effect specifies that an operating system (OS) process is to provide responses to a specified user activity relative to a specified UI element outside of an application process. ### Claim 11 (depends on 1) The method of claim 1, wherein the data corresponding to the user activity is a gaze direction within the 3D coordinate system, the gaze direction determined based on sensor data. ### Claim 12 (depends on 1) The method of claim 1, wherein the data corresponding to the user activity is a synthesized direction within the 3D coordinate system, the direction determined based on: determining a hand position of a hand in the 3D coordinate system based on sensor data; determining an intersection position of the hand with at least one UI element based on the hand position; and determining the direction based on the intersection and a viewpoint position. ### Claim 13 (depends on 1) The method of claim 1, wherein the plurality of sample locations are generated by generating a pattern of rays around a gaze direction or a synthesized direction corresponding to user activity. ### Claim 14 (depends on 13) The method of claim 13, wherein the pattern of rays has between 2 and 100 rays. ### Claim 15 (depends on 13) The method of claim 13, wherein the pattern of rays has between 5 and 35 rays. ### Claim 16 (depends on 13) The method of claim 13, wherein the pattern of rays comprises an outer set of rays forming a shape. ### Claim 17 (depends on 16) The method of claim 16, wherein the shape is rotated relative to a horizon or a horizontal. ### Claim 18 (depends on 1) The method of claim 1, wherein the electronic device provides views of a 3D environment including the UI targets, wherein at least some of the UI targets are 2D user interface elements provided by one or more applications, wherein the input support process recognizes the user activity in the 3D coordinate system and provides data to the one or more applications to recognize 2D user interface input. ### Claim 19 A system comprising: memory; and one or more processors coupled to the memory, wherein the memory comprises program instructions that, when executed by the one or more processors, cause the system to perform operations comprising: receiving, at an input support process, data corresponding to user activity in a 3D coordinate system corresponding to a 3D environment in which a plurality of user interface (UI) targets of a user interface are positioned; generating, at the input support process, a plurality of sample locations in the 3D coordinate system based on the data corresponding to the user activity; identifying, at the input support process, a subset of the plurality of (UI) targets within the 3D coordinate system for evaluation using a UI target selection criterion, the subset identified based on the plurality of sample locations; selecting, at the input support process, a UI target of the identified subset to associate with the user activity based on the evaluation of subset of the plurality of UI targets using the UI target selection criterion; and interpreting, at the input support process, the user activity as input associated with the selected UI target based on the selected UI target being associated with the user activity. ### Claim 20 (depends on 19) The system of claim 19, wherein selecting the UI target to associate with the user activity comprises: for each UI target of the identified subset, determining a point on the respective UI target based on the user activity; and prioritizing the UI targets of the subset based on the point computed for each respective UI target. ### Claim 21 (depends on 20) The system of claim 20, wherein determining the point on each of the respective UI targets comprises: determining a closest opaque point to a sample location associated with the user activity; and determining a distance of the closest opaque point of each of the respective to the sample location associated with the user activity. ### Claim 22 (depends on 20) The system of claim 20, wherein selecting the UI target to associate with the user activity is based on determining that a closest opaque point within the UI target is within an angular distance threshold of a sample location associated with the user activity. ### Claim 23 (depends on 20) The system of claim 20, wherein the UI target to associate with the user activity is selected based on: determining that closest opaque points within multiple UI targets are within an angular distance threshold of a sample location associated with the user activity; and selecting the UI target from the multiple UI targets based on a policy that ranks UI targets based on element type, UI layers, UI geometry, or hysteresis logic. ### Claim 24 (depends on 19) The system of claim 19, wherein the identified subset UI comprise: 3D virtual objects; and 2D elements defined by one or more applications. ### Claim 25 A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising: receiving, at an input support process, data corresponding to user activity in a 3D coordinate system corresponding to a 3D environment in which a plurality of user interface (UI) targets of a user interface are positioned; generating, at the input support process, a plurality of sample locations in the 3D coordinate system based on the data corresponding to the user activity; identifying, at the input support process, a subset of the plurality of UI targets within the 3D coordinate system for evaluation using a UI target selection criterion, the subset identified based on the plurality of sample locations; selecting, at the input support process, a UI target of the identified subset to associate with the user activity based on the evaluation of subset of the plurality of UI targets using the UI target selection criterion; and interpreting, at the input support process, the user activity as input associated with the selected UI target based on the selected UI target being associated with the user activity.