# US 12,008,208 B2 · Merging computer-generated objects > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,208 B2 - Document ID: US12008208B2 - Status: Granted - Kind: B2 - Application number: 18/121,673 - Filed: 2023-03-15 - Published: 2023-10-12 - Granted: 2024-06-11 - Earliest claimed: 2020-09-16 - Applicants: Apple Inc. - Inventors: Nicolai Georg; Aaron M. Burns; Adam G. Poulos; Arun Rakesh Yoganandan; Benjamin Hylak; Benjamin R. Blachnitzky - Web page: https://patentgrep.com/patent/US12008208B2 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 1/163: {Wearable computers, e.g. on a belt} - G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer} - G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves} - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 3/0486: Drag-and-drop - G06F 2203/0331: Finger worn pointing device - G06F 2203/04802: 3D-info-object: information is displayed on the internal or external surface of a three dimensional manipulable object, e.g. on the faces of a cube that can be rotated by the user - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method includes displaying a plurality of computer-generated objects, including a first computer-generated object at a first position within an environment and a second computer-generated object at a second position within the environment. The first computer-generated object corresponds to a first user interface element that includes a first set of controls for modifying a content item. The method includes, while displaying the plurality of computer-generated objects, obtaining extremity tracking data. The method includes moving the first computer-generated object from the first position to a third position within the environment based on the extremity tracking data. The method includes, in accordance with a determination that the third position satisfies a proximity threshold with respect to the second position, merging the first computer-generated object with the second computer-generated object in order to generate a third computer-generated object for modifying the content item. The method includes displaying the third computer-generated object. ## Claims ### Claim 1 A method comprising: at an electronic device with one or more processors, a non-transitory memory, a display, and an extremity tracker: displaying, on the display, a plurality of computer-generated objects within an extended reality (XR) environment, including a first computer-generated object at a first position within the XR environment, and a second computer-generated object at a second position within the XR environment, wherein the first computer-generated object corresponds to a first user interface element that includes a first set of controls for modifying a content item; while displaying the plurality of computer-generated objects, obtaining extremity tracking data from the extremity tracker, wherein the extremity tracking data indicates a movement from the first position within the XR environment to a third position within the XR environment; moving the first computer-generated object from the first position to the third position within the XR environment based on the extremity tracking data; and in response to moving the first computer-generated object to the third position, in accordance with a determination that the third position satisfies a proximity threshold with respect to the second position: merging the first computer-generated object with the second computer-generated object in order to generate a third computer-generated object for modifying the content item; and displaying, on the display, the third computer-generated object within the XR environment. ### Claim 2 (depends on 1) The method of claim 1, wherein the extremity tracker includes a communication interface provided to communicate with a finger-wearable device, and wherein the extremity tracking data includes finger manipulation data from the finger-wearable device via the communication interface. ### Claim 3 (depends on 2) The method of claim 2, wherein the finger manipulation data corresponds to sensor data associated with one or more sensors integrated within the finger-wearable device. ### Claim 4 (depends on 3) The method of claim 3, wherein the sensor data includes positional data output from one or more positional sensors integrated in the finger-wearable device. ### Claim 5 (depends on 2) The method of claim 2, wherein the extremity tracker further includes a computer vision system, and wherein the computer vision system generates a respective portion of the extremity tracking data by tracking an extremity within image data. ### Claim 6 (depends on 3) The method of claim 3, wherein the sensor data includes contact intensity data output from a contact intensity sensor integrated in the finger-wearable device. ### Claim 7 (depends on 1) The method of claim 1, wherein moving the first computer-generated object from the first position to the third position includes selecting the first computer-generated object from the plurality of computer-generated objects based on the extremity tracking data. ### Claim 8 (depends on 7) The method of claim 7, further comprising determining that the extremity tracking data is indicative of a first gesture, wherein selecting the first computer-generated object is in response to determining that the extremity tracking data is indicative of the first gesture. ### Claim 9 (depends on 8) The method of claim 8, wherein the first gesture corresponds to a pinch gesture. ### Claim 10 (depends on 1) The method of claim 1, wherein merging the first computer-generated object with the second computer-generated object is further in response to determining that the extremity tracking data is indicative of a de-pinch gesture. ### Claim 11 (depends on 1) The method of claim 1, wherein the plurality of computer-generated objects includes a combination of two-dimensional (2D) and three-dimensional (3D) computer-generated objects. ### Claim 12 (depends on 1) The method of claim 1, wherein merging the first computer-generated object with the second computer-generated object includes ceasing to display the first computer-generated object and the second computer-generated object. ### Claim 13 (depends on 1) The method of claim 1, further comprising: determining whether or not the first computer-generated object and the second computer-generated object together satisfy one or more compatibility criteria, wherein merging the first computer-generated object with the second computer-generated object is further in response to determining that the first computer-generated object and the second computer-generated object together satisfy the one or more compatibility criteria; and in response to determining that the first computer-generated object and the second computer-generated object together do not satisfy the one or more compatibility criteria, foregoing merging the first computer-generated object with the second computer-generated object. ### Claim 14 (depends on 13) The method of claim 13, wherein the one or more compatibility criteria includes a common content type criterion that is satisfied when the first and second computer-generated objects are associated with a common content type. ### Claim 15 (depends on 1) The method of claim 1, wherein the second computer-generated object corresponds to a second user interface element that includes a second set of controls for modifying the content item, and wherein the third computer-generated object includes a portion of the first set of controls and a portion of the second set of controls. ### Claim 16 (depends on 15) The method of claim 15, further comprising: receiving an input that is directed to within the third computer-generated object, wherein the input selects the portion of the first set of controls within the third computer-generated object; and in response to receiving the input: manipulating the portion of the first set of controls based on the input; manipulating the portion of the second set of controls based on the input; and modifying the content item according to the manipulations. ### Claim 17 (depends on 1) The method of claim 1, wherein the second computer-generated object corresponds to the content item. ### Claim 18 (depends on 1) The method of claim 1, wherein the electronic device corresponds to a head-mountable device (HMD). ### Claim 19 An electronic device comprising: one or more processors; a non-transitory memory; a display; an extremity tracker; and one or more programs, wherein the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, the one or more programs including instructions for: displaying, on the display, a plurality of computer-generated objects, including a first computer-generated object at a first position within an XR environment, and a second computer-generated object at a second position within the XR environment, wherein the first computer-generated object corresponds to a first user interface element that includes a first set of controls for modifying a content item; while displaying the plurality of computer-generated objects, obtaining extremity tracking data from the extremity tracker, wherein the extremity tracking data indicates a movement from the first position within the XR environment to a third position within the XR environment; moving the first computer-generated object from the first position to the third position within the XR environment based on the extremity tracking data; and in response to moving the first computer-generated object to the third position, in accordance with a determination that the third position satisfies a proximity threshold with respect to the second position: merging the first computer-generated object with the second computer-generated object in order to generate a third computer-generated object for modifying the content item; and displaying, on the display, the third computer-generated object within the XR environment. ### Claim 20 A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by an electronic device with one or more processors, a display, and an extremity tracker, cause the electronic device to: display, on the display, a plurality of computer-generated objects, including a first computer-generated object at a first position within an XR environment, and a second computer-generated object at a second position within the XR environment, wherein the first computer-generated object corresponds to a first user interface element that includes a first set of controls for modifying a content item; while displaying the plurality of computer-generated objects, obtain extremity tracking data from the extremity tracker, wherein the extremity tracking data indicates a movement from the first position within the XR environment to a third position within the XR environment; move the first computer-generated object from the first position to the third position within the XR environment based on the extremity tracking data; and in response to moving the first computer-generated object to the third position, in accordance with a determination that the third position satisfies a proximity threshold with respect to the second position: merge the first computer-generated object with the second computer-generated object in order to generate a third computer-generated object for modifying the content item; and display, on the display, the third computer-generated object within the XR environment.