# US 12,762,802 B2 · Virtual object interaction in augmented reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,762,802 B2 - Document ID: US12762802B2 - Status: Granted - Kind: B2 - Application number: 18/128,127 - Filed: 2023-03-29 - Published: 2024-05-16 - Granted: 2026-10-13 - Earliest claimed: 2022-11-14 - Assignees: Rockwell Automation Technologies, Inc. - Applicants: Rockwell Automation Technologies, Inc. - Inventors: Rafi Izhar; Simon Blackwell; Pascal Gilbraith; John Howard Pritchard - Web page: https://patentgrep.com/patent/US12762802B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06T 15/205: {Image-based rendering} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method includes using at least one processor to receive image data including one or more virtual objects and determining a distance between a user extremity and the one or more virtual objects based on the image data. Further, the method includes detecting a first gesture of the user extremity based on the image data and generating a virtual sphere within a virtual object of the one or more virtual objects. The method also includes determining that the distance is less than a threshold, detecting movement of the user extremity based on the image data, and adjusting a position of the virtual object based on the movement of the user extremity. ## Claims ### Claim 1 A method, comprising: receiving, via at least one processor, image data comprising two or more virtual objects; determining, via the at least one processor, a distance between a user extremity and the two or more virtual objects based on the image data; detecting, via the at least one processor, a first gesture of the user extremity based on the image data, wherein the first gesture comprises a pinch between a thumb of the user extremity and a finger of the user extremity; generating, via the at least one processor, a virtual sphere to be displayed within or on a virtual object of the two or more virtual objects, wherein the virtual sphere binds to each of the two or more virtual objects while being displayed within or on the virtual object; determining, via the at least one processor, a position of the virtual sphere to be displayed on or within the virtual object based on a line of direction that corresponds to the thumb of the user extremity and the finger of the user extremity; determining, via the at least one processor, that the distance is less than a threshold; locking, via the at least one processor, the position of the virtual sphere individually to each of the two or more virtual objects at a position of the user extremity in response to determining that the distance is less than the threshold; detecting, via the at least one processor, movement of the position of the user extremity based on the image data; and adjusting, via the at least one processor, two or more positions of the two or more virtual objects based on the movement of the position of the virtual sphere and the position of the user extremity, wherein the adjustment of the two or more positions of the two or more virtual objects comprises a rotational movement relative to the virtual sphere. ### Claim 2 (depends on 1) The method of claim 1, wherein adjusting the position of the two or more virtual objects comprises moving the virtual sphere associated with the two or more virtual objects in a three-dimensional coordinate plane with respect to the movement of the position of the user extremity. ### Claim 3 (depends on 1) The method of claim 1, comprising: generating, via the at least one processor, a pointer cursor visualization in the image data based on the user extremity in response to the distance being above the threshold; and adjusting, via the at least one processor, the position of the two or more virtual objects based on an additional movement of the pointer cursor visualization. ### Claim 4 (depends on 1) The method of claim 1, wherein the first gesture corresponds to a command for adjusting the position of the two or more virtual objects. ### Claim 5 (depends on 1) The method of claim 1, comprising detecting a second gesture associated with the user extremity based on the image data. ### Claim 6 (depends on 5) The method of claim 5, comprising disassociating the virtual sphere from the two or more virtual objects in response to detecting the second gesture. ### Claim 7 (depends on 6) The method of claim 6, wherein the second gesture corresponds to the user extremity releasing the two or more virtual objects. ### Claim 8 (depends on 1) The method of claim 1, comprising: generating, via the at least one processor, a pointer cursor visualization based on the user extremity in response to the distance being above an additional threshold. ### Claim 9 (depends on 8) The method of claim 8, comprising: determining, via the at least one processor, that the pointer cursor visualization is not intersecting the virtual object; and querying, via the at least one processor, a range from the position of the user extremity for an interaction, wherein the interaction comprises the range being within the threshold and an intersection of the pointer cursor visualization with the virtual object. ### Claim 10 (depends on 1) The method of claim 1, comprising: detecting, via the at least one processor, a second gesture of the user extremity or an additional user extremity; and adjusting, via the at least one processor, the position of the two or more virtual objects to a position of the additional user extremity in response to detecting the second gesture. ### Claim 11 (depends on 1) The method of claim 1, comprising determining, via the at least one processor, that a first adjusted position of a first virtual object of the two or more virtual objects is colliding with a second adjusted position of a second virtual object of the two or more virtual objects based on a proximity of the first virtual object to the second virtual object. ### Claim 12 A tangible, non-transitory, computer-readable medium configured to store instructions executable by at least one processor in a computing device, wherein the instructions are configured to cause the at least one processor to: receive image data comprising two or more virtual objects; determine a distance between a user extremity and the two or more virtual objects based on the image data; detect a first gesture of the user extremity based on the image data, wherein the first gesture comprises a pinch between a thumb of the user extremity and a finger of the user extremity; generate a virtual sphere to be displayed within or on a virtual object of the two or more virtual objects, wherein the virtual sphere binds to each of the two or more virtual objects while being displayed within or on the virtual object; determine a position of the virtual sphere to be displayed on or within the virtual object based on a line of direction that corresponds to the thumb of the user extremity and the finger of the user extremity; determine that the distance is less than a threshold; lock the position of the virtual sphere individually to each of the two or more virtual objects at a position of the user extremity in response to determining that the distance is less than the threshold; detect movement of the position of the user extremity based on the image data; and adjust two or more positions of the two or more virtual objects based on the movement of the position of the virtual sphere and the position of the user extremity, wherein the adjustment of the two or more positions of the two or more virtual objects comprises a rotational movement relative to the virtual sphere. ### Claim 13 (depends on 12) The computer-readable medium of claim 12, wherein the instructions to adjust the position of the two or more virtual objects are configured to cause the at least one processor to move the virtual sphere associated with the two or more virtual objects in a three-dimensional coordinate plane with respect to the movement of the position of the user extremity. ### Claim 14 (depends on 12) The computer-readable medium of claim 12, wherein the instructions to adjust the position of the two or more virtual objects are configured to cause the at least one processor to: generate a pointer cursor visualization in the image data based on the user extremity in response to the distance being above the threshold; and adjust the position of the two or more virtual objects on an additional movement of the pointer cursor visualization. ### Claim 15 (depends on 12) The computer-readable medium of claim 12, wherein the instructions to adjust the position of the two or more virtual objects are configured to cause the processor to: detect a second gesture associated with the user extremity based on the image data; and disassociate the virtual sphere from the two or more virtual objects in response to detecting the second gesture. ### Claim 16 A system, comprising: an image sensor configured to capture image data; and a processor configured to perform operations comprising: receiving the image data from the image sensor; generating two or more virtual objects and superimposing the two or more virtual objects on the image data; determining a distance between a user extremity and the two or more virtual objects based on the image data; detecting a first gesture of the user extremity based on the image data, wherein the first gesture comprises a pinch between a thumb of the user extremity and a finger of the user extremity; generating a virtual sphere to be displayed within or on a virtual object of the two or more virtual objects, wherein the virtual sphere binds to each of the two or more virtual objects while being displayed within or on the virtual object; determining a position of the virtual sphere to be displayed on or within the virtual object based on a line of direction that corresponds to the thumb of the user extremity and the finger of the user extremity; determining that the distance is less than a threshold; locking the position of the virtual sphere individually to each of the two or more virtual objects at a position of the user extremity in response to determining that the distance is less than the threshold; detecting movement of the position of the user extremity based on the image data; and adjusting two or more positions of the two or more virtual objects based on the movement of the position of the virtual sphere and the position of the user extremity, wherein the adjustment of the two or more positions of the two or more virtual objects comprises a rotational movement relative to the virtual sphere. ### Claim 17 (depends on 16) The system of claim 16, wherein the processor is configured to adjust the position of the two or more virtual objects by moving the virtual sphere associated with the two or more virtual objects in a three-dimensional coordinate plane with respect to the movement of the position of the user extremity. ### Claim 18 (depends on 16) The system of claim 16, wherein the processor is configured to perform the operations comprising: generating a pointer cursor visualization in the image data based on the user extremity in response to the distance being above the threshold; and adjusting the position of the two or more virtual objects based on an additional movement of the pointer cursor visualization. ### Claim 19 (depends on 16) The system of claim 16, wherein the first gesture corresponds to a command for adjusting the position of the two or more virtual objects. ### Claim 20 (depends on 16) The system of claim 16, wherein the processor is configured to perform the operations comprising: detecting a second gesture associated with the user extremity based on the image data; and disassociating the virtual sphere from the two or more virtual objects in response to detecting the second gesture.