# US 2025/0085783 A1 · Virtual manipulation of augmented and virtual reality objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0085783 A1
- Document ID: US20250085783A1
- Status: Published application
- Kind: A1
- Application number: 18/463,113
- Filed: 2023-09-07
- Published: 2025-03-13
- Earliest claimed: 2023-09-07
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Mason Spong
- Web page: https://patentgrep.com/patent/US20250085783A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/0486: Drag-and-drop
- G06F 3/0487: using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88)
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods are provided. For example, a method includes determining a position of a user's hand and identifying a manipulation gesture performed by the user targeting a virtual object. The method also includes determining a three-dimensional (3D) origin point based on the position of the user's hand when the manipulation gesture is performed, and determining a 3D end point based on a movement of the user's hand from the origin point. The method additionally includes deriving a 3D vector based on the 3D origin point and the 3D end point, and applying an action to the targeted virtual object based on the 3D vector, wherein the targeted virtual object is at a distance greater than the user's arm reach.

## Claims

### Claim 1

A system, comprising:
one or more hardware processors; and
at least one memory storing instructions that cause the one or more hardware processors to perform operations comprising:
determining a position of a user's hand;
identifying a manipulation gesture performed by the user targeting a virtual object;
determining a three-dimensional (3D) origin point based on the position of the user's hand when the manipulation gesture is performed;
determining a 3D end point based on a movement of the user's hand from the origin point;
deriving a 3D vector based on the 3D origin point and the 3D end point;
applying an action to the targeted virtual object based on the 3D vector, wherein the targeted virtual object is at a distance greater than the user's arm reach, wherein applying the action comprises imparting a velocity to the targeted virtual object based on the 3D vector, wherein the velocity imparted to the targeted virtual object has a direction corresponding to a vector direction of the 3D vector and a magnitude both derived via observation of the hand only; and
displaying, via a display, the action performed on the targeted virtual object.

### Claim 2 (depends on 1)

The system of claim 1, wherein applying the action comprises imparting a velocity to the targeted virtual object based on the 3D vector, wherein the velocity imparted to the targeted virtual object has a direction corresponding to a vector direction of the 3D vector and a magnitude.

### Claim 3 (depends on 2)

The system of claim 2, wherein the magnitude is based on a length of the 3D vector or on a hand velocity of the user's hand.

### Claim 4 (depends on 2)

The system of claim 2, wherein imparting the velocity comprises moving the targeted virtual object in an augmented reality (AR) or a virtual reality (VR) environment in the direction and at the velocity.

### Claim 5 (depends on 4)

The system of claim 4, comprising stopping the movement of the targeted virtual object based on identifying a stop gesture performed by the user.

### Claim 6 (depends on 1)

The system of claim 1, wherein the manipulation gesture comprises a hand gesture, a body gesture, a voice command, or a combination thereof.

### Claim 7 (depends on 6)

The system of claim 6, wherein the hand gesture comprises a pinch gesture, and wherein determining the 3D end point comprises determining a point in space where the user stops moving the user's hand while maintaining the pinch gesture.

### Claim 8 (depends on 7)

The system of claim 7, wherein identifying the manipulation gesture comprises using an artificial intelligence model to determine that the manipulation gesture is the pinch gesture.

### Claim 9 (depends on 8)

The system of claim 8, wherein the artificial intelligence model comprises a deep learning model configured to detect hand landmarks directly from a camera image.

### Claim 10 (depends on 1)

The system of claim 1, comprising operations for:
determining a second 3D origin point based on a stop position of the user's hand when the user stops moving the hand;
determining a second 3D end point based on a second movement of the user's hand from the second 3D origin point;
deriving a second 3D vector based on the second 3D origin point and the second 3D end point; and
applying a second action to the targeted virtual object based on the second 3D vector.

### Claim 11 (depends on 10)

The system of claim 10, wherein applying the second action comprises imparting a velocity to the targeted virtual object based on the second 3D vector, wherein the velocity imparted to the targeted virtual object has a direction corresponding to a vector direction of the second 3D vector and a magnitude based on a length of the second 3D vector, and wherein imparting the velocity comprises moving the targeted virtual object in an augmented reality (AR) or a virtual reality (VR) environment in the direction and at the velocity.

### Claim 12 (depends on 1)

The system of claim 1, wherein determining the position of the user's hand comprises observing the user's hand via one or more camera sensors, radar sensors, light detection and ranging (lidar) sensors, or a combination thereof, included in an AR device, a VR device, or a combination thereof, worn by the user.

### Claim 13 (depends on 12)

The system of claim 12, wherein determining the position of the user's hand comprises observing the user's hand via one or more camera sensors, radar sensors, lidar sensors, or a combination thereof, disposed external to the AR device and to the VR device.

### Claim 14 (depends on 1)

The system of claim 1, wherein applying the action to the targeted virtual object comprises executing instructions for a joystick driver.

### Claim 15 (depends on 14)

The system of claim 14, wherein the instructions for the joystick driver emulate a physical joystick.

### Claim 16 (depends on 1)

The system of claim 1, comprising deriving that the user is targeting the virtual object based on identifying a targeting gesture performed by the user targeting the virtual object;

### Claim 17 (depends on 16)

The system of claim 16, wherein the targeting gesture comprises a hand gesture, a body gesture, a voice command, or a combination thereof.

### Claim 18 (depends on 17)

The system of claim 17, wherein the hand gesture comprises an extended palm pointing towards the targeted virtual object.

### Claim 19

A method comprising:
determining a position of a user's hand;
identifying a manipulation gesture performed by the user targeting a virtual object;
determining a three-dimensional (3D) origin point based on the position of the user's hand when the manipulation gesture is performed;
determining a 3D end point based on a movement of the user's hand from the origin point;
deriving a 3D vector based on the 3D origin point and the 3D end point;
applying an action to the targeted virtual object based on the 3D vector, wherein the targeted virtual object is at a distance greater than the user's arm reach, wherein applying the action comprises imparting a velocity to the targeted virtual object based on the 3D vector, wherein the velocity imparted to the targeted virtual object has a direction corresponding to a vector direction of the 3D vector and a magnitude both derived via observation of the hand only; and
displaying, via a display, the action performed on the targeted virtual object.

### Claim 20

A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
determining a position of a user's hand;
identifying a manipulation gesture performed by the user targeting a virtual object;
determining a three-dimensional (3D) origin point based on the position of the user's hand when the manipulation gesture is performed;
determining a 3D end point based on a movement of the user's hand from the origin point;
deriving a 3D vector based on the 3D origin point and the 3D end point;
applying an action to the targeted virtual object based on the 3D vector, wherein the targeted virtual object is at a distance greater than the user's arm reach, wherein applying the action comprises imparting a velocity to the targeted virtual object based on the 3D vector, wherein the velocity imparted to the targeted virtual object has a direction corresponding to a vector direction of the 3D vector and a magnitude both derived via observation of the hand only; and
displaying, via a display, the action performed on the targeted virtual object.
