# US 2024/0201787 A1 · Hand-movement based interaction with augmented reality objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0201787 A1
- Document ID: US20240201787A1
- Status: Published application
- Kind: A1
- Application number: 18/067,956
- Filed: 2022-12-19
- Published: 2024-06-20
- Earliest claimed: 2022-12-19
- Assignees: T-Mobile USA, Inc.
- Applicants: T-Mobile USA, Inc.
- Inventors: Bharath Reddy Medarametla Lakshmi
- Web page: https://patentgrep.com/patent/US20240201787A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- H04M 1/72454: according to context-related or environment-related conditions
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)

## Abstract

This document discloses techniques, apparatuses, and systems for hand-movement based interactions with augmented reality (AR) objects. A set of hand gestures is determined, where each hand gesture is associated with a respective interaction with a virtual object in an AR environment. An environment is captured within the field of view of a viewfinder and included within an AR environment having a virtual object. A hand gesture is performed in the field of view of the viewfinder is captured. The captured hand gesture is determined to correspond to a particular hand gesture from the set of hand gestures. As a result, a particular interaction associated with the particular hand gesture is performed on an object within the augmented reality environment. In doing so, an AR engine can effectively interact with objects in an AR environment using hand gestures.

## Claims

### Claim 1

At least one non-transitory computer-readable storage medium storing instructions, which, when executed by at least one data processor of a system, causes the system to:
determine a set of hand gestures associated with interactions with a virtual object in an augmented reality environment,
wherein each respective hand gesture in the set of hand gestures is associated with a respective interaction with the virtual object in the augmented reality environment;
capture, using a camera of a smartphone, an environment within the field of view of the camera;
present the augmented reality environment on the display of the smartphone,
wherein the augmented reality environment includes the virtual object presented in the captured environment within the field of view of the camera;
capture a first hand gesture performed in the field of view of the camera,
wherein the display of the augmented reality environment includes the first hand gesture;
capture a second hand gesture performed in the field of view of the camera,
wherein the display of the augmented reality environment includes the second hand gesture;
determine that the first hand gesture corresponds to a particular hand gesture from the set of hand gestures,
wherein the particular hand gesture is associated, in the set of hand gestures, with a particular type of interaction with the virtual object in the augmented reality environment;
determine that the second hand gesture corresponds to an additional particular hand gesture from the set of hand gestures,
wherein the additional particular hand gesture is associated, in the set of hand gestures, with an absolute amount of interaction with the virtual object in the augmented reality environment, the absolute amount of interaction unrelated to a relationship between the second hand gesture and the augmented reality environment; and
in response to the determination that the first hand gesture corresponds to the particular hand gesture and the second hand gesture corresponds to the additional particular hand gesture, interact with the virtual object in the augmented reality environment in accordance with a particular interaction,
wherein the particular interaction is of the particular type and by the absolute amount.

### Claim 2 (depends on 1)

The at least one non-transitory computer-readable storage medium of claim 1, wherein the smartphone includes multiple applications capable of providing augmented reality environments, and wherein the set of hand gestures is associated with the interactions in each of the multiple applications.

### Claim 3 (depends on 1)

The at least one non-transitory computer-readable storage medium of claim 1, wherein the system is further caused to:
capture a third hand gesture performed in the field of view of the camera; and
select the virtual object from multiple virtual objects in the augmented reality environment based on the third hand gesture,
wherein the interaction with the virtual object in the augmented reality environment in accordance with the particular interaction is based on the selection of the virtual object from the multiple virtual objects.

### Claim 4 (depends on 1)

The at least one non-transitory computer-readable storage medium of claim 1, wherein:
the smartphone is connected to a telecommunications network;
the set of hand gestures is stored on a remote server of the telecommunications network; and
the system is further caused to receive the set of hand gestures from the remote server.

### Claim 5 (depends on 1)

The at least one non-transitory computer-readable storage medium of claim 1, wherein the particular interaction comprises a movement of the virtual object in the augmented reality environment, and wherein the particular interaction specifies a direction of the movement of the virtual object in the augmented reality environment.

### Claim 6 (depends on 1)

The at least one non-transitory computer-readable storage medium of claim 1, wherein:
the set of hand gestures includes a third hand gesture associated with a first interaction and a fourth hand gesture associated with a second interaction,
the fourth hand gesture different from the third hand gesture,
wherein the first interaction includes a first movement of the virtual object in the augmented reality environment,
wherein the first interaction specifies a first direction as a direction of the first movement and a first amount as an amount of the first movement,
wherein the second interaction includes a second movement of the virtual object in the augmented reality environment, and
wherein the second interaction specifies the first direction as a direction of the second movement and a second amount as an amount of the second movement, the second amount different from the first amount.

### Claim 7 (depends on 1)

The at least one non-transitory computer-readable storage medium of claim 1, wherein:
the set of hand gestures includes a third hand gesture associated with a first interaction and a fourth hand gesture associated with a second interaction,
the fourth hand gesture different from the third hand gesture,
wherein the first interaction includes a first movement of the virtual object in the augmented reality environment,
wherein the first interaction specifies a first direction as a direction of the first movement,
wherein the second interaction includes a second movement of the virtual object in the augmented reality environment, and
wherein the second interaction specifies a second direction as a direction of the second movement, the second direction different from the first direction.

### Claim 8 (depends on 1)

The at least one non-transitory computer-readable storage medium of claim 1, wherein the virtual object comprises a virtual rendering of an object from the captured environment.

### Claim 9 (depends on 1)

The at least one non-transitory computer-readable storage medium of claim 1, wherein the particular interaction comprises a movement of the virtual object in the augmented reality environment, and wherein the system is further caused to:
capture third hand gesture performed in the field of view of the camera,
wherein the third hand gesture specifies a direction of the movement of the virtual object in the augmented reality environment.

### Claim 10

At least one non-transitory computer-readable storage medium storing instructions, which, when executed by at least one data processor of a system, causes the system to:
determine a set of hand gestures associated with interactions with an object in an augmented reality environment,
wherein each respective hand gesture in the set of hand gestures is associated with a respective interaction with the object in the augmented reality environment;
capture, using a viewfinder, an environment within the field of view of the viewfinder;
provide the augmented reality environment,
wherein the augmented reality environment includes a plurality of objects comprising a virtual object and a virtual rendering of a physical object from the captured environment within the field of view of the viewfinder;
capture a first hand gesture performed in the field of view of the viewfinder;
capture a second hand gesture performed in the field of view of the viewfinder;
determine that the first hand gesture corresponds to a particular hand gesture from the set of hand gestures,
wherein the particular hand gesture is associated, in the set of hand gestures, with a particular type of interaction with the object in the augmented reality environment;
determine that the second hand gesture corresponds to an additional particular hand gesture from the set of hand gestures,
wherein the additional particular hand gesture is associated, in the set of hand gestures, with an absolute amount of interaction with the object in the augmented reality environment, the absolute amount of interaction unrelated to a relationship between the second hand gesture and the augmented reality environment; and
in response to the determination that the first hand gesture corresponds to the particular hand gesture and the second hand gesture corresponds to the additional particular hand gesture, interact, in the augmented reality environment, with a particular object of the plurality of objects in accordance with a particular interaction,
wherein the particular interaction is of the particular type and by the absolute amount.

### Claim 11 (depends on 10)

The at least one non-transitory computer-readable storage medium of claim 10, wherein the set of hand gestures associated with the interactions comprises a library of hand gestures, wherein the library of hand gestures is utilized by multiple devices capable of providing augmented reality environments.

### Claim 12 (depends on 10)

The at least one non-transitory computer-readable storage medium of claim 10, wherein the set of hand gestures associated with the interactions comprises a library of hand gestures, wherein the library of hand gestures is utilized by multiple applications capable of providing augmented reality environments.

### Claim 13 (depends on 10)

The at least one non-transitory computer-readable storage medium of claim 10, wherein the system is further caused to:
capture a third hand gesture performed in the field of view of the viewfinder; and
select the particular object from the plurality of objects in the augmented reality environment based on the third hand gesture,
wherein the interaction with the particular object in accordance with the particular interaction is based on the selection of the particular object from the plurality of objects.

### Claim 14 (depends on 10)

The at least one non-transitory computer-readable storage medium of claim 10, wherein the particular object comprises the virtual rendering of the physical object from the captured environment within the field of view of the viewfinder.

### Claim 15 (depends on 10)

The at least one non-transitory computer-readable storage medium of claim 10, wherein the particular interaction comprises a movement of the particular object in the augmented reality environment, and wherein the particular interaction specifies a direction of the movement of the particular object in the augmented reality environment.

### Claim 16

A system comprising:
at least one hardware processor; and
at least one non-transitory memory storing instructions, which, when executed by the at least one hardware processor, cause the system to:
determine a set of hand gestures associated with interactions with an object in an augmented reality environment,
wherein each respective hand gesture in the set of hand gestures is associated with a respective interaction with the object in the augmented reality environment;
capture, using a viewfinder, an environment within the field of view of the viewfinder;
provide the augmented reality environment,
wherein the augmented reality environment includes a plurality of objects comprising a virtual object and a virtual rendering of a physical object from the captured environment within the field of view of the viewfinder;
capture a first hand gesture performed in the field of view of the viewfinder;
capture a second hand gesture performed in the field of view of the camera;
determine that the first hand gesture corresponds to a particular hand gesture from the set of hand gestures,
wherein the particular hand gesture is associated with a particular type of interaction with the object in the augmented reality environment;
determine that the second hand gesture corresponds to an additional particular hand gesture from the set of hand gestures,
wherein the additional particular hand gesture is associated, in the set of hand gestures, with an absolute amount of interaction with the virtual object in the augmented reality environment, the absolute amount of interaction unrelated to a relationship between the second hand gesture and the augmented reality environment; and
in response to the determination that the first hand gesture corresponds to the particular hand gesture and the second hand gesture corresponds to the additional particular hand gesture, interact, in the augmented reality environment, with a particular object of the plurality of objects in accordance with a particular interaction,
wherein the particular interaction is of the particular type and by the absolute amount.

### Claim 17 (depends on 16)

The system of claim 16, wherein the system is further caused to:
capture a third hand gesture performed in the field of view of the viewfinder; and
select the particular object from the plurality of objects in the augmented reality environment based on the third hand gesture,
wherein the interaction with the particular object in accordance with the particular interaction is based on the selection of the particular object from the plurality of objects.

### Claim 18 (depends on 16)

The system of claim 16, wherein the particular object comprises the virtual rendering of the physical object from the captured environment within the field of view of the viewfinder.

### Claim 19 (depends on 16)

The system of claim 16, wherein the particular interaction comprises a movement of the particular object in the augmented reality environment, and wherein the particular interaction specifies a direction of the movement of the particular object in the augmented reality environment.

### Claim 20

(canceled)

### Claim 21 (depends on 1)

The at least one non-transitory computer-readable storage medium of claim 1, wherein:
the smartphone is connected to a telecommunications network;
an alternate reality engine used to create the alternate reality environment is stored on a remote server of the telecommunications network; and
the system is further caused to:
transmit data indicative of the captured environment to the remote server; and
receive data indicative of the augmented reality environment from the remote server.
