# US 11,992,751 B2 · Virtual reality hand gesture generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,992,751 B2
- Document ID: US11992751B2
- Status: Granted
- Kind: B2
- Application number: 17/229,619
- Filed: 2021-04-13
- Published: 2021-07-29
- Granted: 2024-05-28
- Earliest claimed: 2016-10-11
- Assignees: Valve Corporation
- Applicants: Valve Corporation
- Inventors: Scott Douglas Nietfeld; Joe van den Heuvel
- Web page: https://patentgrep.com/patent/US11992751B2

## CPC

- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/218: using pressure sensors, e.g. generating a signal proportional to the pressure applied by the player
- A63F 13/24: Constructional details thereof, e.g. game controllers with detachable joystick handles
- A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
- A63F 13/67: adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0325: {using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/03547: {Touch pads, in which fingers can move on a surface}
- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method including receiving at least one of touch data or force data representing a touch input received at the controller, determining one or more model(s), generating image data using the one or more models, the image data representing at least a hand gesture corresponding to the touch input received at the controller, and transmitting the image data to a virtual reality (VR) environment for display.

## Claims

### Claim 1

A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:
receiving, from a handheld controller:
touch data generated by one or more first sensors of the handheld controller, the touch data indicating a touch input received at the handheld controller; and
force data generated by one or more second sensors of the handheld controller, the force data indicating an amount of force associated with the touch input;
selecting a trained model from among a plurality of trained models based at least in part on a characteristic of at least one of the touch data or the force data and on a hand size of a user holding the handheld controller;
inputting the touch data and the force data to the trained model;
determining, using the trained model, that the touch data and the force data correspond to a hand gesture;
generating image data representing the hand gesture; and
causing an image of the hand gesture to be presented on a display based at least in part on the image data.

### Claim 2 (depends on 1)

The system of claim 1, wherein the causing the image of the hand gesture to be presented on the display comprises transmitting the image data to a Virtual Reality (VR) headset having the display.

### Claim 3 (depends on 1)

The system of claim 1, wherein the characteristic comprises at least one of:
a location on the handheld controller where the touch input is received; or
the amount of the force associated with the touch input.

### Claim 4 (depends on 1)

The system of claim 1, wherein the selecting of the trained model from among the plurality of trained models is further based on an additional characteristic of the user.

### Claim 5 (depends on 4)

The system of claim 4, wherein the additional characteristic comprises an age of the user.

### Claim 6 (depends on 1)

The system of claim 1, wherein the one or more second sensors comprise one or more force sensing resistors (FSRs).

### Claim 7 (depends on 1)

The system of claim 1, wherein the selecting of the trained model from among the plurality of trained models is further based on comparing at least one of:
the touch data to stored touch data associated with the plurality of trained models; or
the force data to stored force data associated with the plurality of trained models.

### Claim 8

A method comprising:
receiving, by one or more processors, from a handheld controller, at least one of:
touch data generated by one or more first sensors of the handheld controller, the touch data indicating a proximity of at least a portion of a hand relative to the handheld controller; or
force data generated by one or more second sensors of the handheld controller, the force data indicating an amount of force of a press on the handheld controller;
selecting a trained model from among a plurality of trained models based at least in part on a characteristic of the at least one of the touch data or the force data and on a size of the hand;
inputting, by the one or more processors, the at least one of the touch data or the force data to the trained model;
determining, using the trained model, that the at least one of the touch data or the force data corresponds to a hand gesture;
generating, by the one or more processors, image data representing the hand gesture; and
causing an image of the hand gesture to be presented on a display based at least in part on the image data.

### Claim 9 (depends on 8)

The method of claim 8, wherein the causing the image of the hand gesture to be presented on the display comprises transmitting the image data to a Virtual Reality (VR) headset having the display.

### Claim 10 (depends on 8)

The method of claim 8, wherein the characteristic comprises at least one of:
a location on the handheld controller where at least the portion of the hand touched the handheld controller; or
the amount of the force of the press.

### Claim 11 (depends on 8)

The method of claim 8, wherein the selecting of the trained model from among the plurality of trained models is further based on an additional characteristic of a user holding the handheld controller.

### Claim 12 (depends on 8)

The method of claim 8, wherein the selecting of the trained model from among the plurality of trained models is further based on comparing at least one of:
the touch data to stored touch data associated with the plurality of trained models; or
the force data to stored force data associated with the plurality of trained models.

### Claim 13 (depends on 8)

The method of claim 8, wherein the hand gesture is a first hand gesture, the method further comprising:
predicting a second hand gesture based at least in part on the first hand gesture;
generating second image data representing the second hand gesture;
receiving, from the handheld controller, after the predicting, at least one of:
second touch data generated by the one or more first sensors of the handheld controller, the second touch data indicating a second proximity of at least the portion of the hand relative to the handheld controller; or
second force data generated by the one or more second sensors of the handheld controller, the second force data indicating a second amount of force of a second press on the handheld controller; and
causing a second image of the second hand gesture to be presented on the display based at least in part on the second image data.

### Claim 14 (depends on 8)

The method of claim 8, wherein:
the receiving comprises receiving the force data;
the one or more second sensors comprise one or more force sensing resistors (FSRs);
the characteristic comprises a characteristic of the force data; and
the inputting comprises inputting the force data to the trained model.

### Claim 15

A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:
receiving, from a handheld controller, at least one of:
touch data indicating a proximity of at least a portion of a hand relative to the handheld controller; or
force data indicating an amount of force of a press on the handheld controller;
selecting a trained model from among a plurality of trained models based at least in part on a characteristic of the at least one of the touch data or the force data and on a size of the hand;
inputting the at least one of the touch data or the force data to the trained model;
determining, using the trained model, that the at least one of the touch data or the force data corresponds to a hand gesture;
generating image data representing the hand gesture; and
causing an image of the hand gesture to be presented on a display based at least in part on the image data.

### Claim 16 (depends on 15)

The system of claim 15, wherein the image comprises at least one of a three dimensional (3D) representation of the hand gesture or an animation of the hand gesture.

### Claim 17 (depends on 15)

The system of claim 15, wherein the characteristic comprises at least one of:
a location on the handheld controller where at least the portion of the hand touched the handheld controller; or
the amount of the force of the press.

### Claim 18 (depends on 15)

The system of claim 15, wherein the selecting of the trained model from among the plurality of trained models is further based on an additional characteristic of a user holding the handheld controller.

### Claim 19 (depends on 15)

The system of claim 15, wherein the selecting of the trained model from among the plurality of trained models is further based on comparing at least one of:
the touch data to stored touch data associated with the plurality of trained models; or
the force data to stored force data associated with the plurality of trained models.

### Claim 20 (depends on 15)

The system of claim 15, wherein:
the touch data was generated by one or more proximity sensors of the handheld controller;
the force data was generated by one or more force sensing resistors (FSRs) of the handheld controller;
the receiving comprises receiving the touch data and the force data; and
the inputting comprises inputting the touch data and the force data to the trained model.