# US 11,307,677 B2 · Method and device for object pointing in virtual reality (VR) scene using a gamepad, and VR apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,307,677 B2
- Document ID: US11307677B2
- Status: Granted
- Kind: B2
- Application number: 17/105,975
- Filed: 2020-11-27
- Published: 2021-03-18
- Granted: 2022-04-19
- Earliest claimed: 2017-06-23
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Dhaval Jitendra Joshi
- Web page: https://patentgrep.com/patent/US11307677B2

## CPC

- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes
- A63F 13/219: for aiming at specific areas on the display, e.g. light-guns
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0338: with detection of limited linear or angular displacement of an operating part of the device from a neutral position, e.g. isotonic or isometric joysticks
- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04892: Arrangements for controlling cursor position based on codes indicative of cursor displacements from one discrete location to another, e.g. using cursor control keys associated to different directions or using the tab key (arrangements for controlling cursor position based on coordinate signals G06F3/038)
- A63F 2300/105: using inertial sensors, e.g. accelerometers, gyroscopes
- A63F 2300/1087: comprising photodetecting means, e.g. a camera
- A63F 2300/8082: Virtual reality
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR

## Abstract

A method and device for pointing at an object in a Virtual Reality (VR) scene, and a VR apparatus are provided. The method includes sensing a movement of a gamepad from a first position to a second position in a reality scene through an Inertial Measurement Unit (IMU) orientation sensor. The IMU orientation sensor is mounted in the gamepad, and the gamepad is configured to control an object in a VR scene. The method also includes generating a first position vector based on the movement; acquiring a first absolute position coordinate of the second position in the reality scene according to a position coordinate of a starting point in the reality scene and the first position vector; converting the first absolute position coordinate into a first virtual position coordinate in the VR scene; and pointing at a target object located at the first virtual position coordinate in the VR scene.

## Claims

### Claim 1

A method for pointing at an object in a Virtual Reality (VR) scene, comprising:
sensing, by a computing device, a movement of a gamepad from a first position to a second position in a reality scene through an Inertial Measurement Unit (IMU) orientation sensor, wherein the gamepad is an old-fashioned gamepad, the IMU orientation sensor is either plugged to or embedded in the gamepad, and the gamepad is configured to, in an object selection scenario in a VR scene and in response to the movement of the gamepad sensed through the IMU orientation sensor, trigger a movement of a virtual pointer to point at an object according to a proportional relationship between a space of the reality scene and a space of the virtual scene, the VR scene being presented by a head-mounted display;
generating, by the computing device, a first position vector based on the movement of the gamepad;
acquiring, by the computing device, a first absolute position coordinate of the second position in the reality scene according to a position coordinate of a starting point in the reality scene and the first position vector;
converting, by the computing device, the first absolute position coordinate into a first virtual position coordinate in the VR scene;
pointing, by the computing device, at a target object located at the first virtual position coordinate in the VR scene, wherein when pointing at the target object, the target object is selected and changed into an operable state; and
executing an operation on the target object at the operable state in the VR scene in response to an instruction from a key of the gamepad.

### Claim 2 (depends on 1)

The method according to claim 1, wherein acquiring the first absolute position coordinate of the second position in the reality scene according to the position coordinate of the starting point in the reality scene and the first position vector comprises:
acquiring the first absolute position coordinate according to the position coordinate of the starting point, the first position vector and a second absolute position coordinate, wherein the second absolute position coordinate is a coordinate of the first position in the reality scene.

### Claim 3 (depends on 1)

The method according to claim 1, wherein pointing at the target object oriented by the first virtual position coordinate in the VR scene comprises:
orienting a laser pointer toward the target object located at the first virtual position coordinate in the VR scene, wherein a coordinate of an intersection of a ray originated from the laser pointer and the target object is the first virtual position coordinate.

### Claim 4 (depends on 3)

The method according to claim 3, wherein orienting the laser pointer toward the target object in the VR scene comprises:
moving the laser pointer from a third position to a fourth position in the VR scene, wherein the laser pointer orients the target object when being positioned at the fourth position in the VR scene, and when the gamepad is positioned at the first position in the reality scene, the laser pointer is positioned at the third position in the VR scene.

### Claim 5 (depends on 3)

The method according to claim 3, further comprising:
when the laser pointer points at the target object, displaying the target object as the operable state in the VR scene.

### Claim 6 (depends on 1)

The method according to claim 1, before acquiring the first absolute position coordinate of the second position in the reality scene according to the position coordinate of the starting point in the reality scene and the first position vector, the method further comprising:
acquiring the position coordinate of the starting point in the reality scene through a camera.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the starting point is a point in an image shot by the camera.

### Claim 8

A Virtual Reality (VR) apparatus, comprising:
a gamepad, configured to, in an object selection scenario in a VR scene and in response to the movement of the gamepad sensed through the IMU orientation sensor, trigger a movement of a virtual pointer to point at an object according to a proportional relationship between a space of the reality scene and a space of the virtual scene, the gamepad being an old-fashioned gamepad;
an Inertial Measurement Unit (IMU) orientation sensor, either plugged to or embedded in the gamepad, and configured to sense a movement of the gamepad from a first position to a second position in a reality scene;
a processor, connected with the IMU orientation sensor, and configured to:
generate a first position vector based on the movement sensed by the IMU orientation sensor;
acquire a first absolute position coordinate of the second position in the reality scene according to a position coordinate of a starting point in the reality scene and the first position vector;
convert the first absolute position coordinate into a first virtual position coordinate in the VR scene;
point at a target object located at the first virtual position coordinate in the VR scene, wherein when pointing at the target object, the target object is selected and changed into an operable state; and
execute an operation on the target object at the operable state in the VR scene in response to an instruction from a key of the gamepad,
a head-mounted display unit, configured to display the VR scene.

### Claim 9 (depends on 8)

The VR apparatus according to claim 8, wherein the processor is further configured to:
acquire the first absolute position coordinate according to the position coordinate of the starting point, the first position vector and a second absolute position coordinate, wherein the second absolute position coordinate is a coordinate of the first position in the reality scene.

### Claim 10 (depends on 8)

The VR apparatus according to claim 8, wherein the processor is further configured to, when the laser pointer points at the target object, display the target object as the operable state in the VR scene.

### Claim 11 (depends on 8)

The VR apparatus according to claim 8, wherein the processor is further configured to:
orient a laser pointer toward the target object located at the first virtual position coordinate in the VR scene, wherein a coordinate of an intersection of a ray originated from the laser pointer and the target object is the first virtual position coordinate.

### Claim 12 (depends on 11)

The VR apparatus according to claim 11, wherein the processor is further configured to:
move the laser pointer from a third position to a fourth position in the VR scene, wherein the laser pointer orients the target object when being positioned at the fourth position in the VR scene, and when the gamepad is positioned at the first position in the reality scene, the laser pointer is positioned at the third position in the VR scene.

### Claim 13 (depends on 8)

The VR apparatus according to claim 8, further comprising:
a camera, configured to acquire the position coordinate of the starting point in the reality scene, before the processor acquires the first absolute position coordinate of the second position in the reality scene according to the position coordinate of the starting point in the reality scene and the first position vector.

### Claim 14 (depends on 13)

The VR apparatus according to claim 13, wherein the starting point is a point in an image shot by the camera.

### Claim 15

A non-transitory computer-readable storage medium storing computer program instructions executable by at least one processor to perform:
sensing a movement of a gamepad from a first position to a second position in a reality scene through an Inertial Measurement Unit (IMU) orientation sensor, wherein the gamepad is an old-fashioned gamepad, the IMU orientation sensor is either plugged to or embedded in the gamepad, and the gamepad is configured to, in an object selection scenario in a VR scene and in response to the movement of the gamepad sensed through the IMU orientation sensor, trigger a movement of a virtual pointer to point at an object according to a proportional relationship between a space of the reality scene and a space of the virtual scene, the VR scene being presented by a head-mounted display;
generating a first position vector based on the movement of the gamepad;
acquiring a first absolute position coordinate of the second position in the reality scene according to a position coordinate of a starting point in the reality scene and the first position vector;
converting the first absolute position coordinate into a first virtual position coordinate in the VR scene;
pointing at a target object located at the first virtual position coordinate in the VR scene, wherein when pointing at the target object, the target object is selected and changed into an operable state; and
executing an operation on the target object at the operable state in the VR scene in response to an instruction from a key of the gamepad.

### Claim 16 (depends on 15)

The storage medium according to claim 15, wherein acquiring the first absolute position coordinate of the second position in the reality scene according to the position coordinate of the starting point in the reality scene and the first position vector comprises:
acquiring the first absolute position coordinate according to the position coordinate of the starting point, the first position vector and a second absolute position coordinate, wherein the second absolute position coordinate is a coordinate of the first position in the reality scene.

### Claim 17 (depends on 15)

The storage medium according to claim 15, wherein before acquiring the first absolute position coordinate of the second position in the reality scene according to the position coordinate of the starting point in the reality scene and the first position vector, the computer program instructions further cause the at least one processor to perform:
acquiring the position coordinate of the starting point in the reality scene through a camera, wherein the starting point is a point in an image shot by the camera.

### Claim 18 (depends on 15)

The storage medium according to claim 15, wherein after pointing at the target object located at the first virtual position coordinate in the VR scene, the computer program instructions further cause the at least one processor to perform:
acquiring an operation instruction generated when a key on the gamepad is operated; and
executing an operation indicated by the operation instruction on the target object in the VR scene.

### Claim 19 (depends on 15)

The storage medium according to claim 15, wherein pointing at the target object oriented by the first virtual position coordinate in the VR scene comprises:
orienting a laser pointer toward the target object located at the first virtual position coordinate in the VR scene, wherein a coordinate of an intersection of a ray originated from the laser pointer and the target object is the first virtual position coordinate.

### Claim 20 (depends on 19)

The storage medium according to claim 19, wherein orienting the laser pointer toward the target object in the VR scene comprises:
moving the laser pointer from a third position to a fourth position in the VR scene, wherein the laser pointer orients the target object when being positioned at the fourth position in the VR scene, and when the gamepad is positioned at the first position in the reality scene, the laser pointer is positioned at the third position in the VR scene.