# US 11,896,897 B2 · Virtual image display program, virtual image display apparatus, and virtual image display method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,896,897 B2
- Document ID: US11896897B2
- Status: Granted
- Kind: B2
- Application number: 17/703,021
- Filed: 2022-03-24
- Published: 2022-07-07
- Granted: 2024-02-13
- Earliest claimed: 2015-09-16
- Assignees: GREE, Inc.
- Applicants: GREE, Inc.
- Inventors: Masashi Watanabe
- Web page: https://patentgrep.com/patent/US11896897B2

## CPC

- A63F 13/26: having at least one additional display device, e.g. on the game controller or outside a game booth
- A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes
- A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 9/453: {Help systems}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A63F 2300/8082: Virtual reality
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A game processing apparatus displays an image of virtual space on a display that displays an image by using a sensor for identifying the position and direction of a certain body part of a player and the display. A controller in the game processing apparatus displays the image of the virtual space on the display in accordance with the position and direction of the certain body part of the player, and, when an information providing condition regarding the position and direction of the certain body part of the player is satisfied, outputs to-be-provided information.

## Claims

### Claim 1

A non-transitory computer-readable medium, on which is embodied a computer program product comprising code executable by a virtual image display apparatus having at least a processor and a memory, the memory being configured to store an information providing condition of the virtual image display apparatus and being further configured to store to-be-provided information, to cause the virtual image display apparatus to carry out steps of:
detecting, with a sensor operationally linked to the virtual image display apparatus, a plurality of movements of a body part of a player including a number of times that the player has changed a direction of movement of the body part and a movement speed of the head of the player, the body part comprising at least one of a head of the player and an eye of the player, and the sensor being at least one of the set of: a gyro sensor configured to measure movement of the head of the player, an acceleration sensor configured to measure movement of the head of the player, a geomagnetic sensor configured to measure movement of the head of the player and a line-of-sight sensor configured to measure movement of the eye of the player;
determining, based on each of the plurality of movements of the body part of the player, a position and a direction of the body part of the player during said plurality of movements;
distinguishing, based on whether the number of times that the player has changed the direction of movement of the body part is at least a threshold value and based on whether the movement speed of the head of the player satisfies a movement speed condition, whether each movement of the body part of the player is a predetermined movement associated with the information providing condition or is a normal movement associated with other gameplay, and classifying at least one first movement in the plurality of movements as a predetermined movement when the number of times that the player has changed the direction of movement of the body part is larger than the threshold value and at least one second movement in the plurality of movements as a normal movement when the at least one second movement is not the predetermined movement;
displaying, on a display operationally linked to the virtual image display apparatus, in accordance with the position and direction of the body part of the player, an image of a virtual space; and
triggering, upon classification of the at least one first movement of the body part of the player as the predetermined movement and upon satisfaction of the information providing condition, provision, with the virtual image display apparatus, of the to-be-provided information to the player by displaying the to-be-provided information.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium of claim 1, further comprising:
based on the at least one second movement being classified as the normal movement, identifying, with the sensor operationally linked to the virtual image display apparatus, an object in the virtual space corresponding to the position and the direction of the body part of the player; and
with the sensor operationally linked to the virtual image display apparatus, determining, based on the plurality of movements of the body part of the player, whether the object in the virtual space has been selected by the player.

### Claim 3 (depends on 2)

The non-transitory computer-readable medium of claim 2, wherein determining whether the object in the virtual space has been selected by the player comprises:
determining a gaze time over which a gaze of the player has been focused on the object; and
comparing the gaze time to a predetermined time and determining that the object has been selected when the gaze time is at least the predetermined time.

### Claim 4 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein:
the virtual space includes a target object; and
providing the to-be-provided information to the player comprises identifying an area of the virtual space other than the target object and displaying the to-be-provided information in the area of the virtual space other than the target object.

### Claim 5 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein:
the virtual space includes a moving object; and
providing the to-be-provided information to the player comprises displaying the to-be-provided information in association with the moving object and moving a display of the to-be-provided information along with the moving object.

### Claim 6 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein:
the virtual image display apparatus is configured to execute a game, the game comprising a step of displaying the image of the virtual space; and
providing the to-be provided information to the player comprises providing a hint to the player related to a next step for clearing the game.

### Claim 7 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein:
the virtual image display apparatus is configured to execute an application, the application comprising a step of displaying the image of the virtual space; and
providing the to-be-provided information to the player comprises displaying a menu item of the application.

### Claim 8 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein:
detecting, with the sensor operationally linked to the virtual image display apparatus, the plurality of movements of the body part of the player including the number of times that the player has changed the direction of movement of the body part and the movement speed of the head of the player comprises detecting, with the sensor operationally linked to the virtual image display apparatus, the plurality of movements of the body part of the player including a number of times per unit time that the player has changed the direction of movement of the body part,
distinguishing, based on whether the number of times that the player has changed the direction of movement of the body part is at least the threshold value and based on whether the movement speed of the head of the player satisfies the movement speed condition, whether each movement of the body part of the player is the predetermined movement associated with the information providing condition or is the normal movement associated with other gameplay comprises distinguishing, based on whether the number of times per unit time that the player has changed the direction of movement of the body part is at least the threshold value, whether each movement of the body part of the player is the predetermined movement associated with the information providing condition or is the normal movement associated with other gameplay, and
classifying the at least one first movement in the plurality of movements as the predetermined movement when the number of times that the player has changed the direction of movement of the body part is larger than the threshold value and the at least one second movement in the plurality of movements as the normal movement when the at least one second movement is not the predetermined movement comprises classifying the at least one first movement in the plurality of movements as the predetermined movement when the number of times per unit time that the player has changed the direction of movement of the body part is larger than the threshold value and at least one second movement in the plurality of movements as the normal movement when the at least one second movement is not the predetermined movement.

### Claim 9

A virtual image display method for displaying an image of virtual space using a virtual image display apparatus having a processor and a memory, the memory comprising a recording of an information providing condition of the virtual image display apparatus and to-be-provided information, the method comprising:
detecting, with a sensor operationally linked to the virtual image display apparatus, a plurality of movements of a body part of a player including a number of times that the player has changed a direction of movement of the body part and a movement speed of the head of the player, the body part comprising at least one of a head of the player and an eye of the player, and the sensor being at least one of the set of: a gyro sensor configured to measure movement of the head of the player, an acceleration sensor configured to measure movement of the head of the player, a geomagnetic sensor configured to measure movement of the head of the player and a line-of-sight sensor configured to measure movement of the eye of the player;
determining, based on each of the plurality of movements of the body part of the player, a position and a direction of the body part of the player during said plurality of movements;
distinguishing, based on whether the number of times that the player has changed the direction of movement of the body part is at least a threshold value and based on whether the movement speed of the head of the player satisfies a movement speed condition, whether each movement of the body part of the player is a predetermined movement associated with the information providing condition or is a normal movement associated with other gameplay, and classifying at least one first movement in the plurality of movements as a predetermined movement when the number of times that the player has changed the direction of movement of the body part is larger than the threshold value and at least one second movement in the plurality of movements as a normal movement when the at least one second movement is not the predetermined movement;
displaying, on a display operationally linked to the virtual image display apparatus, in accordance with the position and direction of the body part of the player, the image of the virtual space; and
with the virtual image display apparatus, triggering, upon classification of the at least one first movement of the body part of the player as the predetermined movement and upon satisfaction of the information providing condition, provision of the to-be-provided information to the player by displaying the to-be-provided information.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
based on the at least one second movement being classified as the normal movement, identifying, with the sensor operationally linked to the virtual image display apparatus, an object in the virtual space corresponding to the position and the direction of the body part of the player; and
with the sensor operationally linked to the virtual image display apparatus, determining, based on the plurality of movements of the body part of the player, whether the object in the virtual space has been selected by the player.

### Claim 11 (depends on 10)

The method of claim 10, wherein determining whether the object in the virtual space has been selected by the player comprises:
determining a gaze time over which a gaze of the player has been focused on the object; and
comparing the gaze time to a predetermined time and determining that the object has been selected when the gaze time is at least the predetermined time.

### Claim 12 (depends on 9)

The method of claim 9, wherein:
the virtual space includes a target object; and
providing the to-be-provided information to the player comprises identifying an area of the virtual space other than the target object and displaying the to-be-provided information in the area of the virtual space other than the target object.

### Claim 13 (depends on 9)

The method of claim 9, wherein:
the virtual space includes a moving object; and
providing the to-be-provided information to the player comprises displaying the to-be-provided information in association with the moving object and moving a display of the to-be-provided information along with the moving object.

### Claim 14 (depends on 9)

The method of claim 9, wherein:
the virtual image display apparatus is configured to execute a game, the game comprising a step of displaying the image of the virtual space; and
providing the to-be provided information to the player comprises providing a hint to the player related to a next step for clearing the game.

### Claim 15 (depends on 9)

The method of claim 9, wherein:
the virtual image display apparatus is configured to execute an application, the application comprising a step of displaying the image of the virtual space; and
providing the to-be-provided information to the player comprises displaying a menu item of the application.

### Claim 16 (depends on 9)

The method of claim 9, wherein:
detecting, with the sensor operationally linked to the virtual image display apparatus, the plurality of movements of the body part of the player including the number of times that the player has changed the direction of movement of the body part and the movement speed of the head of the player comprises detecting, with the sensor operationally linked to the virtual image display apparatus, the plurality of movements of the body part of the player including a number of times per unit time that the player has changed the direction of movement of the body part,
distinguishing, based on whether the number of times that the player has changed the direction of movement of the body part is at least the threshold value and based on whether the movement speed of the head of the player satisfies the movement speed condition, whether each movement of the body part of the player is the predetermined movement associated with the information providing condition or is the normal movement associated with other gameplay comprises distinguishing, based on whether the number of times per unit time that the player has changed the direction of movement of the body part is at least the threshold value, whether each movement of the body part of the player is the predetermined movement associated with the information providing condition or is the normal movement associated with other gameplay, and
classifying the at least one first movement in the plurality of movements as the predetermined movement when the number of times that the player has changed the direction of movement of the body part is larger than the threshold value and the at least one second movement in the plurality of movements as the normal movement when the at least one second movement is not the predetermined movement comprises classifying the at least one first movement in the plurality of movements as the predetermined movement when the number of times per unit time that the player has changed the direction of movement of the body part is larger than the threshold value and at least one second movement in the plurality of movements as the normal movement when the at least one second movement is not the predetermined movement.

### Claim 17

A virtual image display apparatus comprising a processor, a memory, and a sensor, and configured to display an image of virtual space on a display, the memory being configured to store an information providing condition of the virtual image display apparatus and being further configured to store to-be-provided information, the virtual image display apparatus configured to execute steps of:
detecting, with the sensor, a plurality of movements of a body part of a player including a number of times that the player has changed a direction of movement of the body part and a movement speed of the head of the player, the body part comprising at least one of a head of the player and an eye of the player, and the sensor being at least one of the set of: a gyro sensor configured to measure movement of the head of the player, an acceleration sensor configured to measure movement of the head of the player, a geomagnetic sensor configured to measure movement of the head of the player and a line-of-sight sensor configured to measure movement of the eye of the player;
determining, based on each of the plurality of movements of the body part of the player, a position and a direction of the body part of the player during said plurality of movements;
distinguishing, based on whether the number of times that the player has changed the direction of movement of the body part is at least a threshold value and based on whether the movement speed of the head of the player satisfies a movement speed condition, whether each movement of the body part of the player is a predetermined movement associated with the information providing condition or is a normal movement associated with other gameplay, and classifying at least one first movement in the plurality of movements as a predetermined movement when the number of times that the player has changed the direction of movement of the body part is larger than the threshold value and at least one second movement in the plurality of movements as a normal movement when the at least one second movement is not the predetermined movement;
displaying, on the display, in accordance with the position and direction of the body part of the player, the image of the virtual space; and
triggering, upon classification of the at least one first movement of the body part of the player as the predetermined movement and upon satisfaction of the information providing condition, provision, with the virtual image display apparatus, of the to-be-provided information to the player by displaying the to-be-provided information.

### Claim 18 (depends on 17)

The virtual image display apparatus of claim 17, wherein the memory is further configured to store at least one of:
instructions for executing a game, the game comprising a step of displaying the image of the virtual space, wherein providing the to-be provided information to the player comprises providing a hint to the player related to a next step for clearing the game; and
instructions for executing an application, the application comprising the step of displaying the image of the virtual space, wherein providing the to-be-provided information to the player comprises displaying a menu item of the application.