# US 11,907,412 B2 · Contextual spectator inclusion in a virtual reality experience

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,412 B2
- Document ID: US11907412B2
- Status: Granted
- Kind: B2
- Application number: 17/034,353
- Filed: 2020-09-28
- Published: 2022-03-31
- Granted: 2024-02-20
- Earliest claimed: 2020-09-28
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Zachary A. Silverstein; Cesar Augusto Rodriguez Bravo; Hemant Kumar Sivaswamy; Carlos E Aguilera
- Web page: https://patentgrep.com/patent/US11907412B2

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes
- A63F 13/212: using sensors worn by the player, e.g. for measuring heart beat or leg activity
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/25: Output arrangements for video game devices
- A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
- A63F 13/5255: according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
- A63F 13/86: Watching games played by other players
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- A63F 2300/8082: Virtual reality
- G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR

## Abstract

A method of generating a virtual reality (“VR”) experience includes detecting, using external sensors, a real-world spectator that is in a spectator space that is adjacent to a user that is wearing a VR headset and experiencing a VR space. The method also includes detecting a real-world physical state of the spectator, and rendering, in response to detecting, the spectator in the VR space as an avatar that reflects the real-world physical state of the spectator.

## Claims

### Claim 1

A method of generating a virtual reality (“VR”) experience, the method comprising:
detecting, using external sensors, a real-world spectator that is in a spectator space that is adjacent to a user that is wearing a VR headset and experiencing a VR space;
detecting a real-world physical state of the spectator comprising sounds of the spectator;
rendering, in response to detecting the real-world physical state of the spectator, the spectator in the VR space as an avatar that reflects the real-world physical state of the spectator;
rendering a plurality of VR fans in the VR space;
boosting experience points of the user based on sounds from the spectator; and
transforming the sounds of the spectator to make the sounds sound like they are being made by at least some of the plurality of VR fans;
wherein the real-world physical state of the spectator reveals a real-world emotional state of the spectator;
wherein the avatar reflects the real-world emotional state of the spectator; and
wherein at least some of the plurality of VR fans reflect the real-world movements of the spectator, facial expressions of the spectator, or gestures of the spectator.

### Claim 2 (depends on 1)

The method of claim 1, wherein detecting the real-world physical state of the spectator is selected from the group consisting of detecting gestures of the spectator, detecting facial expressions of the spectator, detecting utterances of the spectator, and detecting noises of the spectator.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
the real-world physical state of the spectator includes apparel worn by the spectator; and
the avatar is rendered to reflect a color and/or a pattern of the apparel.

### Claim 4 (depends on 1)

The method of claim 1, wherein a form of the avatar is a non-human entity.

### Claim 5

A virtual reality (“VR”) system comprising:
a VR controller including one or more processors and a computer-readable storage medium coupled to the one or more processors storing program instructions, the VR controller being configured to generate a VR experience;
a VR headset communicatively connected to the VR controller that is configured to display a VR space to a user;
a sensor communicatively connected to the VR controller that is configured to monitor a spectator space proximate to the user;
a display communicatively connected to the VR controller that is viewable from the spectator space; and
boosting experience points of the user based on sounds from the spectator;
wherein the program instructions, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
detecting, using the sensor, a first person in the spectator space;
analyzing the first person to determine whether the first person is a spectator or a bystander based on whether the first person is engaged in another activity that is unrelated to the VR system;
prompting, in response to determining that the first person is a spectator because the first person is not engaged in the another activity, the first person to opt-in to the VR experience; and
rendering, in response to the first person opting-in, an avatar in the VR space that represents the spectator.

### Claim 6 (depends on 5)

The VR system of claim 5, wherein the sensor comprises:
a first plurality of cameras mounted on the VR headset; and
a second plurality of cameras that are mounted off of the VR headset.

### Claim 7 (depends on 5)

The VR system of claim 5, wherein rendering the avatar in the VR space that represents the spectator comprises:
rendering an entrance animation in the VR space involving the avatar in response to the first person opting-in.

### Claim 8 (depends on 5)

The VR system of claim 5, wherein rendering the avatar in the VR space that represents the spectator comprises:
rendering an exit animation in the VR space involving the avatar in response to the first person leaving the spectator space.

### Claim 9 (depends on 8)

The VR system of claim 8, wherein rendering the avatar in the VR space that represents the spectator comprises:
rendering an entrance animation in the VR space involving the avatar in response to the first person reentering the spectator space.

### Claim 10 (depends on 5)

The VR system of claim 5, wherein the program instructions, when executed by the one or more processors, cause the one or more processors to perform operations further comprising:
determining that the first person is no longer a spectator based on eye movements of the first person; and
rendering an exit animation in the VR space involving the avatar in response to determining that the first person is no longer a spectator.

### Claim 11 (depends on 5)

The VR system of claim 5, wherein the program instructions, when executed by the one or more processors, cause the one or more processors to perform further operations comprising:
detecting, using the sensor, a second person in the spectator space;
analyzing the second person to determine whether the second person is a spectator or a bystander based on whether the second person is engaged in another activity that is unrelated to the VR system;
determining that the second person is a bystander in response to the second person being engaged in the another activity; and
declining to prompt, in response to determining that the second person is a bystander, the second person to opt-in to the VR experience.

### Claim 12

A method of generating a virtual reality (“VR”) experience, the method comprising:
detecting, using external sensors, a real-world spectator that is in a spectator space that is adjacent to a user that is wearing a VR headset and experiencing a VR space;
analyzing the VR space to determine a VR context, wherein analyzing the VR space to determine the VR context comprises analyzing music in the VR experience;
rendering, in response to analyzing, the spectator in the VR space as an avatar that reflects the VR context;
detecting a real-world physical state of the spectator comprising sounds of the spectator;
rendering, in response to detecting the real-world physical state of the spectator, the spectator in the VR space as an avatar that reflects the real-world physical state of the spectator;
boosting experience points of the user based on sounds from the spectator; and
recording a portion of the VR experience in response to a dramatic and/or loud expression by the spectator.

### Claim 13 (depends on 12)

The method of claim 12, wherein a form of the avatar is a non-human entity.

### Claim 14 (depends on 12)

The method of claim 12, wherein analyzing the VR space to determine the VR context further comprises:
analyzing the VR space for color themes, geography, wildlife, characters, architecture, vehicles, apparel, language, speech patterns, time period, and/or fictional beings.

### Claim 15 (depends on 12)

The method of claim 12, wherein analyzing the VR space to determine the VR context comprises:
analyzing a description of the VR space.

### Claim 16 (depends on 12)

The method of claim 12, wherein rendering the spectator in the VR space comprises:
selecting an avatar from a predetermined list of avatars for the VR context.

### Claim 17 (depends on 12)

The method of claim 12, wherein rendering the spectator in the VR space comprises:
rendering an entrance animation in the VR space involving the avatar.

### Claim 18 (depends on 12)

The method of claim 12, wherein rendering the spectator in the VR space comprises:
rendering an exit animation in the VR space involving the avatar in response to the real-world spectator leaving the spectator space.