# US 2020/0286398 A1 · Virtual reality system and virtual reality method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0286398 A1
- Document ID: US20200286398A1
- Status: Published application
- Kind: A1
- Application number: 16/750,296
- Filed: 2020-01-23
- Published: 2020-09-10
- Earliest claimed: 2019-03-08
- Assignees: Toyota Jidosha Kabushiki Kaisha; Nikon Corporation
- Applicants: Toyota Jidosha Kabushiki Kaisha; Nikon Corporation
- Inventors: Naoya INOUE; Yasumasa NAKAJIMA
- Web page: https://patentgrep.com/patent/US20200286398A1

## CPC

- G09B 9/052: characterised by provision for recording or measuring trainee's performance (devices for psychotechnics, e.g. for vehicle drivers A61B5/16, A61B5/18)
- H04N 21/41422: {located in transportation means, e.g. personal vehicle (arrangements specially adapted for transportation systems in broadcast systems H04H20/62)}
- A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes
- A63F 13/212: using sensors worn by the player, e.g. for measuring heart beat or leg activity
- A63F 13/25: Output arrangements for video game devices
- A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
- A63F 13/5255: according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
- A63F 13/803: Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks
- G02B 27/0172: {characterised by optical features}
- G02B 27/0179: {Display position adjusting means not related to the information to be displayed}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G09B 9/05: the view from a vehicle being simulated (G09B9/052, G09B9/058 take precedence)
- H04N 21/42203: {sound input device, e.g. microphone}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/4302: {Content synchronisation processes, e.g. decoder synchronisation}
- H04N 21/816: {involving special video data, e.g 3D video}
- G02B 2027/0183: {Adaptation to parameters characterising the motion of the vehicle}
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A virtual reality system allows a user to experience simulated vehicle traveling. The virtual reality system includes an image capturing device mounted on a vehicle and configured to capture an image of the surroundings of the vehicle during traveling of the vehicle and generate video data, an acquisition device mounted on the vehicle and configured to acquire behavior of the vehicle during traveling of the vehicle and generate behavior data based on information on the behavior, a computer configured to store the video data received from the image capturing device and the behavior data received from the acquisition device, and time-synchronize the video data and the behavior data, an HMD configured to play back the video data, and a seat device configured to play back the behavior data.

## Claims

### Claim 1

A virtual reality system of allowing a user to experience simulated vehicle traveling, the virtual reality system comprising:
an image capturing device mounted on a vehicle and configured to capture an image of surroundings of the vehicle during traveling of the vehicle and generate video data;
an acquisition device mounted on the vehicle and configured to acquire behavior of the vehicle during traveling of the vehicle and generate behavior data based on information on the behavior;
a computer configured to store the video data received from the image capturing device and the behavior data received from the acquisition device, and time-synchronize the video data and the behavior data;
a first playback device configured to play back the video data; and
a second playback device configured to play back the behavior data.

### Claim 2 (depends on 1)

The virtual reality system according to claim 1, wherein:
the acquisition device is configured to acquire meter information of the vehicle during traveling of the vehicle and generate meter display data based on the meter information;
the computer is configured to store the meter display data received from the acquisition device, time-synchronize the meter display data with the video data and the behavior data; and
the first playback device is configured to play back the meter display data.

### Claim 3 (depends on 1)

The virtual reality system according to claim 1, wherein the image capturing device is configured to capture the image of the surroundings of the vehicle in a range of 360 degrees.

### Claim 4 (depends on 1)

The virtual reality system according to claim 1, wherein the acquisition device is configured to acquire the behavior of the vehicle from an in-vehicle network of the vehicle.

### Claim 5 (depends on 1)

The virtual reality system according to claim 1, wherein:
the image capturing device includes a microphone and is configured to collect sound in a cabin of the vehicle via the microphone during traveling of the vehicle and generate sound data based on the sound; and
the image capturing device is configured to associate the video data with the sound data and transmit the video data and the sound data to the computer.

### Claim 6 (depends on 1)

The virtual reality system according to claim 1, wherein the image capturing device is configured to add a recording start time of the image to the video data.

### Claim 7 (depends on 5)

The virtual reality system according to claim 5, wherein the image capturing device is configured to add a recording start time of the sound to the sound data.

### Claim 8 (depends on 1)

The virtual reality system according to claim 1, wherein the acquisition device is configured to add a recording start time of the behavior to the behavior data.

### Claim 9 (depends on 8)

The virtual reality system according to claim 8, wherein the behavior data includes changes with time in longitudinal acceleration, lateral acceleration, vertical acceleration, a roll angle, a pitch angle, and a yaw angle of the vehicle from a start to an end of the recording.

### Claim 10 (depends on 2)

The virtual reality system according to claim 2, wherein the acquisition device is configured to add a recording start time of the meter information to the meter display data.

### Claim 11 (depends on 1)

The virtual reality system according to claim 1, wherein the first playback device is a head-mounted display.

### Claim 12 (depends on 1)

The virtual reality system according to claim 1, wherein the second playback device is a seat device.

### Claim 13

A virtual reality method of allowing a user to experience simulated vehicle traveling, the virtual reality method comprising:
by an image capturing device mounted on a vehicle, capturing an image of surroundings of the vehicle during traveling of the vehicle and generating video data, and, by an acquisition device mounted on the vehicle, acquiring behavior of the vehicle during traveling of the vehicle and generating behavior data;
storing, by a computer, the video data received from the image capturing device and the behavior data received from the acquisition device;
time-synchronizing, by the computer, the video data and the behavior data; and
playing back, by a first playback device, the video data, and playing back, by a second playback device, the behavior data.
