# US 10,482,669 B2 · Augmented virtual display

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,669 B2
- Document ID: US10482669B2
- Status: Granted
- Kind: B2
- Application number: 15/713,421
- Filed: 2017-09-22
- Published: 2018-03-29
- Granted: 2019-11-19
- Earliest claimed: 2016-09-23
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Mark B. Rober; Sawyer I. Cohen; Daniel Kurz; Tobias Holl; Benjamin B. Lyon; Peter Georg Meier; Jeffrey M. Riepling; Holly Gerhard
- Web page: https://patentgrep.com/patent/US10482669B2

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- B60K 35/00: Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/211: {producing three-dimensional [3D] effects, e.g. stereoscopic images}
- B60K 35/25: using haptic output
- B60K 35/26: using acoustic output
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- B60Q 9/00: Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 {, e.g. haptic signalling}
- B60W 10/04: including control of propulsion units
- B60W 10/18: including control of braking systems
- B60W 10/20: including control of steering systems
- B60W 10/22: including control of suspension systems
- B60W 50/08: Interaction between the driver and the control system
- G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00)
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06V 20/20: in augmented reality scenes
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- B60K 35/80: Arrangements for controlling instruments
- B60K 2360/177: Augmented reality
- B60R 2300/207: using multi-purpose displays, e.g. camera image and navigation or video on same display
- B60R 2300/60: characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
- B60W 2540/22: Psychological state; Stress level or workload
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR

## Abstract

A VR system for vehicles that may implement methods that address problems with vehicles in motion that may result in motion sickness for passengers. The VR system may provide augmented or virtual views that match visual cues with the physical motions that a passenger experiences. The VR system may project virtual content so that the content appears as a distant object stabilized or fixed in the external environment. The VR system may aid in productivity, as passengers may perform work while riding in the vehicle without experiencing motion sickness. In addition, the VR system may provide enhanced virtual experiences to passengers in moving vehicles by matching accelerations and motions of the vehicle to accelerations and motions in the virtual experiences.

## Claims

### Claim 1

A system, comprising:
a virtual reality (VR) controller in a vehicle, wherein the VR controller is configured to:
obtain inputs from one or more sources in the vehicle;
obtain display screen content from a user device of a passenger in the vehicle;
render virtual content including the display screen content from the user device based at least in part on the inputs from the one or more sources in the vehicle;
alter the virtual content according to one or more visual or audio accommodations to mitigate motion sickness of the passenger according to preference data for the passenger; and
send the virtual content to a VR device for display in a 3D virtual view to the passenger in the vehicle, wherein the display screen content from the user device is rendered and displayed to appear to the passenger to be at a fixed distance in front of the vehicle.

### Claim 2 (depends on 1)

The system as recited in claim 1, wherein the VR device includes:
a head-mounted device (HMD) worn by the passenger that displays the virtual content for viewing by the passenger; or
a projector that projects the virtual content to a window of the vehicle for viewing by the passenger.

### Claim 3 (depends on 1)

The system as recited in claim 1, wherein the user device comprises one or more of a notebook computer, a mobile device, a smartphone, or a tablet.

### Claim 4 (depends on 1)

The system as recited in claim 1, wherein the virtual content is displayed in a view of a real world scene in front of the vehicle to provide an augmented or mixed reality view to the passenger.

### Claim 5 (depends on 1)

The system as recited in claim 1, wherein the virtual content including the display screen content represents a virtual environment that replaces a view of a real world scene in front of the vehicle with a virtual reality view, wherein the virtual reality view includes visual cues that indicate motion of the vehicle in the virtual environment to the passenger.

### Claim 6 (depends on 5)

The system as recited in claim 5, wherein the VR controller is further configured to:
determine that the passenger is prone to or is exhibiting signs of motion sickness; and
upon determining that the passenger is prone to or is exhibiting signs of motion sickness, alter the virtual environment according to one or more accommodations to mitigate motion sickness.

### Claim 7 (depends on 6)

The system as recited in claim 6, wherein the VR controller is configured to determine that the passenger is prone to motion sickness according to the preference data for the passenger or input from the passenger.

### Claim 8 (depends on 6)

The system as recited in claim 6, wherein the VR controller is configured to determine that the passenger is exhibiting signs of motion sickness according to passenger sensor data obtained from one or more sensors that monitor the passenger.

### Claim 9 (depends on 6)

The system as recited in claim 6, wherein the one or more accommodations include:
displaying a ground plane beneath the passenger in the virtual reality view;
reducing a number of visual cues that indicate motion of the vehicle in the virtual environment;
slowing the visual cues that indicate motion of the vehicle in the virtual environment so that there is a 1:1 ratio between motion in the virtual environment and motion in the real environment;
providing visual, audio, or haptic indications of upcoming vehicle maneuvers;
reducing the virtual content displayed in the virtual reality view to provide a sparse visual environment;
placing an anchored object in the virtual environment that remains fixed relative to the real world;
providing a visual stimulus at stops of the vehicle to counter visual motion aftereffects; or
providing audio output that blocks real world noise.

### Claim 10 (depends on 1)

The system as recited in claim 1, wherein the one or more sources include:
one or more vehicle sensors, wherein the inputs from the vehicle sensors include one or more of depth map information from external vehicle sensors, localization information for the vehicle, video from one or more cameras on the vehicle, or motion and orientation information for the vehicle from an inertial-measurement unit (IMU) of the vehicle; and
one or more vehicle control systems, wherein the one or more vehicle control systems include one or more of a throttle control system, a brake control system, an active suspension control system, or a steering control system.

### Claim 11 (depends on 1)

The system as recited in claim 1, wherein the one or more sources include sources external to the vehicle that are accessed through wireless connections, wherein inputs from the external sources include one or more of world map data, 3D models of a local environment, information about objects or features in the local environment, video streams, or audio.

### Claim 12 (depends on 1)

The system as recited in claim 1, wherein the one or more sources include an inertial-measurement unit (IMU) of the VR device, wherein inputs from the IMU include motion and orientation information for the passenger's head.

### Claim 13 (depends on 1)

The system as recited in claim 1, wherein the vehicle is an autonomous vehicle.

### Claim 14 (depends on 1)

The system as recited in claim 1, wherein, in rendering the virtual content, the VR controller synchronizes motions and accelerations of the virtual content with motions and accelerations of the vehicle based on the inputs from the one or more sources in the vehicle.

### Claim 15

A method, comprising:
obtaining, by a virtual reality (VR) system in a vehicle, inputs from one or more sources in the vehicle;
obtaining, by the VR system, display screen content from a user device of a passenger in the vehicle;
rendering, by the VR system, virtual content including the display screen content from the user device based at least in part on the inputs from the one or more sources in the vehicle;
altering the virtual content according to one or more visual or audio accommodations to mitigate motion sickness of the passenger according to preference data for the passenger; and
displaying, by the VR system, the virtual content in a 3D virtual view to a passenger in the vehicle, wherein the display screen content appears to the passenger to be at a fixed distance in front of the vehicle.

### Claim 16 (depends on 15)

The method as recited in claim 15, wherein the VR system includes a VR controller and a VR device, wherein the VR device is one of a head-mounted device (HMD) configured to be worn by the passenger and to display or project the virtual content for viewing by the passenger or a projector configured to project the virtual content to a window of the vehicle for viewing by the passenger.

### Claim 17 (depends on 15)

The method as recited in claim 15, wherein the virtual content is displayed in a view of a real world scene in front of the vehicle to provide an augmented or mixed reality view to the passenger.

### Claim 18 (depends on 15)

The method as recited in claim 15, wherein the virtual content including the display screen content represents a virtual environment that replaces a view of a real world scene in front of the vehicle with a virtual reality view, wherein the virtual reality view includes visual cues to indicate motion of the vehicle in the virtual environment to the passenger.

### Claim 19 (depends on 15)

The method as recited in claim 15, further comprising:
determining that the passenger is prone to or is exhibiting signs of motion sickness, wherein altering the virtual content according to the one or more visual or audio accommodations to mitigate motion sickness is performed upon determining that the passenger is prone to or is exhibiting signs of motion sickness.

### Claim 20 (depends on 15)

The method as recited in claim 15, wherein rendering the virtual content comprises synchronizing motions and accelerations of the virtual content with motions and accelerations of the vehicle based on the inputs from the one or more sources in the vehicle.