# US 12,008,722 B2 · Multi-plane augmented reality image generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,722 B2
- Document ID: US12008722B2
- Status: Granted
- Kind: B2
- Application number: 17/736,158
- Filed: 2022-05-04
- Published: 2023-11-09
- Granted: 2024-06-11
- Earliest claimed: 2022-05-04
- Assignees: Ford Global Technologies, LLC
- Applicants: Ford Global Technologies, LLC
- Inventors: Jian Wan; Anthony Gerald King
- Web page: https://patentgrep.com/patent/US12008722B2

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/211: {producing three-dimensional [3D] effects, e.g. stereoscopic images}
- B60K 35/233: controlling the size or position in display areas of virtual images depending on the condition of the vehicle or the driver
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- B60K 35/50: Instruments characterised by their means of attachment to or integration in the vehicle (B60K35/231 takes precedence)
- B60K 35/654: {the user being the driver}
- B60K 35/81: for controlling displays
- G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- B60K 35/23: Head-up displays [HUD] (optical aspects of head-up displays G02B27/01)
- B60K 35/60: Instruments characterised by their location or relative disposition in or on vehicles (arrangements of lighting devices on dashboards B60Q3/10)
- B60K 35/65: Instruments specially adapted for specific vehicle types or users, e.g. for left- or right-hand drive
- B60K 2360/166: Navigation
- B60K 2360/167: Vehicle dynamics information
- B60K 2360/177: Augmented reality
- B60K 2360/21: using cameras
- B60K 2360/25: using filters
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

A set of training images are input into a neural network that outputs a pixel-wise phase matrix identifying pixels in the training images. Based on the pixel-wise phase matrix, a spatial light modulator (SLM) is actuated to output, onto a vehicle windshield, an augmented reality (AR) image including a plurality of sub-images each output in one of a plurality of focal planes. Each training image corresponds to one respective sub-image. A feedback image of the AR image is obtained via an image sensor. An offset is determined based on comparing the training images to the feedback image. Parameters of a loss function are updated based on the offset, and the updated parameters are provided to the neural network to obtain an updated offset.

## Claims

### Claim 1

A system, comprising:
a spatial light modulator (SLM) arranged to output, onto a vehicle windshield, an augmented reality (AR) image including a plurality of sub-images each output in one of a plurality of focal planes;
an image sensor positioned to obtain a feedback image of the AR image; and
a computer including a processor and a memory, the memory storing instructions executable by the processor programmed to:
input a set of training images into a neural network that outputs a pixel-wise phase matrix identifying pixels in the training images, wherein each training image corresponds to one respective sub-image;
actuate the SLM to output the AR image based on the pixel-wise phase matrix;
determine an offset based on comparing the training images to the feedback image; and
update parameters of a loss function based on the offset and provide the updated parameters to the neural network to obtain an updated offset.

### Claim 2 (depends on 1)

The system of claim 1, wherein the instructions further include instructions to update parameters of the loss function until the updated offset is less than a predetermined threshold.

### Claim 3 (depends on 2)

The system of claim 2, wherein the neural network is trained to output the pixel-wise phase matrix calibrated to the vehicle windshield and the SLM when the updated offset is less than the predetermined threshold.

### Claim 4 (depends on 1)

The system of claim 1, wherein the image sensor is spaced from the vehicle windshield to correspond to an expected pose of an occupant.

### Claim 5 (depends on 1)

The system of claim 1, wherein the instructions further include instructions to select the focal planes for the sub-images based on a user input.

### Claim 6 (depends on 1)

The system of claim 1, wherein each focal plane is defined by a respective distance from an expected pose of an occupant.

### Claim 7 (depends on 1)

The system of claim 1, wherein the instructions further include instructions to, while training the neural network, actuate a projector to provide the training images to the SLM.

### Claim 8 (depends on 1)

The system of claim 1, wherein the instructions further including instructions to, while training the neural network, select the training images from a plurality of training images.

### Claim 9 (depends on 1)

The system of claim 1, wherein the instructions further include instructions to:
after training the neural network, generate a three-dimensional (3D) image based on vehicle operation data; and
determine, via the trained neural network, a calibrated pixel-wise phase matrix for the 3D image.

### Claim 10 (depends on 9)

The system of claim 9, wherein the instructions further include instructions to, upon generating a set of masked images from the 3D image, input the masked images to the trained neural network that outputs the calibrated pixel-wise phase matrix for the 3D image.

### Claim 11 (depends on 10)

The system of claim 10, wherein the instructions further include instructions to generate the masked images by masking a plurality of pixels in the 3D image based on the focal planes, each masked pixel being a pixel of the 3D image having a depth corresponding to one of the focal planes.

### Claim 12 (depends on 9)

The system of claim 9, wherein the instructions further include instructions to actuate the SLM to output the AR image based on the calibrated pixel-wise phase matrix.

### Claim 13 (depends on 12)

The system of claim 12, wherein the instructions further include instructions to actuate a projector to provide the 3D image to the SLM.

### Claim 14

A method, comprising:
inputting a set of training images into a neural network that outputs a pixel-wise phase matrix identifying pixels in the training images;
based on the pixel-wise phase matrix, actuating a spatial light modulator (SLM) to output, onto a vehicle windshield, an augmented reality (AR) image including a plurality of sub-images each output in one of a plurality of focal planes, wherein each training image corresponds to one respective sub-image;
obtaining a feedback image of the AR image via an image sensor;
determining an offset based on comparing the training images to the feedback image; and
updating parameters of a loss function based on the offset and providing the updated parameters to the neural network to obtain an updated offset.

### Claim 15 (depends on 14)

The method of claim 14, further comprising updating parameters of the loss function until the updated offset is less than a predetermined threshold.

### Claim 16 (depends on 15)

The method of claim 15, wherein the neural network is trained to output the pixel-wise phase matrix calibrated to the vehicle windshield and the SLM when the updated offset is less than the predetermined threshold.

### Claim 17 (depends on 14)

The method of claim 14, wherein the image sensor is spaced from the vehicle windshield to correspond to an expected pose of an occupant.

### Claim 18 (depends on 14)

The method of claim 14, further comprising selecting the focal planes for the sub-images based on a user input.

### Claim 19 (depends on 14)

The method of claim 14, wherein each focal plane is selected based on a respective distance from an expected pose of an occupant.

### Claim 20 (depends on 14)

The method of claim 14, further comprising, while training the neural network, actuating a projector to provide the training image to the SLM.