# US 2023/0274386 A1 · Systems and methods for digital display stabilization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274386 A1
- Document ID: US20230274386A1
- Status: Published application
- Kind: A1
- Application number: 17/652,756
- Filed: 2022-02-28
- Published: 2023-08-31
- Earliest claimed: 2022-02-28
- Assignees: Ford Global Technologies, LLC
- Applicants: Ford Global Technologies, LLC
- Inventors: David Herman; Yashanshu Jain
- Web page: https://patentgrep.com/patent/US20230274386A1

## CPC

- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06V 10/82: using neural networks
- G06F 18/253: {of extracted features}
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 40/18: Eye characteristics, e.g. of the iris
- G06V 40/193: {Preprocessing; Feature extraction}
- G06T 2207/30248: Vehicle exterior or interior
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The disclosure is generally directed to a vehicle, systems and methods for display stabilization including receiving a plurality of image data from sensors in a vehicle, receiving a plurality of measurements of road excitations from sensors in the vehicle, estimating a three-dimensional position of a driver or a passengers eyes based on the received plurality of image data, receiving apriori data from high definition maps, recorded accelerations of the vehicle, and stored data via a controller area network (CAN) bus, determining a prediction of motion of a display in the vehicle based on a fusion of the received image data, the plurality of measurements of road excitations and the apriori data, modeling the prediction of motion of the display in a convolutional neural network to form an initial estimate of a display stabilization position, and display computationally corrected images on the display based on the display stabilization position.

## Claims

### Claim 1

A method comprising:
receiving a plurality of image data from sensors in a vehicle;
receiving a plurality of measurements of road excitations from the sensors in the vehicle;
estimating a three-dimensional position of a eyes of a driver or passenger based on the plurality of image data;
receiving apriori data from high definition maps, recorded accelerations of the vehicle, and stored data via a controller area network (CAN) bus;
determining a prediction of motion of a display in the vehicle based on a fusion of the plurality of image data, the plurality of measurements of road excitations, and the apriori data; and
modeling the prediction of motion of the display in a convolutional neural network to form an initial estimate of a display stabilization position; and
displaying computationally corrected images on the display based on the display stabilization position and the three-dimensional position of the eyes of the driver of passenger.

### Claim 2 (depends on 1)

The method of claim 1 wherein the determining the prediction of motion of the display in the vehicle based on the fusion of the plurality of image data, the plurality of measurements of road excitations and the apriori data further comprises:
applying a neural network to predict motion of the display based on road accelerations over time using the apriori data.

### Claim 3 (depends on 2)

The method of claim 2 wherein applying the neural network for the prediction of motion of the display based on the road accelerations over time using the apriori data further comprises:
obtaining a confidence for the apriori data as a scaling factor, k, where high confidence results in a scale factor of 1 and a low confidence scale factor of 0 ignores the prediction of motion of the display.

### Claim 4 (depends on 1)

The method of claim 1 wherein the determining the prediction of motion of the display in the vehicle further comprises:
applying a limiting function to a maximum motion estimation to avoid erroneous edge data.

### Claim 5 (depends on 1)

The method of claim 1 wherein the plurality of image data includes captured camera images, captured driver state monitoring camera (DSMC) images, and the estimate of the three-dimensional position of the of the driver or passenger.

### Claim 6 (depends on 1)

The method of claim 1, wherein receiving the apriori data from the high definition maps, the recorded accelerations of the vehicle, and the stored data via the CAN bus further comprises:
computationally transforming the stored data over time to minimize vibration on elements that affect viewability of the display.

### Claim 7 (depends on 1)

The method of claim 1 wherein the determining the prediction of motion of the display in the vehicle based on the plurality of image data, the measurements of road excitations, and the apriori data further comprises:
fusing the prediction of motion of the display with prediction of motion of the sensors of the vehicle.

### Claim 8 (depends on 1)

The method of claim 1 wherein the determining the prediction of motion of the display in the vehicle based on the plurality of image data, the measurements of road excitations, and the apriori data further comprises:
correcting the prediction of motion of the display based on a number of viewers of the display.

### Claim 9 (depends on 1)

The method of claim 1 wherein the determining the prediction of motion of the display in the vehicle based on the plurality of image data, the measurements of road excitations, and the apriori data further comprises:
applying a transformation matrix to the display based on the prediction of motion of the display.

### Claim 10 (depends on 9)

The method of claim 9 wherein applying the transformation matrix to the display based on the prediction of motion of the display further comprises:
applying a temporal filtering to account for temporal motion of the image data tuned to a known response of the sensors to vehicle accelerations.

### Claim 11 (depends on 1)

The method of claim 1 further comprising:
receiving measurements from an accelerometer built into the display to measure accelerations that contribute to the prediction of motion of the display over time.

### Claim 12 (depends on 1)

The method of claim 1 further comprising:
receiving vibration data from internal microphones or audio output from speaker systems in the vehicle as input data, wherein predicting the motion of the display is based on the vibrational data, and wherein the motion of the display is a function of the vibration data including the audio output and vehicle accelerations.

### Claim 13

A system for a vehicle comprising:
a memory that stores computer-executable instructions;
a processor configured to access the memory and execute the computer-executable instructions to:
receive a plurality of image data from sensors in the vehicle;
receive a plurality of measurements of road excitations from the sensors in the vehicle;
estimate a three-dimensional position of a eyes of a driver or passenger based on the plurality of image data;
receive apriori data from high definition maps, recorded accelerations of the vehicle, and stored data via a controller area network (CAN) bus;
determine a prediction of motion of a display in the vehicle based on a fusion of the plurality of image data, the plurality of measurements of road excitations, and the apriori data;
model the prediction of motion of the display in a convolutional neural network to form an initial estimate of a display stabilization position; and
display computationally corrected images on the display based on the display stabilization position and the three-dimensional position of the eyes of the driver of passenger.

### Claim 14 (depends on 13)

The system of claim 13 wherein the processor is configured to execute instructions to:
apply a neural network to predict motion of the display based on road accelerations over time using the apriori data; and
obtain a confidence for the apriori data as a scaling factor, k, where high confidence results in a scale factor of 1 and a low confidence scale factor of 0 ignores the prediction of motion.

### Claim 15 (depends on 13)

The system of claim 13 wherein the processor is configured to execute instructions to:
computationally transform the stored data over time to minimize vibration on elements that affect viewability of the display.

### Claim 16 (depends on 13)

The system of claim 13 wherein the processor configured to execute instructions to:
fuse the prediction of motion of the display with prediction of motion of the sensors of the vehicle.

### Claim 17 (depends on 13)

The system of claim 13 wherein the processor configured to execute instructions to:
apply a transformation matrix to the display based on the prediction of motion of the display.

### Claim 18 (depends on 13)

The system of claim 13 wherein the processor is further configured to execute instructions to:
receive measurements from an accelerometer built into the display to measure accelerations that contribute to the prediction of motion of the display over time.

### Claim 19 (depends on 13)

The system of claim 13 wherein the processor is further configured to execute instructions to:
receive vibration data from internal microphones or audio output from speaker systems in the vehicle as input data to predict motion of the display, wherein the motion of the display is a function of the vibration data including the audio output and vehicle accelerations.

### Claim 20

A vehicle comprising:
sensors coupled to the vehicle, the sensors including cameras and radar;
a processor coupled to a memory, the processor configured to access the memory and execute instructions to:
receive a plurality of image data from the sensors in the vehicle;
receive a plurality of measurements of road excitations from the sensors in the vehicle;
estimate a three-dimensional position of a eyes of a driver or passenger based on the plurality of image data;
receive apriori data from high definition maps, recorded accelerations of the vehicle, and stored data via a controller area network (CAN) bus;
applying a limiting function to a maximum motion estimation to avoid erroneous edge data;
determine a prediction of motion of a display in the vehicle based on a fusion of the plurality of image data, the plurality of measurements of road excitations and the apriori data;
applying a transformation matrix to the display based on the prediction of motion of the display;
applying a temporal filtering to account for temporal motion of the image data tuned to a known response of the sensors to vehicle accelerations;
model the prediction of motion of the display in a convolutional neural network to form an initial estimate of a display stabilization position; and
display computationally corrected images on the display based on the display stabilization position and the three-dimensional position of the eyes of the driver of passenger.
