# US 12,225,291 B2 · System, method, and computer program product for online sensor motion compensation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,225,291 B2
- Document ID: US12225291B2
- Status: Granted
- Kind: B2
- Application number: 17/591,049
- Filed: 2022-02-02
- Published: 2023-08-03
- Granted: 2025-02-11
- Earliest claimed: 2022-02-02
- Assignees: Ford Global Technologies, LLC
- Applicants: Ford Global Technologies, LLC
- Inventors: Casey Sennott
- Web page: https://patentgrep.com/patent/US12225291B2

## CPC

- H04N 23/6812: {based on additional sensors, e.g. acceleration sensors}
- G01L 5/1627: of strain gauges (using piezoresistors G01L5/162)
- G06N 20/00: Machine learning
- G06T 5/80: Geometric correction
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 7/85: {Stereo camera calibration}
- H04N 17/002: {for television cameras}
- H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory}
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G03B 17/561: {Support related camera accessories}
- G06N 3/08: Learning methods
- G06T 2207/10012: Stereo images
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20201: Motion blur correction
- G06T 2207/30244: Camera pose
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01L: MEASURING FORCE, STRESS, TORQUE, WORK, MECHANICAL POWER, MECHANICAL EFFICIENCY, OR FLUID PRESSURE (weighing G01G)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D)

## Abstract

Disclosed herein are system, method, and computer program product embodiments for online sensor motion compensation. For example, the method includes: applying a random mechanical excitation to a support structure, wherein a plurality of image capture devices and a plurality of sets of strain gauges are coupled to the support structure; measuring, with each set of strain gauges of the plurality of sets of strain gauges, simultaneous to the application of the random mechanical excitation, a strain; capturing, with each image capture device of the plurality of image capture devices, simultaneous to the application of the random mechanical excitation, a series of images of a calibration target; and generating, based on the strain and the series of images, a mapping between the strain and a displacement between the plurality of image capture devices.

## Claims

### Claim 1

A method for a vehicle, comprising:
applying, via a vibration exciter, a random mechanical excitation to a support structure of the vehicle over a period of time, wherein a plurality of image capture devices and a plurality of sets of strain gauges are coupled to the support structure, and wherein the plurality of image capture devices are calibrated to each other before applying the random mechanical excitation to the support structure;
measuring, with each set of strain gauges of the plurality of sets of strain gauges, simultaneous to the application of the random mechanical excitation to the support structure, a strain at a location of each set of strain gauges on the support structure over the period of time;
capturing, with each image capture device of the plurality of image capture devices, simultaneous to the application of the random mechanical excitation to the support structure, image data associated with a series of images of a calibration target; and
generating, with at least one processor, based on the strain measured at a location of each set of strain gauges on the support structure over the period of time by each set of strain gauges and the series of images of the calibration target captured by each image capture device, a mapping between the strain measured by each set of strain gauges and a displacement between the plurality of image capture devices.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
capturing, with each image capture device, one or more further images;
measuring, with each set of strain gauges, simultaneous to capturing the one or more images with each image capture device, a further strain at the location of each set of strain gauges on the support structure;
applying, with the at least one processor, the mapping to the further strain measured by each set of strain gauges to generate a motion compensation parameter associated with a predicted displacement between the plurality of image capture devices for calibrating the one or more further images captured by each image capture device; and
correcting, with the at least one processor, using the motion compensation parameter, a distortion between the one or more further images captured by each image capture device.

### Claim 3 (depends on 1)

The method of claim 1, wherein each set of strain gauges includes three strain gauges, and wherein the three strain gauges are in a delta-rosette configuration.

### Claim 4 (depends on 1)

The method of claim 1, wherein the plurality of image capture devices includes at least three image capture devices.

### Claim 5 (depends on 1)

The method of claim 1, wherein the support structure includes a first face facing in a first direction and a second face facing in a second direction different than the first direction, wherein at least one first set of strain gauges of the plurality of sets of strain gauges is coupled to the first face of the support structure, and wherein at least one second set of strain gauges of the plurality of sets of strain gauges is coupled to the second face of the support structure.

### Claim 6 (depends on 1)

The method of claim 1, wherein the mapping includes a machine learning model, and wherein generating the mapping includes training the machine learning model using the strain measured over the period of time by each set of strain gauges and the series of images captured by each image capture device to generate a predicted displacement between the plurality of image capture devices.

### Claim 7 (depends on 1)

The method of claim 1, wherein the strain at the location of each set of strain gauges on the support structure over the period of time is measured as a series of strain tensors corresponding to the series of images of the calibration target, and wherein each strain tensor of the series of strain tensors is measured with six degrees of freedom.

### Claim 8

A system for a vehicle, comprising:
a support structure mounted on the vehicle;
a plurality of image capture devices coupled to the support structure;
a plurality of sets of strain gauges coupled to the support structure;
a vibration exciter configured to apply a random mechanical excitation to the support structure;
a memory; and
at least one processor coupled to the memory and configured to:
receive a strain measured over a period of time, at a location of each set of strain gauges of the plurality of sets of strain gauges coupled to the support structure, simultaneous to an application of the random mechanical excitation to the support structure over the period of time, wherein the plurality of image capture devices are calibrated to each other before the random mechanical excitation is applied to the support structure;
receive image data associated with a series of images of a calibration target captured, by each image capture device of the plurality of image capture devices, simultaneous to the application of the random mechanical excitation to the support structure; and
generate, based on the strain measured over the period of time by each set of strain gauges and the image data associated with the series of images of the calibration target captured by each image capture device, a mapping between the strain measured by each set of strain gauges and a displacement between the plurality of image capture devices.

### Claim 9 (depends on 8)

The system of claim 8, wherein the at least one processor coupled to the memory is further configured to:
receive further image data associated with one or more further images captured by each image capture device;
receive a further strain measured, by each set of strain gauges, simultaneous to the capture of the one or more images by each image capture device;
apply the mapping to the further strain measured by each set of strain gauges to generate a motion compensation parameter associated with a predicted displacement between the plurality of image capture devices for calibrating the one or more further images captured by each image capture device; and
correct, using the motion compensation parameter, a distortion between the one or more further images captured by each image capture device.

### Claim 10 (depends on 8)

The system of claim 8, wherein each set of strain gauges includes three strain gauges, and wherein the three strain gauges are in a delta-rosette configuration.

### Claim 11 (depends on 8)

The system of claim 8, wherein the plurality of image capture devices includes at least three image capture devices.

### Claim 12 (depends on 8)

The system of claim 8, wherein the support structure includes a first face facing in a first direction and a second face facing in a second direction different than the first direction, wherein at least one first set of strain gauges of the plurality of sets of strain gauges is coupled to the first face of the support structure, and wherein at least one second set of strain gauges of the plurality of sets of strain gauges is coupled to the second face of the support structure.

### Claim 13 (depends on 8)

The system of claim 8, wherein the mapping includes a machine learning model, and wherein generating the mapping includes training the machine learning model using the strain measured over the period of time by each set of strain gauges and the image data associated with the series of images captured by each image capture device to generate a predicted displacement between the plurality of image capture devices.

### Claim 14 (depends on 8)

The system of claim 8, wherein the strain at the location of each set of strain gauges on the support structure over the period of time is measured as a series of strain tensors corresponding to the series of images of the calibration target, and wherein each strain tensor of the series of strain tensors is measured with six degrees of freedom.

### Claim 15

A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device of a vehicle, cause the at least one computing device to perform operations comprising:
applying, via a vibration exciter, a random mechanical excitation to a support structure connected to the vehicle;
receiving a strain measured over a period of time, at a location of each set of strain gauges of a plurality of sets of strain gauges coupled to the support structure, simultaneous to an application of the random mechanical excitation to the support structure over the period of time, wherein a plurality of image capture devices are coupled to the support structure, and wherein the plurality of image capture devices are calibrated to each other before the random mechanical excitation is applied to the support structure;
receiving image data associated with a series of images of a calibration target captured, by each image capture device of the plurality of image capture devices, simultaneous to the application of the random mechanical excitation to the support structure;
generating, based on the strain measured over the period of time by each set of strain gauges and the image data associated with the series of images of the calibration target captured by each image capture device, a mapping between the strain measured by each set of strain gauges and a displacement between the plurality of image capture devices;
receiving further image data associated with one or more further images captured by each image capture device;
receiving a further strain measured, by each set of strain gauges, simultaneous to the capture of the one or more images by each image capture device;
applying the mapping to the further strain measured by each set of strain gauges to generate a motion compensation parameter associated with a predicted displacement between the plurality of image capture devices for calibrating the one or more further images captured by each image capture device; and
correcting, using the motion compensation parameter, a distortion between the one or more further images captured by each image capture device.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein each set of strain gauges includes three strain gauges, and wherein the three strain gauges are in a delta-rosette configuration.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the plurality of image capture devices includes at least three image capture devices.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the support structure includes a first face facing in a first direction and a second face facing in a second direction different than the first direction, wherein at least one first set of strain gauges of the plurality of sets of strain gauges is coupled to the first face of the support structure, and wherein at least one second set of strain gauges of the plurality of sets of strain gauges is coupled to the second face of the support structure.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the mapping includes a machine learning model, and wherein generating the mapping includes training the machine learning model using the strain measured over the period of time by each set of strain gauges and the image data associated with the series of images captured by each image capture device to generate a predicted displacement between the plurality of image capture devices.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the strain at the location of each set of strain gauges on the support structure over the period of time is measured as a series of strain tensors corresponding to the series of images of the calibration target, and wherein each strain tensor of the series of strain tensors is measured with six degrees of freedom.