# US 11,445,169 B2 · Camera image and sensor fusion for mirror replacement system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,445,169 B2
- Document ID: US11445169B2
- Status: Granted
- Kind: B2
- Application number: 16/944,338
- Filed: 2020-07-31
- Published: 2021-02-04
- Granted: 2022-09-13
- Earliest claimed: 2019-08-01
- Assignees: Stoneridge Electronics AB
- Applicants: Orlaco Products B.V.
- Inventors: Milan Gavrilovic; Frans Kievith; Bárbara Martín Leung; Annette Lef; Parinaz Kasebzadeh
- Web page: https://patentgrep.com/patent/US11445169B2

## CPC

- H04N 17/002: {for television cameras}
- G01C 9/00: Measuring inclination, e.g. by clinometers, by levels
- G01C 19/00: Gyroscopes; Turn-sensitive devices using vibrating masses; Turn-sensitive devices without moving masses; Measuring angular rate using gyroscopic effects
- G01P 3/42: Devices characterised by the use of electric or magnetic means (G01P3/66 takes precedence; measuring electric or magnetic values in general G01R)
- G01P 13/00: Indicating or recording presence, absence, or direction, of movement (electric switches H01H; counting moving objects G06M7/00)
- G01P 15/08: with conversion into electric or magnetic values
- G01P 15/18: in two or more dimensions
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- H04N 5/145: {Movement estimation (for video coding H04N19/51)}
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- B60R 2300/101: using cameras with adjustable capturing direction
- B60R 2300/402: Image calibration
- B60R 2300/8046: for replacing a rear-view mirror system
- G06T 2207/20081: Training; Learning
- G06T 2207/30244: Camera pose
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G01P: MEASURING LINEAR OR ANGULAR SPEED, ACCELERATION, DECELERATION, OR SHOCK; INDICATING PRESENCE, ABSENCE, OR DIRECTION, OF MOVEMENT (measuring or recording blood flow A61B5/02, A61B8/06; monitoring speed or deceleration of electrically-propelled vehicles B60L3/00; vehicle lighting systems adapted to indicate speed B60Q1/54; determining position or course in navigation, measuring ground distance in geodesy or surveying G01C; combined measuring devices for measuring two or more variables of movement G01C23/00; measuring velocity of sound G01H; measuring velocity of light G01J7/00; measuring direction or velocity of solid objects by reception or emission of radiowaves or other waves and based on propagation effects, e.g. Doppler effect, propagation time, direction of propagation, G01S; measuring speed of nuclear radiation G01T; measuring acceleration of gravity G01V; {measuring or recording the speed of trains B61L23/00; speed indicators incorporated in motor vehicles B60K35/00; measuring frequency or phase G01R; traffic control G08G})
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR

## Abstract

A measurement system for a vehicle including a first camera defining a field of view having a corresponding optical axis, and a motion detection sensor mechanically fixed to the first camera such that the motion detection sensor is configured to detect motion of the optical axis.

## Claims

### Claim 1

A measurement system for a vehicle comprising:
a first camera defining a field of view having a corresponding optical axis;
a motion detection sensor mechanically connected to an application specific integrated circuit (ASIC) of the first camera such that motion of the ASIC corresponds to motion of the optical axis in a one to one fashion and the motion detection sensor is configured to detect motion of the optical axis, the ASIC being configured to output a fused data set comprising a combination of image data generated by the first camera and motion data generated by the motion detection sensor;
a controller in communication with the ASIC, the controller including at least a blockage detection system and being configured to receive a motion sensor output and a camera output, and configured to fuse the motion sensor output and the camera output thereby creating a fused data set, the controller being configured to receive the fused data set, the controller being configured to analyze the fused data set using a machine learning based analysis, and the machine learning based analysis being configured to determine an orientation of the camera and alert an operator in response to the first camera facing an incorrect direction; and
wherein the controller is configured to provide the fused data set to the blockage detection system, and wherein the blockage detection system is configured to compare image based motion detection and sensor based motion detection of the fused data to determine when the camera is at least partially blocked.

### Claim 2 (depends on 1)

The measurement system of claim 1, wherein the motion detection sensor includes one of an accelerometer, a gyroscope, and an inclinometer.

### Claim 3 (depends on 1)

The measurement system of claim 1, wherein the fused data is time correlated data.

### Claim 4 (depends on 1)

The measurement system of claim 1 wherein the fused data set is created using an automotive safety integration level (ASIL) decomposition fusion.

### Claim 5 (depends on 1)

The measurement system of claim 1, wherein the fused data set is provided to at least one of a camera orientation monitor system, an object detection system, a camera misalignment system, a field of view adjustment system, an object detection redundancy system, an image stabilization system, an end of line calibration system, a tamper recognition system, and an automatic camera panning system, and a camera displacement detection system.

### Claim 6 (depends on 5)

The measurement system of claim 5, wherein the fused data set is provided to the camera displacement detection system, and wherein the camera displacement detection system is configured to compare a speed of motion detected via image analysis with a speed of motion detected via sensor analysis and determine a camera displacement has occurred in response to the speed of motion detected via the image analysis exceeding speed of the motion detected via the sensor analysis by a predetermined amount.

### Claim 7 (depends on 1)

The measurement system of claim 1, further comprising a plurality of second cameras, each of the plurality of second cameras defining a field of view having a corresponding optical axis, and a plurality of motion detection sensors each of the motion detection sensors in the plurality of motion detection sensors being mechanically fixed to a corresponding one of the second cameras in the plurality of second cameras via a corresponding application specific integrated circuit (ASIC), wherein a motion of each camera in the plurality of second cameras and a motion of the first camera is determined independently of the motion of each other camera.

### Claim 8 (depends on 7)

The measurement system of claim 7, wherein each of the corresponding ASICs are configured to output a fused data set comprising a combination of image data generated by the corresponding camera and motion data generated by the motion detection sensor.

### Claim 9 (depends on 1)

The measurement system of claim 1, wherein the blockage detection system is configured to determine that the camera is at least partially blocked in response to the image based motion detection system detecting no motion and the sensor based motion detection system detecting motion.

### Claim 10

A method for detecting a blocked camera lens for a vehicle comprising:
receiving image data and inertial motion unit (IMU) data from a camera system, the camera system including a motion detection sensor mechanically connected to an application specific integrated circuit (ASIC) of a first camera such that motion of the ASIC corresponds to motion of the optical axis in a one to one fashion and the motion detection sensor is configured to detect motion of the optical axis, the ASIC being configured to output a fused data set comprising a combination of image data generated by the first camera and motion data generated by the motion detection sensor;
analyzing the IMU data to detect motion of the camera system using a sensor based motion detection system including a controller configured to receive the fused data set;
analyzing the fused data set to detect motion of the camera using an image based machine learning motion detection analysis, the machine learning motion detection analysis being configured to determine at least an orientation of the camera and alert an operator in response to the first camera facing an incorrect direction; and
determining a presence of an at least partially blocked lens in response to the image based motion detection analysis indicating no motion and the sensor based motion detection indicating motion.

### Claim 11 (depends on 10)

The method of claim 10, wherein the camera system includes a first camera defining a field of view having a corresponding optical axis and a motion detection sensor mechanically fixed to the first camera such that the motion detection sensor is configured to detect motion of the optical axis.

### Claim 12 (depends on 10)

The method of claim 10, wherein the image based motion detection analysis includes:
identifying a reference in a first frame of the image data;
identifying the reference in a second frame of the image data; and
determining that the camera system is in motion in response to a change in position of the reference between frames being in excess of a threshold.

### Claim 13 (depends on 12)

The method of claim 12, wherein the second frame is subsequent to the first frame.

### Claim 14 (depends on 10)

The method of claim 10, wherein the image data and the IMU data are fused while determining the presence of an at least partially blocked lens.

### Claim 15

A method for detecting a displaced camera system for a vehicle comprising:
Receiving, at a controller, fused data including an image portion and an inertial motion unit (IMU) portion from a camera system, the camera system including a motion detection sensor mechanically connected to an application specific integrated circuit (ASIC) of a first camera such that motion of the ASIC corresponds to motion of the optical axis in a one to one fashion and the motion detection sensor is configured to detect motion of the optical axis, and the ASIC being configured to output a fused data set comprising a combination of image data generated by the first camera and motion data generated by the motion detection sensor;
analyzing the image portion using an image based motion detection system of the controller and determining at least one parameter of motion in at least one object within the image;
analyzing the IMU portion using a sensor based motion detection system of the controller and determining at least one parameter of motion of the IMU;
comparing the at least one parameter of motion of the at least one object in the image and the at least one parameter of motion of the IMU, and determining a camera system displacement has occurred when the at least one parameter of motion of the at least one objection differs from the motion of the IMU by at least a predefined amount; and
analyze the fused data set using a machine learning based analysis, the machine learning based analysis being configured to determine an orientation of the camera and alert an operator in response to the first camera facing an incorrect direction.

### Claim 16 (depends on 15)

The method of claim 15, wherein the at least one parameter is a speed of motion.

### Claim 17 (depends on 15)

The method of claim 15, wherein the at least one object within the image is at an outer periphery of the image.

### Claim 18 (depends on 15)

The method of claim 15, further comprising determining an at least partial blockage of the camera in response to the image based motion detection system detecting no motion and the sensor based motion detection system detecting motion.