# US 2021/0197893 A1 · Vehicular trailering assist system with targetless calibration of trailer cameras

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0197893 A1
- Document ID: US20210197893A1
- Status: Published application
- Kind: A1
- Application number: 17/247,616
- Filed: 2020-12-18
- Published: 2021-07-01
- Earliest claimed: 2019-12-26
- Assignees: Magna Electronics Inc.
- Applicants: Magna Electronics Inc.
- Inventors: Galina Okouneva; James Turk; Anam M. Yunus; Alexander Velichko
- Web page: https://patentgrep.com/patent/US20210197893A1

## CPC

- B60D 1/245: {for facilitating push back or parking of trailers}
- B62D 13/025: {the pivoted movement being initiated by the coupling means between tractor and trailer}
- B60D 1/01: Traction couplings or hitches characterised by their type
- B60D 1/30: for sway control {; Sway alarm means}
- B60D 1/62: involving supply lines, electric circuits or the like
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- B60D: VEHICLE CONNECTIONS
- B62D: MOTOR VEHICLES; TRAILERS (steering, or guiding on a desired track, of agricultural machines or implements A01B69/00; wheels, castors, axles, increasing wheel adhesion B60B; vehicle tyres, tyre inflation or tyre changing B60C; connections between vehicles of a train or the like B60D; vehicles for use on rail and road, amphibious or convertible vehicles B60F; suspension arrangements B60G; heating, cooling, ventilating or other air treating devices B60H; windows, windscreens, non-fixed roofs, doors or similar devices, protective coverings for vehicles not in use B60J; propulsion plant arrangements, auxiliary drives, transmissions, controls, instrumentation or dashboards B60K; electric equipment or propulsion of electrically-propelled vehicles B60L; power supply for electrically-propelled vehicles B60M; passenger accommodation not otherwise provided for B60N; adaptations for load transportation or to carry special loads or objects B60P; arrangement of signalling or lighting devices, the mounting or supporting thereof or circuits therefor, for vehicles in general B60Q; vehicles, vehicle fittings or vehicle parts not otherwise provided for B60R; servicing, cleaning, repairing, supporting, lifting, or manoeuvring, not otherwise provided for, B60S; brake arrangements, brake control systems or parts thereof B60T; air-cushion vehicles B60V; motor-cycles, accessories therefor B62J, B62K; testing of vehicles G01M)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A vehicular trailering assist system includes at least one camera disposed at a trailer hitched to a hitch of a vehicle, and an electronic control unit (ECU). The ECU determines, during a first stage of a calibration maneuver of the vehicle and trailer, image coordinates of at least one ground feature point. The ECU, responsive to determining image coordinates of the at least one ground feature point, estimates orientation parameters of the camera based at least on the determined image coordinates and known intrinsic camera parameters. Responsive to estimating the orientation parameters of the camera, and based on the determined parameters of the camera, the ECU determines location of the at least one camera at the trailer.

## Claims

### Claim 1

A vehicular trailering assist system, the vehicular trailering assist system comprising:
at least one camera disposed at a trailer hitched to a hitch ball of a vehicle equipped with the vehicular trailering assist system to form a pivoting joint hitch connection of the trailer to the vehicle, the at least one camera viewing exterior of the trailer hitched to the hitch ball of the vehicle, wherein the at least one camera captures frames of image data;
an electronic control unit (ECU) comprising electronic circuitry and associated software, wherein the electronic circuitry comprises an image processor operable to process frames of image data captured by the at least one camera;
wherein the ECU, as the vehicle and trailer travel along a ground surface during a first stage of a calibration maneuver of the vehicle and trailer, and responsive to image processing at the ECU of multiple frames of image data captured by the at least one camera, determines image coordinates of at least one ground feature relative to the trailer;
wherein, responsive to determining the image coordinates of the at least one ground feature relative to the trailer, the ECU, during the first stage of a calibration maneuver of the vehicle and trailer, estimates orientation parameters of the at least one camera based at least on (i) the determined image coordinates of the at least one ground feature relative to the trailer and (ii) known intrinsic camera parameters; and
wherein, as the vehicle and trailer travel further along the ground surface during a second stage of the calibration maneuver of the vehicle and trailer, and based on the estimated orientation parameters of the at least one camera determined during the first stage of the calibration maneuver, the ECU determines location of the at least one camera at the trailer.

### Claim 2 (depends on 1)

The vehicular trailering assist system of claim 1, wherein, responsive to determining the location of the at least one camera at the trailer, the ECU adjusts processing of frames of image data captured by the at least one camera to accommodate for misalignment of the at least one camera at the trailer.

### Claim 3 (depends on 1)

The vehicular trailering assist system of claim 1, wherein the estimated orientation parameters of the at least one camera comprise (i) a rotation angle of the at least one camera relative to the trailer about a respective vertical axis of the at least one camera, (ii) a rotation angle of the at least one camera relative to the trailer about a respective horizontal axis of the at least one camera and (iii) a rotation angle of the at least one camera relative to the trailer about a respective longitudinal axis of a lens barrel of the at least one camera.

### Claim 4 (depends on 1)

The vehicular trailering assist system of claim 1, wherein the at least one camera comprises four cameras.

### Claim 5 (depends on 1)

The vehicular trailering assist system of claim 1, wherein the ECU, during the first stage of the calibration maneuver, determines the image coordinates of the at least one ground feature relative to the trailer while the vehicle and trailer travel in a straight line, and wherein the ECU, during the second stage of the calibration maneuver, determines the location of the at least one camera at the trailer while the vehicle and trailer travel along a curve.

### Claim 6 (depends on 1)

The vehicular trailering assist system of claim 1, wherein, responsive to determining the image coordinates of the at least one ground feature relative to the trailer, the ECU tracks the at least one ground feature in at least one subsequent frame of captured image data.

### Claim 7 (depends on 1)

The vehicular trailering assist system of claim 1, wherein the known intrinsic camera parameters of the at least one camera comprise at least one selected from the group consisting of (i) focal length, (ii) optical center coordinates, (iii) and coefficients of a polynomial used to unwarp image data.

### Claim 8 (depends on 1)

The vehicular trailering assist system of claim 1, wherein the ECU estimates the orientation parameters of the at least one camera using projective geometry.

### Claim 9 (depends on 1)

The vehicular trailering assist system of claim 1, wherein the ECU estimates a height of the at least one camera relative to the ground surface based on trailer kinematics information.

### Claim 10 (depends on 1)

The vehicular trailering assist system of claim 1, wherein the at least one camera comprises a front camera disposed at the front of the trailer, and wherein the ECU, responsive to image processing of frames of image data captured by the front camera, determines image coordinates of a three dimensional (3D) feature located at one selected from the group consisting of (i) a hitch of the vehicle and (ii) a body of the vehicle.

### Claim 11 (depends on 10)

The vehicular trailering assist system of claim 10, wherein the ECU, responsive to determining the image coordinates of the 3D feature, estimates a virtual rotation matrix of the front camera, and wherein the ECU, responsive to estimating the virtual rotation matrix, performs a nonlinear optimization to optimize parameters of the front camera, and wherein the parameters of the front camera comprise rotation angles of the front camera.

### Claim 12 (depends on 11)

The vehicular trailering assist system of claim 11, wherein the rotation angles of the front camera comprise (i) rotation of the front camera relative to the trailer about a vertical axis of the front camera, (ii) rotation of the front camera relative to the trailer about a horizontal axis of the front camera and (iii) rotation of the front camera relative to the trailer about a longitudinal axis of a lens barrel of the front camera.

### Claim 13 (depends on 12)

The vehicular trailering assist system of claim 12, wherein the ECU determines an initial estimate for the parameters of the front camera based on trailer kinematics.

### Claim 14

A vehicular trailering assist system, the vehicular trailering assist system comprising:
at least one camera disposed at a trailer hitched to a hitch ball of a vehicle equipped with the vehicular trailering assist system to form a pivoting joint hitch connection of the trailer to the vehicle, the at least one camera viewing exterior of the trailer hitched to the hitch ball of the vehicle, wherein the at least one camera captures frames of image data;
an electronic control unit (ECU) comprising electronic circuitry and associated software, wherein the electronic circuitry comprises an image processor operable to process frames of image data captured by the at least one camera;
wherein the ECU, as the vehicle and trailer travel along a ground surface during a first stage of a calibration maneuver of the vehicle and trailer, and responsive to image processing at the ECU of multiple frames of image data captured by the at least one camera, determines image coordinates of at least one ground feature relative to the trailer;
wherein, responsive to determining the image coordinates of the at least one ground feature relative to the trailer, the ECU, during the first stage of a calibration maneuver of the vehicle and trailer, estimates orientation parameters of the at least one camera based at least on (i) the determined image coordinates of the at least one ground feature relative to the trailer and (ii) known intrinsic camera parameters;
wherein the estimated orientation parameters of the at least one camera comprise (i) a rotation angle of the at least one camera relative to the trailer about a respective vertical axis of the at least one camera, (ii) a rotation angle of the at least one camera relative to the trailer about a respective horizontal axis of the at least one camera and (iii) a rotation angle of the at least one camera relative to the trailer about a respective longitudinal axis of a lens barrel of the at least one camera;
wherein, as the vehicle and trailer travel further along the ground surface during a second stage of the calibration maneuver of the vehicle and trailer, and based on the estimated orientation parameters of the at least one camera determined during the first stage of the calibration maneuver, the ECU determines location of the at least one camera at the trailer; and
wherein, responsive to determining the location of the at least one camera at the trailer, the ECU adjusts processing of frames of image data captured by the at least one camera to accommodate for misalignment of the at least one camera at the trailer.

### Claim 15 (depends on 14)

The vehicular trailering assist system of claim 14, wherein the at least one camera comprises four cameras.

### Claim 16 (depends on 14)

The vehicular trailering assist system of claim 14, wherein the ECU, during the first stage of the calibration maneuver, determines the image coordinates of the at least one ground feature relative to the trailer while the vehicle and trailer travel in a straight line, and wherein the ECU, during the second stage of the calibration maneuver, determines the location of the at least one camera at the trailer while the vehicle and trailer travel along a curve.

### Claim 17 (depends on 14)

The vehicular trailering assist system of claim 14, wherein the known intrinsic camera parameters of the at least one camera comprise at least one selected from the group consisting of (i) focal length, (ii) optical center coordinates, (iii) and coefficients of a polynomial used to unwarp image data.

### Claim 18

A vehicular trailering assist system, the vehicular trailering assist system comprising:
at least one camera disposed at a trailer hitched to a hitch ball of a vehicle equipped with the vehicular trailering assist system to form a pivoting joint hitch connection of the trailer to the vehicle, the at least one camera viewing exterior of the trailer hitched to the hitch ball of the vehicle, wherein the at least one camera captures frames of image data;
an electronic control unit (ECU) comprising electronic circuitry and associated software, wherein the electronic circuitry comprises an image processor operable to process frames of image data captured by the at least one camera;
wherein the ECU, as the vehicle and trailer travel along a ground surface during a first stage of a calibration maneuver of the vehicle and trailer, and responsive to image processing at the ECU of multiple frames of image data captured by the at least one camera, determines image coordinates of at least one ground feature relative to the trailer;
wherein, responsive to determining the image coordinates of the at least one ground feature relative to the trailer, the ECU, during the first stage of a calibration maneuver of the vehicle and trailer, estimates orientation parameters of the at least one camera based at least on (i) the determined image coordinates of the at least one ground feature relative to the trailer and (ii) known intrinsic camera parameters;
wherein the estimated orientation parameters of the at least one camera comprise (i) a rotation angle of the at least one camera relative to the trailer about a respective vertical axis of the at least one camera, (ii) a rotation angle of the at least one camera relative to the trailer about a respective horizontal axis of the at least one camera and (iii) a rotation angle of the at least one camera relative to the trailer about a respective longitudinal axis of a lens barrel of the at least one camera;
wherein the known intrinsic camera parameters of the at least one camera comprise at least one selected from the group consisting of (i) focal length, (ii) optical center coordinates, (iii) and coefficients of a polynomial used to unwarp image data; and
wherein, as the vehicle and trailer travel further along the ground surface during a second stage of the calibration maneuver of the vehicle and trailer, and based on the estimated orientation parameters of the at least one camera determined during the first stage of the calibration maneuver, the ECU determines location of the at least one camera at the trailer.

### Claim 19 (depends on 18)

The vehicular trailering assist system of claim 18, wherein the ECU estimates the orientation parameters of the at least one camera using projective geometry.

### Claim 20 (depends on 18)

The vehicular trailering assist system of claim 18, wherein the ECU estimates a height of the at least one camera relative to the ground surface based on trailer kinematics information.
