# US 11,910,092 B2 · Panoramic look-around view generation method, in-vehicle device and in-vehicle system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,910,092 B2
- Document ID: US11910092B2
- Status: Granted
- Kind: B2
- Application number: 17/061,026
- Filed: 2020-10-01
- Published: 2022-04-07
- Granted: 2024-02-20
- Earliest claimed: 2020-10-01
- Assignees: Black Sesame Technologies Inc.
- Applicants: Black Sesame International Holding Limited
- Inventors: Wenxue Li
- Web page: https://patentgrep.com/patent/US11910092B2

## CPC

- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- B60R 1/27: providing all-round vision, e.g. using omnidirectional cameras
- B60R 11/0229: {for displays, e.g. cathodic tubes}
- B60R 11/04: Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
- G06T 3/02: Affine transformations (for image registration G06T3/147; for image mosaicing G06T3/4038)
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 7/33: using feature-based methods
- G06T 7/337: {involving reference images or patches}
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- B60R 2300/105: using multiple cameras
- B60R 2300/30: characterised by the type of image processing
- B60R 2300/302: combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
- B60R 2300/303: using joined images, e.g. multiple camera images
- B60R 2300/60: characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
- B60R 2300/607: from a bird's eye viewpoint
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The current disclosure relates to a panoramic look-around view generation method, an in-vehicle device and an in-vehicle system. The method comprises the following steps of: acquiring images of areas around a vehicle, steering wheel angle information and vehicle speed information; transforming and mosaicking the images to generate a time-related look-around view, and using an ORB algorithm to extract characteristic points; using steering wheel angle information and vehicle speed information to calculate a positron of a characteristic point of the look-around view at previous time in the look-around view at current time according to a vehicle motion model, and selecting a characteristic point located near the position in the look-around view at current time to be matched with the characteristic point of the look-around view at previous time; calculating an affine transformation matrix and performing affine transformation, and performing weighted fusion with the look-around view at current time and storing it; repeating the above steps to obtain a continuously updated panoramic look-around view. The method, the device and the system can eliminate blind area in the underbody region, realize a panoramic perspective view, and have fast calculation and high accuracy.

## Claims

### Claim 1

A panoramic look-around view generation method, comprising steps of:
acquiring images of areas around a vehicle from a plurality of cameras installed on the vehicle, and acquiring steering wheel angle information and vehicle speed information from the vehicle;
generating a look-around view associated with a first time by transforming and mosaicking images acquired by the plurality of cameras, using the look-around view generated at a first time as the look-around view at the first time, and saving the look-around view at the first time;
(i) acquiring images of areas around the vehicle from a plurality of cameras installed on the vehicle, and acquiring steering wheel angle information and vehicle speed information from the vehicle;
(ii) generating the look-around view associated with a current time by transforming and mosaicking images acquired by the plurality of cameras, using the look-around view generated at the current time as the look-around view at the current time, and saving the look-around view at the current time;
(iii) performing a registration calculation on the look-around view at the current time and a saved look-around view at a previous time, wherein the registration calculation comprises following steps of:
(iv) respectively extracting a plurality of characteristic points of the look-around view at the previous time and a plurality of characteristic points of the look-around view at the current time by using an Oriented FAST and Rotated BRIEF (ORB) algorithm;
(v) calculating a position of a characteristic point of the look-around view at the previous time in the look-around view at the current time according to a vehicle motion model using the steering wheel angle information and the vehicle speed information;
selecting a characteristic point located near the position from the characteristic points of the look-around view at the current time to be matched with the characteristic point of the look-around view at the previous time; and
(vi) calculating an affine transformation matrix between the look-around view at the previous time and the look-around view at the current time;
(vii) performing affine transformation on the look-around view at the previous time by using the affine transformation matrix and performing weighted fusion with the look-around view at the current time to generate an updated look-around view and save the updated look-around view;
repeating above steps (i)-(vii) to obtain continuously updated panoramic look-around views; and
wherein a first transformation matrix of image coordinate systems is:
[
u
1
v
1
1
]
=
[
cos
⁢
⁢
θ
sin
⁢
⁢
θ
P
x
-
P
x
⁢
⁢
cos
⁢
⁢
θ
-
P
y
⁢
⁢
sin
⁢
⁢
θ
-
sin
⁢
⁢
θ
cos
⁢
⁢
θ
P
y
-
P
y
⁢
⁢
cos
⁢
⁢
θ
+
P
x
⁢
⁢
sin
⁢
⁢
θ
0
0
1
]
⁡
[
u
0
v
0
1
]
wherein (Px,Py) is the coordinate of a rotation center of the vehicle motion; θ is a rotation angle of the vehicle from the previous time to the current time; (u0,v0) is a coordinate of a characteristic point of the look-around view at the previous time in the look-around view at the previous time; (u1,v1) is the coordinate of the characteristic point of the look-around view at the previous time in the look-around view at the current time.

### Claim 2 (depends on 1)

The panoramic look-around view generation method of claim 1, wherein the registration calculation further comprises using the steering wheel angle information and the vehicle speed information to obtain the first transformation matrix of image coordinate systems from the previous time to the current time according to the vehicle motion model; and
wherein the step of calculating the position of the characteristic point of the look-around view at the previous time in the look-around view at the current time according to the vehicle motion model by using the steering wheel angle information and the vehicle speed information further comprises determining the position of the characteristic point of the look-around view at the previous time in the look-around view at the current time by using the first transformation matrix.

### Claim 3 (depends on 1)

The panoramic look-around view generation method of claim 1, wherein the registration calculation further comprises using the steering wheel angle information and the vehicle speed information to obtain the first transformation matrix of image coordinate systems from the previous time to the current time according to the vehicle motion model; and
wherein the step of calculating the affine transformation matrix between the look-around view at the previous time and the look-around view at the current time further comprises calculating a second transformation matrix by using a RANSAC algorithm according to the extracted characteristic points of the look-around view at the previous time and the characteristic points of the look-around view at the current time;
calculating the similarity between the second transformation matrix and the first transformation matrix, using the first transformation matrix as the affine transformation matrix when the similarity is less than a preset threshold, and using the second transformation matrix as the affine transformation matrix when the similarity is greater than or equal to the preset threshold.

### Claim 4 (depends on 1)

The panoramic look-around view generation method of claim 1, wherein
the extracting a plurality of characteristic points of the look-around view at the previous time and the characteristic points of the look-around view at the current time using the ORB algorithm respectively comprises:
creating an improved quadtree to represent the actually extracted characteristic points, wherein each node of the improved quadtree has a physical space occupied by itself and a key point contained in the node;
equally dividing the key point from one to four according to the physical space, wherein a number of original key points are divided into sub-nodes where they are located, and the dividing of the improved quadtree will not stop until the number of nodes of the improved quadtree is greater than or equal to a number of target key points or the number of nodes of the improved quadtree does not change any more;
selecting the characteristic point with the highest score as the extracted characteristic point when the number of key points in the node of the quadtree is greater than 1.