# US 12,183,022 B2 · Topographic environment detection method and system based on binocular stereo camera, and intelligent terminal

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,183,022 B2
- Document ID: US12183022B2
- Status: Granted
- Kind: B2
- Application number: 17/658,972
- Filed: 2022-04-12
- Published: 2023-05-11
- Granted: 2024-12-31
- Earliest claimed: 2021-11-10
- Assignees: Beijing Smarter Eye Technology Co. Ltd.; Shenzhen Pengcheng Electric Group Co., Ltd; Shenzhen Talent Database Data Technology Co., Ltd
- Applicants: Beijing Smarter Eye Technology Co. Ltd; Shenzhen Pengcheng Electric Group Co., Ltd.; Shenzhen Talent Database Data Technology Co., Ltd
- Inventors: Wentao Lin; Hua Chai; Shanshan Pei; Shuhong Wu; Bin Chen; Yuzhi Lin; Ran Meng; Li Ma; Zhiwei He
- Web page: https://patentgrep.com/patent/US12183022B2

## CPC

- G06T 7/593: from stereo images
- B60W 30/08: {Active safety systems} predicting or avoiding probable or impending collision {or attempting to minimise its consequences}
- B60W 40/076: Slope angle of the road
- B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention
- G06T 7/11: Region-based segmentation
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- H04N 13/128: Adjusting depth or disparity
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- B60W 2050/146: {Display means}
- B60W 2552/25: Road altitude
- G06T 2207/10012: Stereo images
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The topographic environment detection method and a topographic environment detection system based on a binocular stereo camera, and an intelligent terminal are provided. The topographic environment detection method includes: obtaining a left-eye image and a right-eye image about a same road scenario, and processing the left-eye image and the right-eye image to obtain a dense disparity map of the road scenario; converting image information in a detection region into 3D point cloud information in a world coordinate system in accordance with the dense disparity map; fitting a road surface model in accordance with the 3D point cloud information; inputting an image in the detection region into a trained semantic segmentation model, and obtaining a segmentation result from the semantic segmentation model; and obtaining topographic information about the detection region in accordance with the segmentation result, and transmitting the topographic information to a vehicle assistant driving system.

## Claims

### Claim 1

A topographic environment detection method based on a binocular stereo camera, comprising:
S1: obtaining a left-eye image and a right-eye image about a same road scenario, and processing the left-eye image and the right-eye image to obtain a dense disparity map of the road scenario;
S2: converting image information in a detection region into three-dimensional (3D) point cloud information in a world coordinate system in accordance with the dense disparity map;
S3: fitting a road surface model in accordance with the 3D point cloud information;
S4: inputting an image in the detection region into a trained semantic segmentation model, and obtaining a segmentation result from the semantic segmentation model; and
S5: obtaining topographic information about the detection region in accordance with the segmentation result, and transmitting the topographic information to a vehicle assistant driving system, so that the vehicle assistant driving system sends a driving instruction and/or warning information in accordance with the topographic information.

### Claim 2 (depends on 1)

The topographic environment detection method according to claim 1, wherein S2 comprises:
converting an image coordinate system of the dense disparity map into the world coordinate system in accordance with an imaging model of a binocular stereo vision system and a pinhole imaging model;
intercepting the detection region from the dense disparity map with the detection region in a real world coordinate system as a reference; and
converting the image information in the detection region into the 3D point cloud information through Z=B*F/disp, X=(Imgx−cx)*b/disp, and Y=(Imgy−cy)*b/disp, where B represents a distance between an optical center of a left camera and an optical center of a right camera in the binocular stereo vision system, F represents a focal length of the camera in the binocular stereo vision system, cx and cy represent coordinates of a principal point of the camera in the binocular stereo vision system, Imgx and Imgy are coordinates of a point in the image in the detection region, disp represents a disparity value of the point (Imgx, Imgy), X represents a horizontal distance between a 3D point and the camera in the world coordinate system, Y represents a longitudinal distance between the 3D point and the camera in the world coordinate system, and Z represents a depth distance between the 3D point and the camera in the world coordinate system.

### Claim 3 (depends on 1)

The topographic environment detection method according to claim 1, wherein the road surface model is expressed as cos α*X+cos β*Y+cos γ*Z+D=0, wherein cos α represents a cosine of an angle between a normal vector of a road surface and an x-axis of the world coordinate system, cos β represents a cosine of an angle between the normal vector of the road surface and a y-axis of the world coordinate system, cos γ represents a cosine of an angle between the normal vector of the road surface and a z-axis of the world coordinate system, and D represents a distance between an origin of the world coordinate system and the road surface.

### Claim 4 (depends on 1)

The topographic environment detection method according to claim 1, wherein the semantic segmentation model is trained using a method comprising:
analyzing a topographic condition which occurs for a road, and classifying common scenarios;
photographing the scenarios to obtain a plurality of training images;
with respect to each training image, labeling a target region to obtain a masked image, the target regions in different scenarios corresponding to different labeling features, each training image uniquely corresponding to one masked image; and
performing a training operation in accordance with all the training images and the masked images, so as to obtain the semantic segmentation model.

### Claim 5 (depends on 1)

The topographic environment detection method according to claim 1, wherein S5 comprises:
determining that the detection region is a non-flat region in accordance with the segmentation result, projecting a 3D point onto a bird's-eye plane, and generating an elevation map on the bird's-eye plane in accordance with height information about each discrete point in the segmentation result; and
calculating elevation information about a wheel region in accordance with the elevation map.

### Claim 6 (depends on 5)

The topographic environment detection method according to claim 5, wherein the determining that the detection region is the non-flat region in accordance with the segmentation result, projecting the 3D point onto the bird's-eye plane and generating the elevation map on the bird's-eye plane in accordance with height information about each discrete point in the segmentation result comprises:
when a segmented image pixel value is a value other than 1, 2 or 5, determining that the detection region is the non-flat region in accordance with the segmentation result from the semantic segmentation model;
calculating a distance between the 3D point corresponding to the segmented image pixel value in the non-flat region and a plane where the road surface model is located through A=cos α, B=cos β, C=cos γ, and H=|A*X_0+B*Y_0+C*Z_0+D|−|/√(A{circumflex over ( )}2+B{circumflex over ( )}2+C{circumflex over ( )}2), where cos α, cos β, cos γ and D are parameters of the road surface model, X0, Y0 and Z0 are coordinates of a discrete 3D point in the world coordinate system, H is a height of the discrete 3D point (X0, Y0 and Z0) relative to a road surface; and
projecting the discrete 3D point onto the bird's-eye plane, and storing the height information about the discrete 3D point whose segmented image pixel value is a value other than 1, 2 or 5 at a corresponding position, so as to generate the elevation map on the bird's-eye plane.

### Claim 7 (depends on 5)

The topographic environment detection method according to claim 5, wherein the calculating the elevation information about the wheel region in accordance with the elevation map comprises: converting the world coordinate system into an image coordinate system in accordance with a wheel length and a distance between two wheels in the real world coordinate system through the imaging model of the binocular stereo vision system and the pinhole imaging mode to obtain a position of the wheel region on the bird's-eye plane; and
analyzing data in the wheel region in accordance with the elevation map on the bird's-eye plane to obtain the elevation information about the wheel region, wherein the elevation information comprises a distance between an elevation change region and the camera, and a fluctuation level between the elevation change region and a ground point.

### Claim 8 (depends on 1)

An intelligent terminal, comprising a data collection device, a processor and a memory, wherein the data collection device is configured to collect data, the memory is configured to store therein one or more program instructions, and the processor is configured to execute the one or more program instructions so as to implement the topographic environment detection method according to claim 1.

### Claim 9 (depends on 1)

A non-transitory computer-readable storage medium storing therein one or more program instructions, wherein the one or more program instructions are executed so as to implement the topographic environment detection method according to claim 1.

### Claim 10

A topographic environment detection system based on a binocular stereo camera, comprising:
a disparity map obtaining unit configured to obtain a left-eye image and a right-eye image about a same road scenario, and process the left-eye image and the right-eye image to obtain a dense disparity map of the road scenario;
a coordinate conversion unit configured to convert image information in a detection region into 3D point cloud information in a world coordinate system in accordance with the dense disparity map;
a road surface model obtaining unit configured to fit a road surface model in accordance with the 3D point cloud information;
an image segmentation unit configured to input an image in the detection region into a trained semantic segmentation model, and obtain a segmentation result from the semantic segmentation model; and
a result output unit configured to obtain topographic information about the detection region in accordance with the segmentation result, and transmit the topographic information to a vehicle assistant driving system, so that the vehicle assistant driving system sends a driving instruction and/or warning information in accordance with the topographic information.