# US 2020/0402300 A1 · Terrain modeling method that fuses geometric characteristics and mechanical charateristics, computer readable storage medium, and terrain modeling system thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0402300 A1
- Document ID: US20200402300A1
- Status: Published application
- Kind: A1
- Application number: 16/826,326
- Filed: 2020-03-23
- Published: 2020-12-24
- Earliest claimed: 2019-06-21
- Assignees: Harbin Institute of Technology
- Applicants: Harbin Institute of Technology
- Inventors: Liang Ding; Zongquan Deng; Ruyi Zhou; Haibo Gao; Wenhao Feng; Peng Xu; Nan Li; Zhen Liu
- Web page: https://patentgrep.com/patent/US20200402300A1

## CPC

- G06N 3/08: Learning methods
- G06F 18/251: {of input or preprocessed data}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06T 7/11: Region-based segmentation
- G06T 17/05: Geographic models
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/38: {Outdoor scenes}
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30244: Camera pose
- G06V 2201/12: Acquisition of 3D measurements of objects
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A terrain modeling method that fuses geometric characteristics and mechanical characteristics, and a terrain modeling system are provided in the present invention. The method includes steps of: obtaining color images and depth images of the detected region, performing a terrain semantic segmentation on the color images, and fusing ground semantic information obtained by the semantic segmentation and depth information contained in the depth images at the same time to generate the point clouds; mapping the point clouds to a raster map in a map coordinate system to generate corresponding rasters, and updating the elevation values to the corresponding rasters; and calculating the input images in terms of terra-mechanical characteristics, and updating calculation results to the corresponding rasters to generate a terrain model. A mechanical characteristic parameter is added to a terrain factor, and terrain characteristics are innovatively performed from two dimensions of geometric characteristic and mechanical characteristics.

## Claims

### Claim 1

A terrain modeling method that fuses geometric characteristics and mechanical characteristics, comprising steps of:
obtaining input images of a detected region, and processing the input images to generate point clouds comprising elevation values of the input images;
mapping the point clouds to a raster map in a map coordinate system to generate corresponding rasters, and updating the elevation values to the corresponding rasters; and
calculating the input images in terms of terra-mechanical characteristics, and updating calculation results to the corresponding rasters to generate a terrain model.

### Claim 2 (depends on 1)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 1, wherein the step of obtaining input images of a detected region and processing the input images to generate point clouds comprises:
obtaining color images and depth images of the detected region, performing a terrain semantic segmentation on the color images, and fusing ground semantic information obtained by the semantic segmentation and depth information contained in the depth images at the same time to generate the point clouds.

### Claim 3 (depends on 2)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 2, wherein the step of performing a terrain semantic segmentation on the color images comprises:
collecting terrain images as a data source, annotating a target terrain in the collected terrain images and generating corresponding mask labels, and randomly dividing the annotated terrain images into training sets and validation sets in proportion;
calculating a mean and a variance of three RGB channels of the training sets, respectively, performing a data normalization on each of the input images, performing a data enhancement on original images in the training sets, and correcting the corresponding mask labels; and
constructing a convolutional neural network, performing a training on a semantic segmentation model, and selecting a model with the highest accuracy as the semantic segmentation model for performing the terrain semantic segmentation on the color images.

### Claim 4 (depends on 2)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 2, wherein the semantic information comprises a plurality of terrain types with a higher confidence obtained after the terrain semantic segmentation and corresponding probabilities.

### Claim 5 (depends on 2)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 2, wherein the step of fusing ground semantic information obtained by the semantic segmentation and depth information contained in the depth images at the same time to generate the semantic point clouds comprises:
combining the semantic information of each pixel in the color images with the depth information contained in the depth images at the same time; and
performing a coordinate transformation using pose information of a camera carried by a planet rover to generate the semantic point clouds in the map coordinate system.

### Claim 6 (depends on 5)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 5, wherein the step of performing a coordinate transformation using pose information of a camera carried by a planet rover to generate the semantic point clouds in the map coordinate system comprises:
mapping three-dimensional point clouds in a depth camera coordinate system to a color camera coordinate system at the same time to form color three-dimensional point clouds in the color camera coordinate system; and
transforming the color three-dimensional point clouds into the map coordinate system via a planet rover base coordinate system.

### Claim 7 (depends on 6)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 6, wherein the color camera coordinate system and the depth camera coordinate system are fixedly connected to a planet rover body, and the planet rover base coordinate system changes with changes in poses of the planet rover and has a projection of a coordinate origin on an XY plane of the map coordinate system coinciding with a origin of the map coordinate system.

### Claim 8 (depends on 2)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 2, wherein the step of mapping the point clouds to a raster map in a map coordinate system to generate corresponding rasters, and assigning the elevation values to the corresponding rasters comprises:
updating the elevation values and the semantic information contained in the rasters through a data fusion when the point clouds at different times correspond to the same raster.

### Claim 9 (depends on 8)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 8, wherein the elevation values are updated by obtaining an average value of the elevation values contained in the plurality of point clouds.

### Claim 10 (depends on 8)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 8, wherein the step of updating the semantic information through a data fusion comprises:
updating the terrain types contained in the plurality of point clouds and the probabilities through Bayesian updating rule.

### Claim 11 (depends on 8)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 8, wherein the step of updating the semantic information through a data fusion comprises:
remaining the updated terrain types and a corresponding updated probability being an average of the two probabilities if the corresponding types of the maximum probability of the two semantic point clouds are the same; and
remaining the terrain types with a larger probability as the updated terrain types and a corresponding updated probability being a times the corresponding larger probability if the corresponding types of the maximum probability of the two semantic point clouds are different.

### Claim 12 (depends on 1)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 1, wherein the step of estimating the input images in terms of terra-mechanical characteristics, and updating calculation results to the corresponding rasters to generate a terrain model comprises:
fitting a distribution function according to statistical data of the terra-mechanical characteristics; and
performing a numerical estimation of the terra-mechanical characteristics with safety factors in combination with the semantic information, and assigning estimation results to the corresponding rasters to generate the terrain model.

### Claim 13 (depends on 12)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 12, wherein the step of performing a numerical calculation of the terra-mechanical characteristics with safety factors in combination with the semantic information comprises:
calculating safety factors S(Ks) of equivalent stiffness modulus Ks and safety factors S(φ) of equivalent friction modulus φ, respectively:
S

(
K
s
)
=
∑
i

c
i

F
i

(
K
s
)
,
S

(
ϕ
)
=
∑
i

c
i

F
i

(
ϕ
)
;
wherein ci represents a confidence level corresponding to the i-th terrain category in the terrain semantic segmentation results, Fi(Ks) represents a cumulative distribution function of the equivalent stiffness modulus, and Fi(φ) represents a cumulative distribution function of the equivalent friction modulus.

### Claim 14 (depends on 1)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 1, wherein the step of estimating the input images in terms of terra-mechanical characteristics, and updating calculation results to the corresponding rasters to generate a terrain model comprises:
fitting a distribution function according to statistical data of the terra-mechanical characteristics; and
performing a weighted calculation of values of the equivalent stiffness modulus and equivalent friction modulus of different terrain types according to the corresponding probabilities of the terrain types, and assigning calculation results to the corresponding rasters to generate the terrain model.

### Claim 15 (depends on 1)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 1, wherein the step of estimating the input images in terms of terra-mechanical characteristics, and updating calculation results to the corresponding rasters to generate a terrain model comprises:
fitting a distribution function according to statistical data of the terra-mechanical characteristics; and
using a mean obtained by fitting as estimation values of the equivalent stiffness modulus and equivalent friction modulus, and assigning the estimation values to the corresponding rasters to generate the terrain model.

### Claim 16 (depends on 1)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 1, wherein the step of processing the input images comprises:
collecting terrain images as a data source, annotating a target terrain in the collected terrain images and generating corresponding mask labels, and randomly dividing the annotated terrain images into training sets and validation sets in proportion;
calculating a mean and a variance of four RGBD channels in the training sets obtained after combination of the color images and the depth images, respectively, performing a data normalization on each of the input images, and performing an enhancement on original images in the training sets while correcting the corresponding mask labels; and
constructing a convolutional neural network, performing a training on a semantic segmentation model, and selecting a model with the highest accuracy as the semantic segmentation model for performing a terrain semantic segmentation on the input images.

### Claim 17 (depends on 1)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 1, wherein the step of processing the input images to generate point clouds comprises:
emitting a laser light to a target region and receiving through a receiver to obtain point cloud data of the target region with three-dimensional coordinate information, and obtaining color images of the target region for combining with the point cloud data via coordinate transformation through semantic segmentation to generate the point clouds.

### Claim 18 (depends on 1)

The terrain modeling method that fuses geometric characteristics and mechanical characteristics according to claim 1, wherein the step of obtaining input images of a detected region and processing the input images to generate point clouds comprises:
obtaining color images and depth images of the detected region, and combining the color images with the depth images at the same time to generate the point clouds via coordinate transformation.

### Claim 19

A computer readable storage medium having stored therein a computer program that, when read and executed by a processor of a computer, causes the computer to perform a terrain modeling method that fuses geometric characteristics and mechanical characteristics to execute:
obtaining input images of a detected region, and processing the input images to generate point clouds comprising elevation values of the input images;
mapping the point clouds to a raster map in a map coordinate system to generate corresponding rasters, and updating the elevation values to the corresponding rasters; and
calculating the input images in terms of terra-mechanical characteristics, and updating calculation results to the corresponding rasters to generate a terrain model.

### Claim 20

A terrain modeling system that fuses geometric characteristics and mechanical characteristics comprising:
a semantic point cloud generation module, configured to obtain input images of a detected region, and to process the input images to generate point clouds comprising elevation values of the input images;
a geometric characteristics obtaining module, configured to map the point clouds to a raster map in a map coordinate system to generate corresponding rasters;
a mechanical characteristics obtaining module, configured to update the elevation values to the corresponding rasters; and
a calculation and terrain model generation module, configured to calculate the input images in terms of terra-mechanical characteristics, and to update calculation results to the corresponding rasters to generate a terrain model.
