# US 2017/0282869 A1 · Road surface condition detection with multi-scale fusion

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0282869 A1
- Document ID: US20170282869A1
- Status: Published application
- Kind: A1
- Application number: 15/084,648
- Filed: 2016-03-30
- Published: 2017-10-05
- Earliest claimed: 2016-03-30
- Assignees: GM Global Technology Operations LLC
- Applicants: GM Global Technology Operations LLC
- Inventors: WEI TONG; QINGRONG ZHAO; BAKHTIAR B. LITKOUHI; SHUQING ZENG
- Web page: https://patentgrep.com/patent/US20170282869A1

## CPC

- B60T 8/171: Detecting parameters used in the regulation; Measuring values used in the regulation
- B60Q 9/00: Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 {, e.g. haptic signalling}
- B60T 7/12: for automatic initiation; for initiation not subject to will of driver or passenger {(limiting speed of vehicles other than rail vehicles B60K31/00)}
- B60T 7/22: initiated by contact of vehicle, e.g. bumper, with an external object, e.g. another vehicle {, or by means of contactless obstacle detectors mounted on the vehicle}
- B60T 8/175: Brake regulation specially adapted to prevent excessive wheel spin during vehicle acceleration, e.g. for traction control (safety devices for propulsion unit control responsive to, or preventing, skidding of wheels B60K28/16)
- B60T 2201/022: Collision avoidance systems
- B60T 2210/10: Detection or estimation of road conditions
- B60T 2210/12: Friction
- B60T: VEHICLE BRAKE CONTROL SYSTEMS OR PARTS THEREOF; BRAKE CONTROL SYSTEMS OR PARTS THEREOF, IN GENERAL (electrodynamic brake systems for vehicle, in general B60L; brakes per se, i.e. devices where braking effect occurs, including ultimate brake actuators, F16D); ARRANGEMENT OF BRAKING ELEMENTS ON VEHICLES IN GENERAL; PORTABLE DEVICES FOR PREVENTING UNWANTED MOVEMENT OF VEHICLES; VEHICLE MODIFICATIONS TO FACILITATE COOLING OF BRAKES
- B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL

## Abstract

A method of determining a surface condition of a path of travel. A plurality of images is captured of a surface of the path of travel by an image capture device. The image capture device captures images at varying scales. A feature extraction technique is applied by a feature extraction module to each of the scaled images. A fusion technique is applied, by the processor, to the extracted features for identifying the surface condition of the path of travel. A road surface condition signal provide to a control device. The control device applies the road surface condition signal to mitigate the wet road surface condition.

## Claims

### Claim 1

A method of determining a surface condition of a path of travel comprising the steps of:
obtaining data of a surface of the path of travel by sensor-based devices, the sensor-based devices capturing data at varying scales;
applying a feature extraction technique, by a feature extraction module, to each of the scaled data;
applying a fusion technique, by the processor, to the extracted features for identifying the surface condition of the path of travel; and
providing a road surface condition signal to a control device, the control device applying the road surface condition signal to mitigate the wet road surface condition.

### Claim 2 (depends on 1)

The method of claim 1 wherein each of the extracted features from each scale data are concatenated in a feature level fusion module of the processor.

### Claim 3 (depends on 2)

The method of claim 2 wherein an output from the feature level fusion module is input to a classifier for identifying the surface condition of the road.

### Claim 4 (depends on 1)

The method of claim 1 wherein each feature extracted by the feature extraction module is independently input to a classifier, wherein the classifier outputs a respective classified result for each extracted feature input to the processor.

### Claim 5 (depends on 4)

The method of claim 4 wherein the classifier provides the classified results for each extracted feature to a decision level fusion module of the processor, the decision level fusion module determining the surface condition of the path of travel based on fused classified results.

### Claim 6 (depends on 5)

The method of claim 5 wherein the fused classified results are determined based on the following formula:
C=Σi=1, . . . , kwiCi
where Ci is the classified result of each extracted feature by the classifier, wherein wi is the weighting function.

### Claim 7 (depends on 6)

The method of claim 6 wherein the weighting function is determined by a classification confidence of the classified result.

### Claim 8 (depends on 7)

The method of claim 7 wherein the weighting function is determined by prior knowledge of a scaling size of the captured image.

### Claim 9 (depends on 1)

The method of claim 1 wherein each of the extracted features are input to a feature level fusion module of the processor to generate a feature level fusion result and a decision level fusion module of the processor to generate a decision level fusion result.

### Claim 10 (depends on 9)

The method of claim 9 wherein the feature level fusion result is determined as a function of extracting features from each respective scaled data, wherein each extracted feature is input to the feature level fusion module for generating the feature level fusion result.

### Claim 11 (depends on 10)

The method of claim 10 wherein the feature level fusion result is input to a classifier for identifying the surface condition of the road as a function of the extracted features.

### Claim 12 (depends on 11)

The method of claim 11 wherein the decision level fusion result is determined as a function of extracting features from each respective scaled data by the decision level fusion module, wherein each feature extracted by the feature extraction module is independently input to the classifier, wherein the classifier outputs a respective classified result for each extracted feature, and wherein the classifier provides the classified results to the decision level fusion module for generating a road surface condition based on decision level fusion result.

### Claim 13 (depends on 12)

The method of claim 12 wherein the classified feature level fusion result and the decision level fusion result are input to a hybrid fusion module of the processor to generate a hybrid fusion result identifying the surface condition of the path of travel.

### Claim 14 (depends on 13)

The method of claim 13 wherein fusion module determines the hybrid fusion result based on the following formula:
Ci=wfCf+wdCd
where Cf is the feature level fusion result, wf is a classification confidence weight, Cd is decision level fusion result, and wd is a classification confidence result.

### Claim 15 (depends on 1)

The method of claim 1 wherein fusion results are determined as a function of extracting features from each respective scaled data, wherein each feature extracted by the feature extraction module for determining the fusion results is independently determined using a deep learning technique.

### Claim 16 (depends on 1)

The method of claim 1 wherein the road surface condition signal is provided to a vehicle controller, the controller using the road surface condition signal to determine an autonomous actuating vehicle braking strategy.

### Claim 17 (depends on 1)

The method of claim 1 wherein the road surface condition signal is provided to a vehicle controller, the controller using the road surface condition signal to determine a traction control strategy for autonomously actuating a traction control system.

### Claim 18 (depends on 1)

The method of claim 1 wherein the identified road surface condition signal is provided to a wireless communication system for alerting other vehicles of the identified surface condition on the path of travel.

### Claim 19 (depends on 1)

The method of claim 1 wherein the road surface condition signal alerts a driver of a potential reduced traction between vehicle tires and the road surface.

### Claim 20 (depends on 1)

The method of claim 1 wherein the identified road surface condition signal alerts a driver of the vehicle against a use of automated features.

### Claim 21 (depends on 1)

The method of claim 1 wherein the identified road surface condition signal is provided to a vehicle controller, the controller autonomously modifying a control setting of an automated control feature in response to the road surface condition signal.

### Claim 22 (depends on 1)

The method of claim 1 wherein the identified road surface condition signal is provided to a vehicle controller, the controller autonomously disabling cruise control in response to the road surface condition signal.

### Claim 23 (depends on 1)

The method of claim 1 wherein the identified road surface condition signal alerts a driver to reduce a vehicle speed.

### Claim 24 (depends on 1)

The method of claim 1 the data is obtained by active capture devices.

### Claim 25 (depends on 1)

The method of claim 1 the data is obtained by passive capture devices.