# US 2023/0055334 A1 · Road condition deep learning model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0055334 A1
- Document ID: US20230055334A1
- Status: Published application
- Kind: A1
- Application number: 17/978,287
- Filed: 2022-11-01
- Published: 2023-02-23
- Earliest claimed: 2020-06-05
- Assignees: Waymo LLC
- Applicants: Waymo LLC
- Inventors: Xin Zhou; Roshni Cooper; Michael James
- Web page: https://patentgrep.com/patent/US20230055334A1

## CPC

- G06N 20/00: Machine learning
- B60W 40/06: Road conditions
- B60W 60/0015: {specially adapted for safety}
- G05D 1/0231: {using optical position detecting means (position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16)}
- G05D 1/0255: {using acoustic signals, e.g. ultra-sonic singals (sonar systems designed for anti-collision purposes G01S15/93)}
- G05D 1/0257: {using a radar (radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931)}
- G05D 1/247: using signals provided by artificial sources external to the vehicle, e.g. navigation beacons
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2420/408: Radar; Laser, e.g. lidar
- B60W 2420/54: Audio sensitive means, e.g. ultrasound
- B60W 2555/20: Ambient conditions, e.g. wind or rain
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES

## Abstract

The technology relates to using on-board sensor data, off-board information and a deep learning model to classify road wemess and/or to perform a regression analysis on road wetness based on a set of input information. Such information includes on-board and/or off-board signals obtained from one or more sources including on-board perception sensors, other on-board modules. external weather measurement, external weather services, etc. The ground truth includes measurements of water film thickness and/or ice coverage on road surfaces. The ground truth, on-board and off-board signals are used to build the model. The constructed model can be deployed in autonomous vehicles for classifying/regressing the road wetness with on-board and/or off-board signals as the input, without referring to the ground truth. The model can be applied in a variety of ways to enhance autonomous vehicle operation, for instance by altering current driving actions, modifying planned routes or trajectories, activating on-board cleaning systems, etc.

## Claims

### Claim 1

A method for generating a road condition deep learning model, the method comprising:
obtaining as a first set of training inputs, by one or more processors, sensor data of an environment along a portion of a roadway;
obtaining as a second set of training inputs, by the one or more processors, information associated with the portion of the roadway:
evaluating. by the one or more processors, the first set of training inputs and the second set of training inputs with respect to wetness ground truth data for the portion of the roadway, to classify or estimate wetness along one or more areas of the portion of the roadway:
generating the road condition deep learning model based upon the evaluation; and
storing the generated road condition deep learning model in memory.

### Claim 2 (depends on 1)

The method of claim 1. wherein the obtained sensor data includes one or more of lidar returns, camera imagery, radar returns, audio signals, or output from a vehicle on-board module.

### Claim 3 (depends on 1)

The method of claim 1, wherein the information associated with the portion of the roadway includes one or more of weather station information, a weather forecast, road graph data. crowdsourced information, or observations from one or more road users.

### Claim 4 (depends on 1)

The method of claim 1, further comprising performing signal fusion of some or all of the obtained sensor data and off-board information.

### Claim 5 (depends on 1)

The method of claim 1, further comprising applying weighting to different signals of the obtained sensor data.

### Claim 6 (depends on 1)

The method of claim 1, further comprising, prior to generating the road condition deep learning model:
performing a statistical analysis to determine which obtained sensor data correlates with the ground truth data:
removing information of one or more objects on the roadway from the obtained sensor data; or
limiting the obtained sensor data to a selected range or distance.

### Claim 7 (depends on 6)

The method of claim 6, further comprising deemphasizing any received sensor data that does not meet a correlation threshold with the wetness ground truth data.

### Claim 8 (depends on 1)

The method of claim 1, further comprising smoothing ground truth measurements of the wetness ground truth data prior to generating the road condition deep learning model.

### Claim 9 (depends on 1)

The method of claim 1, further comprising applying the road condition deep learning model to one or more roadway regions to either identify a probability of wetness for each of the regions or estimate water film depth for each of the region.

### Claim 10 (depends on 1)

The method of claim 1, further comprising causing one or more systems of a self-driving vehicle to perform at least one of altering a current driving action, modifying a planned route or trajectory. or activating an on-board cleaning system of the self-driving vehicle based on the road condition deep learning model output.

### Claim 11 (depends on 1)

The method of claim 1. wherein storing the generated road condition deep learning model in memory comprises storing the generated road condition deep learning model in memory of one or more vehicles that are not equipped with a ground truth measurement sensor, the stored road condition deep learning model configured for use in evaluating real-time road wetness.

### Claim 12

A method comprising:
receiving sensor data including imagery of a roadway, the sensor data having been captured by a vehicle driving on the roadway;
obtaining map information of the roadway, the map information comprising at least one of geometry information or lane information of the roadway;
performing regression analysis using a trained neural network using the imagery and the map information of the roadway: and
determining an estimated snow depth for one or more lanes of the roadway based on an output of the regression analysis.

### Claim 13 (depends on 12)

The method of claim 12, further comprising operating the vehicle in an autonomous driving mode based on determining the estimated snow depth for the one or more lanes of the roadway.

### Claim 14 (depends on 12)

The method of claim 12, wherein the neural network is training with training data including labeled road wetness ground truth examples.

### Claim 15

A method comprising:
obtaining, by one or more processors, sensor data from one or more sensors of a perception system of a vehicle, the one or more sensors being configured to detect objects or conditions in an environment around the vehicle:
using, by the one or more processors, a stored a road condition deep learning model to generate information associated with a discrete classification or continuous regression/estimation of road wetness based on the obtained sensor data: and
using, by the one or more processors, the generated information to issue a control signal associated with operation of the vehicle.

### Claim 16 (depends on 15)

The method of claim 15, wherein the control signal is configured to trigger a safety precaution for operation of the vehicle in an autonomous driving mode.

### Claim 17 (depends on 15)

The method of claim 15, wherein the control signal is configured to alter a current driving action of the vehicle in an autonomous driving mode.

### Claim 18 (depends on 15)

The method of claim 15, wherein the control signal is configured to modify a trajectory for the vehicle.

### Claim 19 (depends on 15)

The method of claim 15, wherein the control signal is configured to activate an on-board cleaning system.

### Claim 20 (depends on 15)

The method of claim 15, wherein the control signal is configured to modify a threshold for one or more aspects of the perception system.

### Claim 21 (depends on 20)

The method of claim 20, wherein modification of the threshold for one or more aspects of the perception system includes modification of at least one of a threshold for filtering, a sensor noise level, a sensor field of view adaptation, a sensor validation logic, or a pedestrian detector.

### Claim 22 (depends on 21)

The method of claim 21, wherein the control signal is configured to change or update a model for predicting behavior of other road users.

### Claim 23 (depends on 21)

The method of claim 21, wherein the control signal is configured to modify a planned route for the vehicle.

### Claim 24 (depends on 23)

The method of claim 23, wherein modifying a planned route includes changing at least one of a pickup location, a drop-off location, or a lane of travel.
