# US 2022/0032946 A1 · Apparatus and methodology of road condition classification using sensor data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0032946 A1 - Document ID: US20220032946A1 - Status: Published application - Kind: A1 - Application number: 16/944,911 - Filed: 2020-07-31 - Published: 2022-02-03 - Earliest claimed: 2020-07-31 - Assignees: GM Global Technology Operations LLC - Applicants: GM Global Technology Operations LLC - Inventors: Qingrong Zhao; Bakhtiar B. Litkouhi; Mohsen Khalili - Web page: https://patentgrep.com/patent/US20220032946A1 ## CPC - B60W 60/001: {Planning or execution of driving tasks} - G06V 10/764: using classification, e.g. of video objects - B60R 19/48: combined with, or convertible into, other devices or objects, e.g. bumpers combined with road brushes, bumpers convertible into beds - B60W 30/18: Propelling the vehicle - B60W 40/06: Road conditions - B60W 40/105: Speed - G01S 7/417: {involving the use of neural networks} - G01S 13/605: {using a pattern, backscattered from the ground, to determine speed or drift by measuring the time required to cover a fixed distance} - G01S 13/87: Combinations of radar systems, e.g. primary radar and secondary radar - G01S 13/931: of land vehicles - G06F 18/24: Classification techniques - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06V 10/40: Extraction of image or video features - G06V 10/803: {of input or preprocessed data} - G06V 10/82: using neural networks - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - B60R 19/483: {with obstacle sensors of electric or electronic type} - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2420/408: Radar; Laser, e.g. lidar - B60W 2520/10: Longitudinal speed - B60W 2552/00: Input parameters relating to infrastructure - B60W 2555/20: Ambient conditions, e.g. wind or rain - G01S 13/86: Combinations of radar systems with non-radar systems, e.g. sonar, direction finder - G01S 2013/932: {using own vehicle data, e.g. ground speed, steering wheel direction} - G01S 2013/9323: {Alternative operation using light waves} - G01S 2013/9324: {Alternative operation using ultrasonic waves} - G01S 2013/93271: {in the front of the vehicles} - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods and systems are provided for controlling a vehicle action based on a condition of a road on which a vehicle is travelling, including: obtaining first sensor data as to a surface of the road from one or more first sensors onboard the vehicle; obtaining second sensor data from one or more second sensors onboard the vehicle as to a measured parameter pertaining to operation of the vehicle or conditions pertaining thereto; generating a plurality of road surface channel images from the first sensor data, wherein each road surface channel image captures one of a plurality of facets of properties of the first sensor data; classifying, via a processor using a neural network model, the condition of the road on which the vehicle is travelling, based on the measured parameter and the plurality of road surface channel images; and controlling a vehicle action based on the classification of the condition of the road. ## Claims ### Claim 1 A method for controlling a vehicle action based on a condition of a road on which a vehicle is travelling, the method comprising: obtaining first sensor data as to a surface of the road from one or more first sensors onboard the vehicle; obtaining second sensor data from one or more second sensors onboard the vehicle as to a measured parameter pertaining to operation of the vehicle or conditions pertaining thereto; generating a plurality of road surface channel images from the first sensor data, wherein each road surface channel image captures one of a plurality of facets of properties of the first sensor data; classifying, via a processor using a neural network model, the condition of the road on which the vehicle is travelling, based on the measured parameter and the plurality of road surface channel images; and controlling a vehicle action based on the classification of the condition of the road. ### Claim 2 (depends on 1) The method of claim 1, wherein the parameter comprises a speed of the vehicle. ### Claim 3 (depends on 1) The method of claim 1, wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include returned energy at an (x,y) position from the first sensors. ### Claim 4 (depends on 1) The method of claim 1, wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include a Z coordinate at an (x,y) position from the first sensors. ### Claim 5 (depends on 1) The method of claim 1, wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include a Doppler value at an (x,y) position from the first sensors. ### Claim 6 (depends on 1) The method of claim 1, wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include a sensor index value at an (x,y) position from the first sensors. ### Claim 7 (depends on 1) The method of claim 1, wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include each of the following: (i) returned energy at an (x,y) position from the first sensors; (ii) a Z coordinate at the (x,y) position from the first sensors; (iii) a Doppler value at the (x,y) position from the first sensors; and (iv) a sensor index value at the (x,y) position from the first sensors. ### Claim 8 (depends on 2) The method of claim 2, further comprising: generating, via the processor, a speed image channel based on a categorization of the vehicle speed; and fusing, via the processor, the plurality of road surface channel images with the speed image channel; wherein the step of classifying the condition comprises classifying, via the processor using a neural network model, the condition of the road on which the vehicle is travelling, based on the fusing of the plurality of road surface channel images with the speed image channel. ### Claim 9 (depends on 2) The method of claim 2, further comprising: performing, via the processor, feature extraction from the plurality of road surface channel images; and performing feature level fusion between a categorization of the vehicle speed and the feature extraction form the plurality of road surface channel images; wherein the step of classifying the condition comprises classifying, via the processor using a neural network model, the condition of the road on which the vehicle is travelling, based on the feature level fusion between a categorization of the vehicle speed and the feature extraction form the plurality of road surface channel images. ### Claim 10 A system for controlling a vehicle action based on classifying a condition of a road on which a vehicle is travelling, the system comprising: one or more first sensors configured to provide first sensor data as to a surface of the road; one or more second sensors configured to provide second sensor data as to a measured parameter pertaining to operation of the vehicle or conditions pertaining thereto; and a processor disposed coupled to the first sensors and the second sensors and configured to: generate a plurality of road surface channel images from the first sensor data, wherein each road surface channel image captures one of a plurality of facets of properties of the first sensor data; classify, using a neural network model, the condition of the road on which the vehicle is travelling, based on the measured parameter and the plurality of road surface channel images; and provide instructions to control a vehicle action based on the condition of the road. ### Claim 11 A vehicle comprising: a body; a drive system disposed within the body and configured to drive the vehicle; and a control system coupled to the drive system, the control system comprising: one or more first sensors configured to provide first sensor data as to a surface of the road; one or more second sensors configured to provide second sensor data as to a measured parameter pertaining to operation of the vehicle or conditions pertaining thereto; and a processor disposed coupled to the first sensors and the second sensors and configured to: generate a plurality of road surface channel images from the first sensor data, wherein each road surface channel image captures one of a plurality of facets of properties of the first sensor data; classify, using a neural network model, the condition of the road on which the vehicle is travelling, based on the measured parameter and the plurality of road surface channel images; and provide instructions to control a vehicle action based on the condition of the road. ### Claim 12 (depends on 11) The vehicle of claim 11, wherein the parameter comprises a speed of the vehicle. ### Claim 13 (depends on 12) The vehicle of claim 12, wherein: the vehicle includes a front bumper; and the one or more first sensors comprise a plurality of ultra-short range radar (USRR) sensors disposed proximate the front bumper of the vehicle. ### Claim 14 (depends on 11) The vehicle of claim 11, wherein the condition comprises a surface condition of a surface of the road, as to whether the surface is wet, dry, or covered with snow. ### Claim 15 (depends on 11) The vehicle of claim 11, wherein the condition comprises a material of which a surface of the road is made. ### Claim 16 (depends on 11) The vehicle of claim 11, wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include returned energy at an (x,y) position from the first sensors. ### Claim 17 (depends on 11) The vehicle of claim 11, wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include a Doppler value at an (x,y) position from the first sensors. ### Claim 18 (depends on 11) The vehicle of claim 11, wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include a sensor index value at an (x,y) position from the first sensors. ### Claim 19 (depends on 12) The vehicle of claim 12, wherein the processor is further configured to: generate a speed image channel based on a categorization of the vehicle speed; fuse the plurality of road surface channel images with the speed image channel; and classify, using a neural network model, the condition of the road on which the vehicle is travelling, based on the fusing of the plurality of road surface channel images with the speed image channel. ### Claim 20 (depends on 12) The vehicle of claim 12, wherein the processor is further configured to: perform feature extraction from the plurality of road surface channel images; perform feature level fusion between a categorization of the vehicle speed and the feature extraction form the plurality of road surface channel images; and classify, using a neural network model, the condition of the road on which the vehicle is travelling, based on the feature level fusion between a categorization of the vehicle speed and the feature extraction form the plurality of road surface channel images.