# US 10,872,419 B2 · Method and apparatus for evaluating a vehicle travel surface > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,419 B2 - Document ID: US10872419B2 - Status: Granted - Kind: B2 - Application number: 16/120,616 - Filed: 2018-09-04 - Published: 2020-03-05 - Granted: 2020-12-22 - Earliest claimed: 2018-09-04 - Assignees: GM Global Technology Operations LLC - Applicants: GM Global Technology Operations LLC - Inventors: Qingrong Zhao; Mohsen Khalili; Bakhtiar B. Litkouhi; Jingfu Jin; Jinsong Wang - Web page: https://patentgrep.com/patent/US10872419B2 ## CPC - G06T 7/194: involving foreground-background segmentation - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - B60W 40/06: Road conditions - G06T 1/0007: {Image acquisition} - G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529) - G06T 7/90: Determination of colour characteristics - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2552/00: Input parameters relating to infrastructure - B60W 2556/50: of positioning data, e.g. GPS [Global Positioning System] data - B60W 2556/55: using telemetry - G06T 2207/30256: Lane; Road marking - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT ## Abstract A vehicle subsystem includes an on-vehicle camera that is disposed to monitor a field of view (FOV) that includes a travel surface for the vehicle. A controller captures, via the on-vehicle camera, an image file associated with the FOV and segments the image file into a first set of regions associated with the travel surface and a second set of regions associated with an above-horizon portion. Image features on each of the first set of regions and the second set of regions are extracted and classified. A surface condition for the travel surface for the vehicle is identified based upon the classified extracted image features from each of the first set of regions and the second set of regions. Operation of the vehicle is controlled based upon the identified surface condition. ## Claims ### Claim 1 A vehicle subsystem, comprising: an on-vehicle camera disposed to monitor a field of view (FOV) that includes a travel surface for the vehicle; a controller, in communication with the on-vehicle camera, the controller including an instruction set, the instruction set executable to: capture, via the on-vehicle camera, an image file associated with the FOV; segment the image file into a first set of regions associated with the travel surface and a second set of regions associated with an above-horizon portion; extract image features from each of the first set of regions and the second set of regions; classify the extracted image features from each of the first set of regions and the second set of regions; assess a magnitude of ambient lighting; identify a travel surface condition for the vehicle based upon the classified extracted image features from the first set of regions and the second set of regions and the magnitude of ambient lighting; and communicate the identified travel surface condition to a second controller. ### Claim 2 (depends on 1) The vehicle subsystem of claim 1, wherein the instruction set executable to classify the extracted image features from each of the first set of regions and the second set of regions comprises the instruction set being executable to compare the extracted image features from each of the first set of regions and the second set of regions with contents of a classifier database. ### Claim 3 (depends on 2) The vehicle subsystem of claim 2, wherein the contents of the classifier database comprises a plurality of compiled feature sets and corresponding plurality of travel surface conditions, and wherein each of the plurality of compiled feature sets comprises extracted image features from each of the first set of regions and the second set of regions that have been predetermined for a predetermined one of the travel surface conditions. ### Claim 4 (depends on 1) The subsystem of claim 1, wherein the image file comprises a plurality of pixels, and wherein the image features comprise one of texture, brightness or color attributes associated with the pixels. ### Claim 5 (depends on 1) The subsystem of claim 1, further comprising the instruction set being executable to: extract a first set of image features from each of the first set of regions and the second set of regions employing a night-time feature extraction routine; extract a second set of image features from each of the first set of regions and the second set of regions employing a day-time feature extraction routine; concatenate the extracted first and second sets of image features to generate a third set of extracted image features; and classify the third set of extracted image features from each of the first set of regions and the second set of regions. ### Claim 6 (depends on 1) The subsystem of claim 1, wherein the travel surface condition comprises one of a dry state, a wet state, an iced state and a snow-covered state. ### Claim 7 (depends on 1) The subsystem of claim 1, comprising the instruction set being executable to extract image features associated with environmental light from the first set of regions and executable to extract image features associated with projection of environmental light onto the travel surface from the second set of regions. ### Claim 8 (depends on 7) The subsystem of claim 7, wherein the instruction set being executable to extract image features associated with environmental light from the first set of regions comprises compiling information from pixels of a portion of the image file associated with the first set of regions. ### Claim 9 (depends on 7) The subsystem of claim 7, wherein the instruction set being executable to extract image features associated with projection of environmental light onto the travel surface from the second set of regions comprises the instruction set being executable to compile information from pixels of a portion of the image file associated with the second set of regions. ### Claim 10 A method for controlling a vehicle including an on-vehicle camera disposed to monitor a field of view (FOV) that includes a travel surface for the vehicle, comprising: capturing, via the on-vehicle camera, an image file associated with the FOV; segmenting the image file into a first set of regions associated with the travel surface and a second set of regions associated with an above-horizon portion; extracting image features on each of the first set of regions and the second set of regions, including extracting a first set of image features from each of the first set of regions and the second set of regions employing a night-time feature extraction routine and extracting a second set of image features from each of the first set of regions and the second set of regions employing a day-time feature extraction routine; concatenating the extracted first and second sets of image features to generate a third set of extracted image features; classifying the third set of extracted image features from each of the first set of regions and the second set of regions; identifying a travel surface condition for the vehicle based upon the classified third set of extracted image features from each of the first set of regions and the second set of regions; and controlling operation of the vehicle based upon the identified travel surface condition. ### Claim 11 (depends on 10) The method of claim 10, further comprising extracting image features associated with environmental light from the first set of regions and extracting image features associated with projection of environmental light onto the travel surface from the second set of regions. ### Claim 12 (depends on 11) The method of claim 11, wherein extracting image features associated with environmental light from the first set of regions comprises compiling information from pixels of a portion of the image file associated with the first set of regions. ### Claim 13 (depends on 10) The method of claim 10, wherein classifying the extracted image features from each of the first set of regions and the second set of regions comprises comparing the extracted image features from each of the first set of regions and the second set of regions with contents of a classifier database. ### Claim 14 (depends on 10) The method of claim 10, wherein each of the plurality of compiled feature sets comprises extracted image features from each of the first set of regions and the second set of regions that have been predetermined for a predetermined one of the travel surface conditions. ### Claim 15 A method for monitoring, via a camera in communication with a controller, a field of view (FOV) that includes a travel surface, comprising: capturing, via the camera, an image file associated with the FOV; communicating the image file to the controller; segmenting, via the controller, the image file into a first set of regions associated with the travel surface and a second set of regions associated with an above-horizon portion; extracting image features on each of the first set of regions and the second set of regions, including extracting a first set of image features from each of the first set of regions and the second set of regions employing a night-time feature extraction routine and extracting a second set of image features from each of the first set of regions and the second set of regions employing a day-time feature extraction routine; concatenating the extracted first and second sets of image features to generate a third set of extracted image features; classifying the third set of extracted image features from each of the first set of regions and the second set of regions; identifying a surface condition for the travel surface based upon the classified third set of extracted image features from each of the first set of regions and the second set of regions, wherein the surface condition includes one of a dry state, a wet state, an iced state and a snow-covered state; and communicating, via the controller, the identified surface condition for the travel surface to an infrastructure monitoring device. ### Claim 16 (depends on 15) The method of claim 15, further comprising extracting image features associated with environmental light from the first set of regions and extracting image features associated with projection of environmental light onto the travel surface from the second set of regions. ### Claim 17 (depends on 16) The method of claim 16, wherein extracting image features associated with environmental light from the first set of regions comprises compiling information from pixels of a portion of the image file associated with the first set of regions.