# US 9,435,885 B2 · Road-terrain detection method and system for driver assistance systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,435,885 B2 - Document ID: US9435885B2 - Status: Granted - Kind: B2 - Application number: 13/558,407 - Filed: 2012-07-26 - Published: 2013-03-28 - Granted: 2016-09-06 - Earliest claimed: 2011-09-28 - Assignees: Honda Research Institute Europe GmbH - Applicants: Jannik Fritsch; Tobias Kühnl - Inventors: Jannik Fritsch; Tobias Kühnl - Web page: https://patentgrep.com/patent/US9435885B2 ## CPC - G01S 13/89: for mapping or imaging - B62D 6/007: {adjustable by the driver, e.g. sport mode} - G01S 13/93: for anti-collision purposes - G01S 15/89: for mapping or imaging - G01S 15/93: for anti-collision purposes - G01S 17/89: for mapping or imaging - G01S 17/93: for anti-collision purposes - G06T 7/11: Region-based segmentation - G06T 7/143: involving probabilistic approaches, e.g. Markov random field [MRF] modelling - G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road} - G06T 2207/20081: Training; Learning - G06T 2207/30256: Lane; Road marking - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - B62D: MOTOR VEHICLES; TRAILERS (steering, or guiding on a desired track, of agricultural machines or implements A01B69/00; wheels, castors, axles, increasing wheel adhesion B60B; vehicle tyres, tyre inflation or tyre changing B60C; connections between vehicles of a train or the like B60D; vehicles for use on rail and road, amphibious or convertible vehicles B60F; suspension arrangements B60G; heating, cooling, ventilating or other air treating devices B60H; windows, windscreens, non-fixed roofs, doors or similar devices, protective coverings for vehicles not in use B60J; propulsion plant arrangements, auxiliary drives, transmissions, controls, instrumentation or dashboards B60K; electric equipment or propulsion of electrically-propelled vehicles B60L; power supply for electrically-propelled vehicles B60M; passenger accommodation not otherwise provided for B60N; adaptations for load transportation or to carry special loads or objects B60P; arrangement of signalling or lighting devices, the mounting or supporting thereof or circuits therefor, for vehicles in general B60Q; vehicles, vehicle fittings or vehicle parts not otherwise provided for B60R; servicing, cleaning, repairing, supporting, lifting, or manoeuvring, not otherwise provided for, B60S; brake arrangements, brake control systems or parts thereof B60T; air-cushion vehicles B60V; motor-cycles, accessories therefor B62J, B62K; testing of vehicles G01M) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present invention describes a road terrain detection system that comprises a method for classifying selected locations in the environment of a vehicle based on sensory input signals such as pixel values of a camera image. The method comprises a high level spatial feature generation for selected locations in the environment called base points. The spatial feature generation of the base points is based on a value-continuous confidence representation that captures visual and physical properties of the environment, generated by so called base classifiers operating on raw sensory data. Consequently, the road terrain detection incorporates both local properties of sensor data and their spatial relationship in a two-step feature extraction process. ## Claims ### Claim 1 A road-terrain detection method for driver assistance systems, wherein the road-terrain detection method comprises sensing the environment of a vehicle with at least one sensor; transforming a sensor signal from the at least one sensor into at least one confidence map of local properties of the environment by using at least one base classifier; generating, for base points on the at least one confidence map, each reflecting a specific location in the environment of the vehicle, spatial features for the local properties based on the at least one confidence map; and classifying, based on the generated spatial features, locations in the environment of the vehicle to a certain category of road terrain corresponding to a type of surface of a road, wherein the generation of a spatial feature for a specific location in the environment of the vehicle comprises extracting a ray, wherein a ray is defined as a directed line with a certain angular orientation starting from the location in the environment, and analyzing confidence values along the ray to extract spatial feature. ### Claim 2 (depends on 1) The method according to claim 1, wherein the at least one sensor is one or more cameras, radars, laser scanners or GPS\navigation systems. ### Claim 3 (depends on 1) The method according to claim 1, wherein based on the classified locations in the environment a visual or acoustic signal is generated or an effector of the vehicle is operated. ### Claim 4 (depends on 3) The method according to claim 3, wherein the effector comprises at least one of a steering wheel, an accelerator, or a brake. ### Claim 5 (depends on 1) The method according to claim 1, wherein the method applied for a specific road terrain is automatically parameterized by using positive and negative samples, which are given by training regions. ### Claim 6 (depends on 5) The method according to claim 5, wherein the specific road terrain term comprises at least one of “road-like area”, “drivable road”, “ego-lane”, “non-ego-lane”, “non-drivable road”, “sidewalk”, “traffic island”, or “off-limits terrain”. ### Claim 7 (depends on 5) The method according to claim 5, wherein the training regions comprise polygons. ### Claim 8 (depends on 1) The method according to claim 1 comprising a step of sensory preprocessing for all sensor signals to obtain a common representation of the environment. ### Claim 9 (depends on 8) The method according to claim 8, wherein the sensory preprocessing is an inverse perspective mapping of a camera image to get a metric image representation. ### Claim 10 (depends on 1) The method according to claim 1, wherein the analysis of the confidence values along the ray is performed by integrating the confidence values along the ray; extracting the ray length, at which the integral exceeds an absorption threshold, which is a certain numeric value. ### Claim 11 (depends on 10) The method according to claim 10, wherein for a specific location in the environment at least one ray and at least one absorption threshold is used to generate a feature vector that encodes the relative position to a local property given by a confidence map. ### Claim 12 (depends on 1) The method according to claim 1 comprising a base classifier for road classification that generates a confidence map, wherein locations in the environment that correspond to a road-like area have high confidence values. ### Claim 13 (depends on 1) The method according to claim 1 comprising a base classifier for lane marking detection that generates a confidence map, wherein locations in the environment that correspond to lane markings have high confidence values. ### Claim 14 (depends on 1) A road-terrain detection system for driver assistance systems, the road-terrain detection system being adapted to perform a method according to claim 1. ### Claim 15 A road-terrain detection system for driver assistance systems, the road-terrain detection system being adapted to sense with at least one sensor the environment of a vehicle; transform at least one sensor signal from at least one sensor into at least one confidence map of local properties of the environment by using at least one base classifier; generate, for base points on the at least one confidence map, each reflecting a specific location in the environment of the vehicle, spatial features for local properties based on the at least one confidence map; and classify, based on the generated spatial features, locations in the environment of the vehicle to a certain category of road terrain corresponding to a type of surface of a road, wherein the generation of a spatial feature for a specific location in the environment of the vehicle comprises extracting a ray, wherein a ray is defined as a directed line with a certain angular orientation starting from the location in the environment, and analyzing confidence values along the ray to extract spatial feature. ### Claim 16 (depends on 15) The system according to claim 15, wherein the road terrain detection system can select one out of multiple road terrain classification outputs as a system output, in order to internally have different subsystems for coping with different weather conditions resulting in different visual properties; different road types resulting in different physical and visual properties. ### Claim 17 (depends on 15) A land vehicle, being provided with a system according to claim 15. ### Claim 18 A road-terrain detection method for driver assistance systems, wherein the road-terrain detection method comprises sensing the environment of a vehicle with at least one sensor; transforming a sensor signal from the at least one sensor into at least one confidence map of local properties of the environment by using at least one base classifier; generating, for base points on the at least one confidence map, each reflecting a specific location in the environment of the vehicle, spatial features for the local properties based on the at least one confidence map; and classifying, based on the generated spatial features, locations in the environment of the vehicle to a certain category of road terrain corresponding to the type of surface of the road, wherein the method comprises a base classifier for visual boundary classification, in order to find visual features that discriminate a road boundary from a road-like area, wherein the method includes feature generation, uses a positive training set with polygonal training regions for the road boundary, and uses a negative training set with polygonal training regions for road-like area excluding areas with lane-markings. ### Claim 19 (depends on 18) The method according to claim 18, wherein the visual features comprise curbstones. ### Claim 20 (depends on 18) The method according to claim 18, wherein feature generation comprises generating at least one of color, texture, appearance, flow, or depth. ### Claim 21 A non-transitory computer-readable medium encoded with instructions that, when run on a computing device in a land vehicle, perform a method, the method comprising: sensing the environment of a vehicle with at least one sensor; transforming a sensor signal from the at least one sensor into at least one confidence map of local properties of the environment by using at least one base classifier; generating, for base points on the at least one confidence map, each reflecting a specific location in the environment of the vehicle, spatial features for the local properties based on the at least one confidence map; and classifying, based on the generated spatial features, locations in the environment of the vehicle to a certain category of road terrain corresponding to a type of surface of a road, wherein the generation of a spatial feature for a specific location in the environment of the vehicle comprises extracting a ray, wherein a ray is defined as a directed line with a certain angular orientation starting from the location in the environment, and analyzing confidence values along the ray to extract spatial feature. ### Claim 22 A road-terrain detection system for driver assistance systems, the road-terrain detection system being adapted to sense the environment of a vehicle with at least one sensor; transform a sensor signal from the at least one sensor into at least one confidence map of local properties of the environment by using at least one base classifier; generate, for base points on the at least one confidence map, each reflecting a specific location in the environment of the vehicle, spatial features for the local properties based on the at least one confidence map; and classify, based on the generated spatial features, locations in the environment of the vehicle to a certain category of road terrain corresponding to the type of surface of the road, wherein a base classifier is provided for visual boundary classification, in order to find visual features that discriminate a road boundary from a road-like area, wherein the system is adapted to include feature generation, use a positive training set with polygonal training regions for the road boundary, and use a negative training set with polygonal training regions for road-like area excluding areas with lane-markings.