# US 2023/0264688 A1 · Method, computer-implemented tool, vehicle control unit, and vehicle for locating obstacles in vehicle roadway regions, which are characterized by landmarks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0264688 A1 - Document ID: US20230264688A1 - Status: Published application - Kind: A1 - Application number: 18/017,972 - Filed: 2021-07-15 - Published: 2023-08-24 - Earliest claimed: 2020-08-07 - Assignees: Siemens Mobility GmbH - Applicants: Siemens Mobility GmbH - Inventors: Frederik Deroo; Kristian Weiss - Web page: https://patentgrep.com/patent/US20230264688A1 ## CPC - B60W 30/08: {Active safety systems} predicting or avoiding probable or impending collision {or attempting to minimise its consequences} - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - B60W 40/06: Road conditions - G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road} - B60W 2420/403: Image sensing, e.g. optical camera - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The disclosed locates obstacles in vehicle roadway regions, without needing to precisely locate the obstacles or landmarks associating sensor measurement data, which is detected by a vehicle sensor system and belongs to sensor measurement objects and by obstacles or landmarks can be represented, with stored landmark data; and ascertaining a sensor detection-specific locating distance between a sensor measurement object based on associated and unassociated sensor measurement data in that pieces of sensor detection-specific information contained in the sensor measurement data relating to the locating distance to be ascertained are put into a relationship with one another, an obstacle in a vehicle roadway region is located is determined on a roadway region constituting an obstacle to the vehicle if the sensor measurement object with unassociated sensor measurement data on the roadway can be correlated at least on the basis of the sensor detection-specific locating distance. ## Claims ### Claim 1 A method for locating obstructing objects in vehicle roadway regions characterized by landmarks, in which method a) primary sensor measurement data, detectable by at least one sensor of a vehicle sensor system, from a roadway region having at least one landmark and at least one obstructing object is used, the primary sensor measurement data belonging to at least one sensor measurement object that can represent one of the at least one landmark, and secondary sensor measurement data is used, belonging to at least one sensor measurement object that can represent one of the at least one obstructing object, b) stored landmark reference data, stored in particular in the form of map material, representing the at least one landmark is used, characterized by c) associating the primary sensor measurement data and the secondary sensor measurement data with the landmark reference data representing the at least one landmarks for landmark association, wherein the data association between the primary sensor measurement data and the landmark reference data has been accomplished and consequently, by associated primary sensor measurement data, a landmark association for the at least one sensor measurement object is possible and therefore the respective landmark is represented by the associated primary sensor measurement data, the data association between the secondary sensor measurement data and the landmark reference data has not been accomplished and consequently a landmark association for the at least one sensor measurement object is impossible, d) ascertaining sensor-capture-specific locating distances between the at least one non-associable sensor measurement object and the at least one landmark represented by the respective associated primary sensor measurement data on the basis of the associated primary sensor measurement data and the secondary sensor measurement data by putting sensor-capture-specific information items contained in the associated primary sensor measurement data and in the secondary sensor measurement data relating to the locating distances to be ascertained into a relationship with one another, e) locating the at least one obstructing object on a roadway in the roadway region, where it constitutes an obstacle for a vehicle, if the at least one non-associable sensor measurement object can be associated with the roadway, either on the basis of the ascertained sensor-capture-specific locating distances, or on the basis of the ascertained sensor-capture-specific locating distances with the addition of the landmark reference data and/or stored reference object data, stored in particular in the form of map material, which represents at least one reference object, in particular a noise protection wall on a road or an overhead line mast on a railroad track, of the vehicle roadway region, in addition to the at least one landmark, having a defined, constant distance from a roadway of the vehicle roadway region, and otherwise, if the at least one non-associable sensor measurement object cannot be associated with the roadway, then the at least one obstructing object is located outside of the roadway in the roadway region, where it does not constitute an obstacle for the vehicle. ### Claim 2 (depends on 1) The method according to claim 1, wherein the locating of the at least one obstructing object with the sensor measurement data capture, the data association and the locating distance ascertainment is performed dynamically at different locating times, at least in relation to the sensor data capture and the data association, for purposes of object tracking. ### Claim 3 (depends on 1) The method according to claim 1 , wherein the landmark reference data contains data that represents a grouping or formation of landmarks for a group-based or formation-based landmark association. ### Claim 4 (depends on 1) The method according to claim 1, wherein if the type of measurement data capture of the at least one sensor of the vehicle sensor system is based on a Cartesian coordinate system, a spherical coordinate system or a cylindrical coordinate system, in particular if the at least one sensor is used as a radar or a lidar sensor, the sensor-capture-specific information of the associated primary sensor measurement data and secondary sensor measurement data is Cartesian coordinate information, spherical coordinate information or cylindrical coordinate information, with which the locating distances are ascertained by angle difference determination or by determining a Euclidean distance as the putting-into-a-relationship with one another. ### Claim 5 (depends on 1) The method according to claim 1, wherein if the type of measurement data capture of the at least one sensor of the at least one vehicle sensor system is based on an imaging principle, in particular if the at least one sensor is used as a camera sensor, the sensor-capture-specific information of the associated primary sensor measurement data and secondary sensor measurement data is pixel information, with which the locating distances are ascertained by determining pixel distances as the putting-into-a-relationship with one another ### Claim 6 (depends on 1) The method according to claim 1, wherein the data association landmark association is performed with the aid of probabilistic factor graph methods, which are implemented with software libraries such as “Georgia Tech Smoothing and Mapping ” or “ as a General Framework for Graph Optimization”, or association/optimization methods such as “Global Nearest Neighbor ”, Munkres algorithms, “Joint Probabilistic Data Association ” or “Multi-Hypothesis-Tracking ”. ### Claim 7 (depends on 6) The method according to claim 6, wherein the landmark association is filtered with the aid of probabilistic factor graph methods over multiple time steps. ### Claim 8 A computer-implemented tool for locating obstructing objects in vehicle roadway regions characterized by landmarks, comprising a nonvolatile readable memory in which processor-readable control program commands of a program module for locating obstructing objects are stored, and a processor that is connected to the memory and executes the control program commands of the program module for locating obstructing objects, in which a) primary sensor measurement data, detectable by at least one sensor of a vehicle sensor system, from a roadway region having at least one landmark and at least one obstructing object, the primary sensor measurement data belonging to at least one sensor measurement object that can represent one of the at least one landmark, and secondary sensor measurement data, belonging to at least one sensor measurement object that can represent one of the at least one obstructing object, is input into the processor, in particular read in by the processor or uploaded into the processor, b) stored landmark reference data, stored in particular in the form of map material, which represents the at least one landmark, is input into the processor, in particular read in by the processor or uploaded into the processor, wherein c) the processor and the program module for locating obstructing objects are designed in such a way that c1) the primary sensor measurement data and the secondary sensor measurement data are associated with the landmark reference data representing the at least one landmark for landmark association, wherein the data association between the primary sensor measurement data and the landmark reference data has been accomplished and consequently, by means of associated primary sensor measurement data, a landmark association for the at least one sensor measurement object is possible and therefore the respective landmark is represented by the associated primary sensor measurement data, the data association between the secondary sensor measurement data and the landmark reference data has not been accomplished and consequently a landmark association for the at least one sensor measurement object is impossible, d) sensor-capture-specific locating distances between the at least one non-associable sensor measurement object SMOHO and the at least one landmark represented by the respective associated primary sensor measurement data are ascertained on the basis of the associated primary sensor measurement data and the secondary sensor measurement data by putting sensor-capture-specific information items contained in the associated primary sensor measurement data and in the secondary sensor measurement data relating to the locating distances to be ascertained in a relationship with one another, e) the at least one obstructing object is located on a roadway in the roadway region, where it constitutes an obstacle for a vehicle, if the at least one non-associable sensor measurement object of can be associated with the roadway, either on the basis of the ascertained sensor-capture-specific locating distances, or on the basis of the ascertained sensor-capture-specific locating distances with the addition of the landmark of map material, which represents at least one reference object, in particular a noise protection wall on a road or an overhead line mast on a railroad track, of the vehicle roadway region, in addition to the at least one landmark, having a defined, constant distance from a roadway of the roadway region, and otherwise, if the at least one non-associable sensor measurement object cannot be associated with the roadway, the at least one obstructing object is located outside of the roadway in the roadway region, where it does not constitute an obstacle for the vehicle. ### Claim 9 (depends on 8) The computer-implemented tool according to claim 8, wherein the processor and the program module for locating obstructing objects are designed in such a way that the locating of the at least one obstructing object with the sensor measurement data capture, the data association and the locating distance ascertainment is performed dynamically at different locating times, at least in relation to the sensor data capture and the data association, for the purposes of object tracking. ### Claim 10 (depends on 8) The computer-implemented tool according to claim 8, wherein the landmark reference data contains data that represents a grouping or formation of landmarks for a group-based or formation-based landmark association. ### Claim 11 (depends on 8) The computer-implemented tool according to claim 8, wherein if the type of measurement data capture of the at least one sensor of the vehicle sensor system is based on a Cartesian coordinate system, a spherical coordinate system or a cylindrical coordinate system, in particular if the at least one sensor is used as a radar or a lidar sensor, the sensor-capture-specific information of the associated primary sensor measurement data and secondary sensor measurement data is Cartesian coordinate information, spherical coordinate information or cylindrical coordinate information, with which the locating distances are ascertained by angle difference determination or by determining a Euclidean distance as the putting-into-a-relationship with one another. ### Claim 12 (depends on 8) The computer-implemented tool according to claim 8, wherein if the type of measurement data capture of the at least one sensor of the vehicle sensor system is based on an imaging principle, in particular if the at least one sensor is used as a camera sensor, the sensor-capture-specific information of the associated primary sensor measurement data and secondary sensor measurement data is pixel information, with which the locating distances are ascertained by determining pixel distances as the putting-into-a-relationship with one another. ### Claim 13 (depends on 8) The computer-implemented tool according to claim 8, wherein processor and the program module for locating obstructing objects arc designed such that the data association for landmark association is performed with the aid of probabilistic factor graph methods, which are implemented with software libraries such as “Georgia Tech Smoothing and Mapping ” or “ as a General Framework for Graph Optimization”, or association/optimization methods such as “Global Nearest Neighbor ”, Munkres algorithms, “Joint Probabilistic Data Association ” or “Multi-Hypothesis-Tracking ”. ### Claim 14 (depends on 13) The computer-implemented tool according to claim 13, wherein the processor and the program module for locating obstructing objects arc designed such that the landmark association is filtered with the aid of probabilistic factor graph methods over multiple time steps. ### Claim 15 (depends on 8) A vehicle control unit for locating obstructing objects in vehicle roadway regions characterized by landmarks, wherein a storage device for storing a) primary sensor measurement data, detectable by at least one sensor of a vehicle sensor system, from a roadway region having at least one landmark and at least one obstructing object, the primary sensor measurement data belonging to at least one sensor measurement object that can represent one of the at least one landmark, and secondary sensor measurement data belonging to at least one sensor measurement object that can represent one of the at least one obstructing object, and b) landmark reference data representing at least one landmark, stored in particular in the form of map material, is either connected to a control device in the vehicle control unit (option A) or is associated with a control device in the vehicle control unit (option B), characterized by a computer-implemented tool according to claim 8 that can be uploaded into the control device. ### Claim 16 (depends on 15) A vehicle for locating obstacles in vehicle roadway regions characterized by landmarks, comprising a vehicle sensor system in which at least one sensor is contained, and wherein primary sensor measurement data, belonging to at least one sensor measurement object that can represent the respective landmark, and secondary sensor measurement data belonging to at least one sensor measurement object that can represent the respective obstructing object, is captured with the at least one sensor from a roadway region having at least one landmark and at least one obstructing object, and comprising the vehicle control unit as claimed in claim 15 that is connected to the vehicle sensor system.