# US 10,370,014 B2 · Vegetation detection and alert system for a railway vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,370,014 B2 - Document ID: US10370014B2 - Status: Granted - Kind: B2 - Application number: 16/166,945 - Filed: 2018-10-22 - Published: 2019-02-21 - Granted: 2019-08-06 - Earliest claimed: 2017-04-14 - Assignees: Bayer CropScience AG - Applicants: Bayer CropScience LP - Inventors: Kris Matson; Pascal Day; Lloyd Hinnant; Alvaro Ortiz Fernandez; Eduardo Back; Jonathon Brent Slone - Web page: https://patentgrep.com/patent/US10370014B2 ## CPC - B61L 23/041: {Obstacle detection} - B61L 25/025: {Absolute localisation, e.g. providing geodetic coordinates} - G06T 17/05: Geographic models - G06V 20/188: {Vegetation} - B61L 2205/04: Satellite based navigation systems, e.g. global positioning system [GPS] - B61L: GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC (brakes or auxiliary equipment B61H, B61K; point or crossing construction E01B) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present disclosure relates generally to vegetation detection and, in particular, to a vegetation detection and alert system for a railway vehicle. ## Claims ### Claim 1 An acquisition device mountable to a railway vehicle for detecting vegetation, the acquisition device comprising: an imaging sensor configured to capture images of an environment of the railway vehicle including a clearance zone around a railroad track on which the railway vehicle moves, the imaging sensor being configured to capture the images from the railway vehicle as the railway vehicle moves on the railroad track; a geospatial position sensor configured to determine a geolocation of the geospatial position sensor and thereby the environment imaged by the imaging sensor; and a processor coupled to the imaging sensor and geospatial position sensor, and programmed to at least: reference the images captured by the imaging sensor to the geolocation determined by the geospatial position sensor to form geospatial images of the environment including the clearance zone; and for one or more geospatial images of the geospatial images, process the one or more geospatial images to produce a geometric computer model of the environment in which objects in the environment are represented by a collection of geometry; detect vegetation in the clearance zone based on the geometric computer model, wherein a 3D volumetric model of the clearance zone is created for the instant in time of image capture; and upload the one or more geospatial images and a notification of detected vegetation from which a backend server is configured to generate an alert for a client. ### Claim 2 A method comprising detecting vegetation using an acquisition device mountable to a railway vehicle, the acquisition device including imaging sensors and a geospatial position sensor, the method comprising: capturing, by the imaging sensor, images of an environment of the railway vehicle including a clearance zone around a railroad track on which a railway vehicle moves, the imaging sensor capturing the images from the railway vehicle as the railway vehicle moves on the railroad track; determining, by the geospatial positioning sensor, a geolocation of the geospatial position sensor and thereby the environment imaged by the imaging sensor; referencing the images captured by the imaging sensor to the geolocation determined by the geospatial position sensor to form geospatial images of the environment including the clearance zone; and for one or more geospatial images of the geospatial images, processing the one or more geospatial images to produce a 3D geometric computer model of the environment in which objects in the environment are represented by a collection of geometry, wherein a 3D volumetric model of the clearance zone is created for the instant in time of image capture; detecting vegetation in the clearance zone based on the 3D geometric computer model; and uploading the one or more geospatial images and a notification of detected vegetation from which a backend server is configured to generate an alert for a client. ### Claim 3 (depends on 2) A method of claim 2, wherein consecutively captured image sets are used to construct consecutive, time-sliced volumes that are joined together to create an accurate clearance volume around the railroad track and along a rail bed. ### Claim 4 (depends on 3) A method of claim 3, wherein in parallel with the 3D photogrammetric volume construction, a point cloud of object surfaces in the images is generated using multiple image time slices to build the point cloud around the clearance zone, and further wherein subsequent to point cloud and clearance zone volume generation, the point cloud is intersected with the volume to classify points inside the 3D volume such that density and distances within the point cloud indicate the relative size of the objects and vegetation inside the clearance zone. ### Claim 5 (depends on 4) The method of claim 4 wherein imagery, the point cloud, and volume are joined in a machine learning algorithm to identify the types of objects intruding into the clearance zone. ### Claim 6 An acquisition device mountable to a railway vehicle for detecting objects, optionally vegetation, the acquisition device comprising: an imaging sensor configured to capture images of an environment of the railway vehicle including a volumetric clearance zone around a railroad track on which the railway vehicle moves, the imaging sensor being configured to capture images from the railway vehicle as the railway vehicle moves on the railroad track, wherein a 3D volumetric model of the clearance zone is created for the instant in time of image capture; a geospatial position sensor configured to determine a geolocation of the geospatial position sensor and together with computer vision as needed to thereby contribute to accurate localization of the environment and objects imaged by the imaging sensor; and a processor coupled to the imaging sensor and geospatial position sensor, and programmed to at least: reference the images captured by the imaging sensor to the geolocation determined by the geospatial position sensor, the geometric parameters of the imaging sensors and object properties in the images to form geospatially referenced images of the environment including the clearance zone; and for one or more geospatial images of the geospatial images, process the one or more geospatially referenced images to produce a localized geometric computer model of the 3D volumetric environment in which objects in the environment are represented by a collection of geometry and plurality of image pixels and whole images; detect and localize objects in the volumetric clearance zone based on the geometric computer model and the plurality of image pixels and whole images; and upload the one or more geospatial images and a notification of detected and localized objects from which an alert can optionally be generated. ### Claim 7 (depends on 6) A device of claim 6, wherein in parallel with 3D photogrammetric volume construction, there is provided a means for creating a point cloud of object surfaces in the images which can be generated using multiple image time slices, and further wherein there is provided means for utilizing computer vision and photogrammetric techniques to build the point cloud around the clearance zone. ### Claim 8 (depends on 6) A method comprising detecting objects, optionally vegetation, using an acquisition device of claim 6 mountable to a railway vehicle, the acquisition device including one or more imaging sensors and a geospatial position sensor, the method comprising: capturing, by the imaging sensor, images of an environment occupied by the railway vehicle including a volume clearance zone around a railroad track on which a railway vehicle moves, the imaging sensor capturing a plurality of images from the railway vehicle as the railway vehicle moves on the railroad track; determining, by the geospatial positioning sensor, a geolocation of the geospatial positioning sensor and thereby contributing to localization of the environment imaged by the imaging sensor; referencing the images captured by the imaging sensor to the geolocation determined by the geospatial position sensor and image object geometry to form geospatial images of the environment including the clearance zone; and for one or more geospatial images of the geospatial images, processing the one or more geospatial images to produce a 3D geometric computer model of the environment in which objects in the environment are represented by a collection of geometry and images; wherein a 3D volumetric model of the clearance zone is created for the instant in time of image capture detecting objects in the clearance zone based on the geometric computer model and images; and optionally uploading the one or more geospatial images to a desired format, and/or to an environment model, and a notification of detected object can optionally be generated. ### Claim 9 (depends on 8) The method of claim 8 wherein in parallel with 3D photogrammetric volume construction, a point cloud of object surfaces in the images is generated using multiple image time slices, and further wherein computer vision and photogrammetric techniques are used to build the point cloud around the clearance zone.