# US 10,311,551 B2 · Machine vision based track-occupancy and movement validation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,551 B2 - Document ID: US10311551B2 - Status: Granted - Kind: B2 - Application number: 15/377,451 - Filed: 2016-12-13 - Published: 2018-06-14 - Granted: 2019-06-04 - Earliest claimed: 2016-12-13 - Assignees: Westinghouse Air Brake Technologies Corporation - Applicants: Westinghouse Air Brake Technologies Corporation - Inventors: Prahlad G. Menon Gopalakrishna - Web page: https://patentgrep.com/patent/US10311551B2 ## CPC - G06T 5/70: Denoising; Smoothing - B61L 15/0072: {On-board train data handling} - B61L 23/00: Control, warning or like safety means along the route or between vehicles or trains - B61L 23/041: {Obstacle detection} - G06T 7/0004: {Industrial image inspection} - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B61L: GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC (brakes or auxiliary equipment B61H, B61K; point or crossing construction E01B) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A train monitoring system includes an image sensor positioned on a rail vehicle to sense an area in front. The image sensor generates sensor signals. At least one non-transitory computer-readable medium has program instructions that, when executed by at least one processor in communication with the image sensor, causes the at least one processor to: (a) generate static image data based on the sensor signals, the static image data including a line representation of rails located in the area in front of the rail vehicle; (b) determine track occupancy of the rail vehicle by determining the set of rails occupied by the rail vehicle based on the static image data; and (c) determine an upcoming direction of movement of the rail vehicle by determining a direction of the set of rails occupied by the rail vehicle based on the static image data. ## Claims ### Claim 1 A train monitoring system comprising: an image sensor disposed on a rail vehicle and positioned to sense an area in front of the rail vehicle, the image sensor generating sensor signals; and at least one non-transitory computer-readable medium comprising program instructions that, when executed by at least one processor in communication with the image sensor, cause the at least one processor to: (a) generate static image data based, at least in part, on the sensor signals, the static image data comprising a line representation of rails located in the area in front of the rail vehicle; (b) determine a track occupancy of the rail vehicle by determining a set of rails occupied by the rail vehicle based, at least in part, on the static image data; (c) determine an upcoming direction of movement of the rail vehicle by determining a direction of the set of rails occupied by the rail vehicle based, at least in part, on the static image data; and (d) validate the determined track occupancy based, at least in part, on the upcoming direction of movement of the rail vehicle, wherein generating the static image data comprises filtering the sensor signals with a low pass filter to smoothen an instantaneous image of the area in front of the rail vehicle, and wherein generating the static image data comprises generating an enhanced smoothened image to exaggerate pixels of the smoothened image that belong to substantially tubular structures in the area in front of the rail vehicle. ### Claim 2 (depends on 1) The train monitoring system of claim 1, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to validate the determined track occupancy of the rail vehicle and the determined upcoming direction of movement of the rail vehicle based, at least in part, on time series data, wherein the time series data comprises data from the sensor signals obtained at two or more different times. ### Claim 3 (depends on 1) The train monitoring system of claim 1, wherein determining the track occupancy of the rail vehicle further comprises determining a number of sets of rails located in the area in front of the rail vehicle. ### Claim 4 (depends on 1) The train monitoring system of claim 1, wherein the image sensor is a monochrome camera, an RGB camera without depth perception, or a depth-sensing camera. ### Claim 5 (depends on 1) The train monitoring system of claim 1, further comprising an on-board computer system in communication with the image sensor, the on-board computer system comprising the at least one processor configured to perform steps (a)-(d). ### Claim 6 (depends on 1) The train monitoring system of claim 1, wherein the at least one processor comprises a microprocessor dedicated solely to processing sensor data from the image sensor. ### Claim 7 (depends on 1) The train monitoring system of claim 1, wherein generating the static image data comprises determining two largest components in the enhanced smoothened image in each quantified angular direction category. ### Claim 8 (depends on 7) The train monitoring system of claim 7, wherein generating the static image data comprises quantifying lines corresponding to detected connected-component regions representing the rails in front of the rail vehicle in each quantified angular direction category. ### Claim 9 (depends on 8) The train monitoring system of claim 8, wherein generating the static image data comprises separating lines representing rails into categories based, at least in part, on relative slopes of the rails. ### Claim 10 (depends on 1) The train monitoring system of claim 1, wherein determining the upcoming direction of movement of the rail vehicle comprises determining an upcoming angular direction of the set of rails occupied by the rail vehicle. ### Claim 11 (depends on 1) The train monitoring system of claim 1, further comprising an event data recorder in communication with the at least one processor, the event data recorder configured to store information from the sensor signals. ### Claim 12 (depends on 1) The train monitoring system of claim 1, wherein the program instructions cause the processor to detect obstacles along the rails based, at least in part, on the sensor signals from the image sensor. ### Claim 13 (depends on 1) The train monitoring system of claim 1, wherein the program instructions cause the processor to evaluate a condition of the rails based, at least in part, on the sensor signals from the image sensor. ### Claim 14 A method of monitoring a train comprising: sensing an area in front of a rail vehicle using an image sensor, the image sensor generating sensor signals; generating static image data based, at least in part, on the sensor signals with at least one processor in communication with the image sensor, wherein the static image data comprises a line representation of rails located in the area in front of the rail vehicle; determining, with the at least one processor, track occupancy of the rail vehicle by determining a set of rails occupied by the rail vehicle based, at least in part, on the static image data; determining, with the at least one processor, an upcoming direction of movement of the rail vehicle by determining a direction of the set of rails occupied by the rail vehicle based, at least in part, on the static image data; and validating the determined track occupancy based, at least in part, on the upcoming direction of movement of the rail vehicle, wherein generating the static image data comprises filtering the sensor signals with a low pass image filter to smoothen an instantaneous image of the area in front of the rail vehicle, and wherein generating the static image data comprises generating an enhanced smoothened image to exaggerate pixels of the smoothened image that belong to substantially tubular structures in the area in front of the rail vehicle. ### Claim 15 (depends on 14) The method of claim 14, further comprising: validating, with the at least one processor, the determined track occupancy of the rail vehicle and the determined upcoming direction of movement of the rail vehicle based, at least in part, on time series data, wherein the time series data comprises data from the sensor signals obtained at two or more different times. ### Claim 16 (depends on 14) The method of claim 14, further comprising: determining the track occupancy of the rail vehicle further comprises determining a number of sets of rails located in the area in front of the rail vehicle. ### Claim 17 (depends on 14) The method of claim 14, wherein generating the static image data comprises determining two largest the two largest components in the enhanced smoothened image in each quantified angular direction category. ### Claim 18 (depends on 17) The method of claim 17, wherein generating the static image data comprises quantifying lines corresponding to detected connected-component regions representing the rails in front of the rail vehicle in each quantified angular direction category. ### Claim 19 (depends on 18) The method of claim 18, wherein generating the static image data comprises separating lines representing rails into categories based, at least in part, on relative slopes of the rails. ### Claim 20 (depends on 14) The method of claim 14, wherein determining the upcoming direction of movement of the rail vehicle comprises determining an upcoming angular direction of the set of rails occupied by the rail vehicle. ### Claim 21 (depends on 14) The method of claim 14, wherein an event data recorder is in communication with the at least one processor, the event data recorder configured to store information from the sensor signals. ### Claim 22 (depends on 14) The method of claim 14, wherein the at least one processor is configured to detect obstacles along the rails based, at least in part, on the sensor signals from the image sensor. ### Claim 23 (depends on 14) The method of claim 14, wherein the at least one processor is configured to evaluate a condition of the rails based, at least in part, on the sensor signals from the image sensor. ### Claim 24 A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one computer including at least one processor, causes the computer to: generate static image data based, at least in part, on sensor signals generated from an image sensor in communication with the at least one processor, wherein the image sensor is disposed on a rail vehicle and positioned to sense an area in front of the rail vehicle, wherein the static image data comprises a line representation of rails located in the area in front of the rail vehicle; determine track occupancy of the rail vehicle by determining a set of rails occupied by the rail vehicle based, at least in part, on the static image data; determine an upcoming direction of movement of the rail vehicle by determining a direction of the set of rails occupied by the rail vehicle based, at least in part, on the static image data; and validate the determined track occupancy based, at least in part, on the upcoming direction of movement of the rail vehicle, wherein generating the static image data comprises filtering the sensor signals with a low pass image filter to smoothen an instantaneous image of the area in front of the rail vehicle, and wherein generating the static image data comprises generating an enhanced smoothened image to exaggerate pixels of the smoothened image that belong to substantially tubular structures in the area in front of the rail vehicle. ### Claim 25 (depends on 24) The computer program product of claim 24, wherein the program instructions cause the computer to validate the determined track occupancy of the rail vehicle and the determined upcoming direction of movement of the rail vehicle based, at least in part, on time series data, wherein the time series data comprises data from the sensor signals obtained at two or more different times. ### Claim 26 (depends on 24) The computer program product of claim 24, wherein determining the track occupancy of the rail vehicle further comprises determining a number of sets of rails located in the area in front of the rail vehicle. ### Claim 27 (depends on 24) The computer program product of claim 24, wherein determining the upcoming direction of movement of the rail vehicle comprises determining an upcoming angular direction of the set of rails occupied by the rail vehicle.