# US 11,078,631 B2 · System and method for inspecting a rail using machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,078,631 B2
- Document ID: US11078631B2
- Status: Granted
- Kind: B2
- Application number: 15/826,780
- Filed: 2017-11-30
- Published: 2019-05-30
- Granted: 2021-08-03
- Earliest claimed: 2017-11-30
- Assignees: Sperry Rail Holdings, Inc.
- Applicants: Sperry Rail Holdings, Inc.
- Inventors: David Henry Gilbert
- Web page: https://patentgrep.com/patent/US11078631B2

## CPC

- B61D 15/08: Railway inspection trolleys
- E01B 35/06: for measuring irregularities in longitudinal direction
- B61K 9/10: for detecting cracks in rails or welds thereof
- B61L 23/044: {Broken rails}
- B61L 23/045: {Rail wear}
- G01N 27/025: {a current being generated within the material by induction}
- G01N 27/9046: {by analysing electrical signals}
- G01N 29/043: {in the interior, e.g. by shear waves}
- G01N 29/225: {Supports, positioning or alignment in moving situation}
- G01N 29/265: by moving the sensor relative to a stationary material
- G01N 29/4481: {Neural networks}
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G01N 27/9093: Arrangements for supporting the sensor; Combinations of eddy-current sensors and auxiliary arrangements for marking or for rejecting
- G01N 2291/0289: Internal structure, e.g. defects, grain size, texture
- G01N 2291/2623: Rails; Railroads
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- B61D: BODY DETAILS OR KINDS OF RAILWAY VEHICLES (vehicles in general B60; adaptation of vehicles to special systems B61B; underframes B61F)
- E01B: PERMANENT WAY; PERMANENT-WAY TOOLS; MACHINES FOR MAKING RAILWAYS OF ALL KINDS (derailing or rerailing blocks on track, track brakes or retarders B61K)
- B61K: AUXILIARY EQUIPMENT SPECIALLY ADAPTED FOR RAILWAYS, NOT OTHERWISE PROVIDED FOR
- B61L: GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC (brakes or auxiliary equipment B61H, B61K; point or crossing construction E01B)
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An aspect includes a vehicle that includes rail inspection sensors configured for capturing transducer data describing the rail, and a processor configured for receiving and processing the transducer data in near-real time to determine whether the captured transducer data identifies a suspected rail flaw. The processing includes inputting the captured transducer data to a machine learning system that has been trained to identify patterns in transducer data that indicate rail flaws. The processing also includes receiving an output from the machine learning system, the output indicating whether the captured transducer data identifies a suspected rail flaw. An alert is transmitted to an operator of the vehicle based at least in part on the output indicating that the captured transducer data identifies a suspected rail flaw. The alert includes a location of the suspected rail flaw and instructs the operator to stop the vehicle and to perform a repair action.

## Claims

### Claim 1

A vehicle for inspecting a rail, the vehicle comprising:
rail inspection sensors configured for capturing transducer data describing the rail; and
a processor configured for:
receiving the transducer data; and
processing the captured transducer data to determine whether the captured transducer data identifies a suspected rail flaw, the processing comprising:
inputting the captured transducer data as a stream of overlapping sequences in a form suitable for a machine learning system that has been trained to identify patterns in the transducer data that indicate rail flaws, the machine learning system comprising a deep learning convolutional neural network to perform image pattern recognition that is combined with a long short-term memory recurrent neural network to identify rail flaws of the rail of indeterminate length; and
receiving an output from the machine learning system, the output indicating whether the captured transducer data identifies the suspected rail flaw;
auditing at least one of the deep learning convolutional neural network or the long short-term memory recurrent neural network to identify at least one of a false positive or a false negative, the auditing comparing the output from the machine learning system to output from a human analyst;
adjusting the at least one of the deep learning convolutional neural network on the long short-term memory recurrent neural network based on the at least one of the false positive or the false negative; and
transmitting an alert to an operator of the vehicle based at least in part on the output indicating that the captured transducer data identifies a suspected rail flaw, the alert including a location of the suspected rail flaw and instructing the operator to stop the vehicle and to perform a repair action,
wherein the stream of overlapping sequences comprises a sequence of overlapping 2-dimensional images.

### Claim 2 (depends on 1)

The vehicle of claim 1, further comprising a camera for capturing a camera image of the location, wherein the alert includes the camera image.

### Claim 3 (depends on 1)

The vehicle of claim 1, wherein the output is transmitted, via a network, to a remote storage device or a remote processor.

### Claim 4 (depends on 1)

The vehicle of claim 1, wherein the rail inspection sensors include one or more of an ultrasonic transducer, an induction transducer, and an eddy current transducer.

### Claim 5 (depends on 1)

The vehicle of claim 1, wherein the processing further comprises:
generating a flaw suspect list table comprising:
a test segment indicator that indicates a data acquisition recording run on the rail of the suspected rail flaw;
a rail identifier that identifies whether the suspected rail flaw is in a left rail or a right rail;
a rail pulse count indicator that indicates a distance along the rail from a start of a recording;
a code identifier that identifies a code corresponding to a type of the suspected flaw; and
a confidence identifier that indicates a level of confidence of the machine learning system.

### Claim 6 (depends on 1)

The vehicle of claim 1, wherein at least one of the deep learning convolutional neural network or the long short-term memory recurrent neural network is trained by comparing a predicted label associated with a training image to a known label associated with the training image, and wherein the processing is performed in a cloud computing environment.

### Claim 7 (depends on 1)

The vehicle of claim 1, wherein the overlapping 2-dimensional images overlap by a predetermined number of pixels.

### Claim 8 (depends on 7)

The vehicle of claim 7, wherein the predetermined number of pixels is greater than half the width of the overlapping 2-dimensional images.

### Claim 9 (depends on 1)

The vehicle of claim 1, wherein the overlapping 2-dimensional images overlap in all but four-pixel increments.

### Claim 10

A method for inspecting a rail, the method comprising:
receiving transducer data from rail inspection sensors mounted on a vehicle that is located on the rail; and
processing the transducer data to identify locations of suspected rail flaws in the rail, the processing comprising:
inputting the transducer data as a stream of overlapping sequences in a form suitable for a machine learning system that has been trained to identify patterns in the transducer data that indicate rail flaws, the machine learning system comprising a deep learning convolutional neural network to perform image pattern recognition that is combined with a long short-term memory recurrent neural network to identify rail flaws of the rail of indeterminate length;
receiving an output from the machine learning system, the output including a list of suspected rail flaws and their corresponding locations on the rail;
auditing at least one of the deep learning convolutional neural network or the long short-term memory recurrent neural network to identify at least one of a false positive or a false negative, the auditing comparing the output from the machine learning system to output from a human analyst;
adjusting the at least one of the deep learning convolutional neural network on the long short-term memory recurrent neural network based on the at least one of the false positive or the false negative; and
initiating a repair action based on the list of suspected rail flaws.

### Claim 11 (depends on 10)

The method of claim 10, wherein the transducer data includes images.

### Claim 12 (depends on 11)

The method of claim 11, wherein the inputting the transducer data includes streaming overlapped images.

### Claim 13 (depends on 10)

The method of claim 10, wherein the transducers include ultrasonic transducers.

### Claim 14 (depends on 10)

The method of claim 10, wherein the transducers include induction transducers.

### Claim 15 (depends on 10)

The method of claim 10, wherein the transducers include eddy current transducers.

### Claim 16 (depends on 10)

The method of claim 10, wherein the receiving is from a requestor via a network and the method further comprises sending the list of suspected rail flaws to the requestor.

### Claim 17 (depends on 10)

The method of claim 10, wherein the processing and the initiating are performed by a processor located on the vehicle.

### Claim 18 (depends on 10)

The method of claim 10, further comprising, performing the repair action, the repair action comprising performing a verification of one of the suspected rail flaws using a hand held sensor.

### Claim 19

A system for inspecting a rail, the system comprising:
a non-transitory memory having computer readable instructions; and
one or more processors for executing the computer readable instructions, the computer readable instructions controlling the one or more processors to perform operations comprising:
receiving transducer data from rail inspection sensors mounted on a vehicle that is located on the rail;
processing the transducer data to identify locations of suspected rail flaws in the rail, the processing comprising:
inputting the transducer data as a stream of overlapping sequences in a form suitable for a machine learning system that has been trained to identify patterns in the transducer data that indicate rail flaws, the machine learning system comprising a deep learning convolutional neural network to perform image pattern recognition that is combined with a long short-term memory recurrent neural network to identify rail flaws of the rail of indeterminate length; and
receiving an output from the machine learning system, the output including a list of suspected rail flaws and their corresponding locations on the rail;
auditing at least one of the deep learning convolutional neural network or the long short-term memory recurrent neural network to identify at least one of a false positive or a false negative, the auditing comparing the output from the machine learning system to output from a human analyst;
adjusting the at least one of the deep learning convolutional neural network on the long short-term memory recurrent neural network based on the at least one of the false positive or the false negative; and
initiating a repair action based on the list of suspected rail flaws.

### Claim 20

A computer program product for inspecting a rail, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform operations comprising:
receiving transducer data from rail inspection sensors mounted on a vehicle that is located on the rail; and
processing the transducer data to identify locations of suspected rail flaws in the rail, the processing comprising:
inputting the transducer data as a stream of overlapping sequences in a form suitable for a machine learning system, that has been trained to identify patterns in the transducer data that indicate rail flaws, the machine learning system comprising a deep learning convolutional neural network to perform image pattern recognition that is combined with a long short-term memory recurrent neural network to identify rail flaws of the rail of indeterminate length; and
receiving an output from the machine learning system, the output including a list of suspected rail flaws and their corresponding locations on the rail;
auditing at least one of the deep learning convolutional neural network or the long short-term memory recurrent neural network to identify at least one of a false positive or a false negative, the auditing comparing the output from the machine learning system to output from a human analyst;
adjusting the at least one of the deep learning convolutional neural network on the long short-term memory recurrent neural network based on the at least one of the false positive or the false negative; and
initiating a repair action based on the list of suspected rail flaws.