# US 2021/0279488 A1 · Method and system for railway obstacle detection based on rail segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279488 A1
- Document ID: US20210279488A1
- Status: Published application
- Kind: A1
- Application number: 17/258,446
- Filed: 2019-07-10
- Published: 2021-09-09
- Earliest claimed: 2018-07-10
- Assignees: Rail Vision Ltd
- Applicants: Rail Vision Ltd
- Inventors: Sagi DOLBERG; Natalie DAY; Michael LEVY; Avishay ALFANDARY; Avraham Ram GUISSIN; Sharon BASON; Zohar Zeev SHECHTER; Shahar HANIA; Yotam AZRIEL
- Web page: https://patentgrep.com/patent/US20210279488A1

## CPC

- G06N 3/08: Learning methods
- B61L 23/041: {Obstacle detection}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 7/11: Region-based segmentation
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06T 2207/10048: Infrared image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- B61L: GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC (brakes or auxiliary equipment B61H, B61K; point or crossing construction E01B)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for rails and obstacles detection based on forward-looking electrooptical imaging, novel system architecture and novel scene analysis and image processing are disclosed. The processing solution utilizes a deep learning semantic scene segmentation approach based on a rails and switches states detection neural network that determines the railway path of the train in the forward-looking imagery, and an objects and obstacles detection and tracking neural network that analyzes the vicinity of the determined railway path and detects impending obstacles.

## Claims

### Claim 1

A method of rails and obstacles detection, the method comprising:
receiving a plurality of image frames from at least one imaging sensor installed on a locomotive of a train, wherein the at least one imaging sensor facing a direction of travel of the train;
determining, for each image frame of a subset of the plurality of image frames, an input data structure comprising the respective image frame and a history element, wherein the history element comprises a segmentation mask prediction determined for an image frame preceding to the respective image frame; and
determining, based on the input data structure of each of the image frames of the subset and using a rails and switches states detection neural network (RSSD NN), a segmentation mask prediction of the respective image frame, wherein the segmentation mask prediction comprises at least a current railway path along which the train travels.

### Claim 2 (depends on 1)

The method of claim 1, wherein the segmentation mask prediction further comprises at least one turnout along the current railway path representing a state of corresponding at least one switch along the current railway path.

### Claim 3 (depends on 1)

The method of claim 1, further comprising training the RSSD NN prior to inferring thereof, the training is based on a training subset of training image frames and comprises a predetermined number of training cycles, wherein each of the training cycles comprises:
determining, for each training image frame of the training subset, a training input data structure comprising the respective training image frame and a training history element, wherein the training history element comprises a training segmentation mask prediction determined for a training image frame preceding to the respective training image frame;
determining, based on the training input data structure of each of the training image frames of the training subset and using the RSSD NN, a training segmentation mask prediction of the respective training image frame;
comparing the training segmentation mask prediction determined for each of the training image frames of the training subset with at least one predetermined annotated mask of the training image frame; and
updating the training segmentation mask prediction determined for each of the training image frame of the training subset based on the comparison thereof.

### Claim 4 (depends on 3)

The method of claim 3, further comprising augmenting the training segmentation mask prediction determined during at least one training cycle of the predetermined number of cycles, wherein augmenting of the training segmentation mask prediction comprises at least one of:
zeroing the entire training segmentation mask prediction;
zeroing a predetermined portion of the training segmentation mask prediction; and
replacing the predetermined portion of the training segmentation mask prediction with a specified label that retains a shape and location of a current railway path and ignores a distinction between the current railway path and at least one turnout along the current railway path.

### Claim 5 (depends on 1)

The method of claim 1, further comprising determining, in at least one image frame of the subset, based on the segmentation mask prediction determined for the respective frame and using an objects and obstacles detection and tracking neural network (OODT NN), at least one object/obstacle on or in vicinity of the current railway path.

### Claim 6

A system for rails and obstacles detection, the system comprising:
at least one imaging sensor installed one a locomotive of a train to face a direction of travel of the train, to obtain a plurality of image frames;
a rails and switches states detection (RSSD) module comprising a RSSD neural network (RSSD NN), the RSSD module is configured to:
determine, for each image frame of a subset of the plurality of image frames, an input data structure comprising the respective image frame and a history element, wherein the history element comprises a segmentation mask prediction determined for an image frame preceding to the respective image frame; and
determine, based on the input data structure of each of the image frames of the subset and using the RSSD NN, a segmentation mask prediction of the respective image frame, wherein the segmentation mask prediction comprises at least a current railway path along which the train travels.

### Claim 7 (depends on 6)

The system of claim 6, wherein the segmentation mask prediction further comprises at least one turnout along the current railway path representing a state of corresponding at least one switch along the current railway path.

### Claim 8 (depends on 6)

The system of claim 6, further comprising an objects and obstacles detection and tracking module comprising an OODT neural network (OODT), wherein the OODT module is configured to determine, in at least one image frame of the subset, based on the segmentation mask prediction determined for the respective frame and using the OODT NN, at least one object/obstacle on or in vicinity of the current railway path.

### Claim 9 (depends on 6)

The system of claim 6, wherein the at least one sensor is at least one of: a visual band imaging sensor and a thermal infrared sensor.

### Claim 10 (depends on 6)

The system of claim 6, wherein the at least one sensor is at least one of: a narrow field-of-view imaging sensor and a wide field-of-view imaging sensor.
