# US 2026/0244809 A1 · Reinforced generation method for hard-to-reach scenarios of radio block center (rbc) of train autonomous circulation and navigation (tacn) and generation system thereof, and rbc test system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0244809 A1
- Document ID: US20260244809A1
- Status: Published application
- Kind: A1
- Application number: 19/642,742
- Filed: 2026-04-09
- Published: 2026-08-20
- Earliest claimed: 2024-09-03
- Assignees: Casco Signal Ltd.
- Applicants: Casco Signal Ltd.
- Inventors: Ming XIA; Jiao Liu; Xiaowei Hou; Jiao Chen; Hao Lai; Lei Feng; Tianhe Wang; Fangyu Zeng
- Web page: https://patentgrep.com/patent/US20260244809A1

## CPC

- G06F 30/15: Vehicle, aircraft or watercraft design
- B61L 15/0058: {On-board optimisation of vehicle or vehicle train operation}
- B61L 15/0081: {On-board diagnosis or maintenance}
- B61L 25/025: {Absolute localisation, e.g. providing geodetic coordinates}
- B61L 27/04: Automatic systems, e.g. controlled by train; Change-over to manual control
- B61L 27/16: Trackside optimisation of vehicle or train operation
- B61L 27/20: Trackside control of safe travel of vehicle or train, e.g. braking curve calculation
- B61L 27/50: Trackside diagnosis or maintenance, e.g. software upgrades
- B61L 27/60: Testing or simulation
- B61L 27/70: Details of trackside communication
- G05D 1/646: Following a predefined trajectory, e.g. a line marked on the floor or a flight path
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/048: Activation functions
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G05D 2101/15: using machine learning, e.g. neural networks
- G05D 2109/10: Land vehicles
- Y02T 10/40: Engine management systems
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- B61L: GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC (brakes or auxiliary equipment B61H, B61K; point or crossing construction E01B)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

A reinforced generation method for hard-to-reach scenarios of a radio block center (RBC) of train autonomous circulation and navigation (TACN) and a generation system thereof, and an RBC test system are provided. This method models a function of a train operation dynamic system equation as a discrete event dynamic system to describe status transitions of a test environment system. Moreover, this method combines a Q neural network and a target neural network to implement a double deep Q network learning algorithm, thereby constructing a hard-to-reach scenario generator. This method obtains the hard-to-reach scenario generator based on the double deep Q network learning algorithm, which can effectively solve the problem that existing RBC test tools are unable to discover hard-to-reach scenarios, and help improve the test quality and efficiency in testing the capability of the RBC system to cope with the hard-to-reach scenarios.

## Claims

### Claim 1

A reinforced generation method for hard-to-reach scenarios of a radio block center (RBC) of train autonomous circulation and navigation (TACN), comprising:
S1, constructing a discrete event dynamic system according to environmental status variables of train operation;
S2, constructing a Q neural network and a target neural network by a reinforcement learning algorithm;
S3, randomly selecting a group of environmental status variable data as a current environmental status variable St corresponding to a current time t;
S4, acquiring action information ac corresponding to the current environmental status variable St by the Q neural network;
S5, obtaining a new environmental status variable St+1 corresponding to a next time point t+1 based on the action information ac and the discrete event dynamic system, and calculating a reward value rt of the Q neural network based on a reward function R when the action information ac is implemented in an environment, wherein during the calculation based on the reward function R, the higher a degree of matching between the environmental status variables and hard-to-reach scenario conditions, the higher the reward value;
S6, inputting the new environmental status variable St+1 into the target neural network to calculate a maximum Q value of the target neural network, and acquiring a reward value yt of the target neural network in an St+1 status based on the reward value rt of the Q neural network;
S7, calculating a loss function according to the reward value rt of the Q neural network, a Q value of the Q neural network, and the maximum Q value of the target neural network, calculating a Q value deviation Δθ between the Q neural network and the target neural network, and updating a network parameter θ of the Q neural network based on the loss function and the Q value deviation Δθ;
S8, determining whether a first preset condition for reaching a terminal status is satisfied; if the first preset condition is not satisfied, determining a second preset condition for updating a target neural network parameter θ− is satisfied; if the second preset condition is satisfied, updating the target neural network parameter θ− and then proceeding to S3; and if the second preset condition is not satisfied, proceeding to S3; and
S9, if the first preset condition is satisfied, terminating training to obtain a hard-to-reach scenario generator of the RBC, wherein the first preset condition and the second preset condition are dynamically adjusted to dynamically adjust a training period.

### Claim 2 (depends on 1)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 1, wherein the discrete event dynamic system is represented as:
y
⁡
(
t
)
=
f
⁡
(
x
⁡
(
t
)
,
u
⁡
(
t
)
)
(
1
)
x
⁡
(
t
+
1
)
=
g
⁡
(
x
⁡
(
t
)
,
u
⁡
(
t
)
)
,
(
2
)
wherein x(t) is the current environmental status variable St corresponding to the current time t, u(t) is a system input quantity and the action information ac, y(t) is a system output quantity, f( ) is a processing procedure function, and g( ) is a status transition operation process function;
a status equation corresponding to the discrete event dynamic system is shown in Equation (3) and Equation (4):
x
⁡
(
t
+
1
)
=
A
[
sec
route
traincom
⁢
trainpos
trainMA
]
+
B
[
u
⁡
(
t
)
]
(
3
)
y
⁡
(
t
)
=
C
[
x
⁡
(
t
)
,
u
⁡
(
t
)
]
,
(
4
)
wherein x(t+1) is the new environmental status variable St+1 corresponding to the next time point t+1; A is a status matrix of the discrete event dynamic system, configured to reflect a coupling relationship among status variables in the discrete event dynamic system; B is an input matrix, configured to reflect how the system input quantity affects the status variables; C is an output matrix, indicating a conversion relationship from the status variables to outputs; sec represents a section status, route represents a route status, traincom represents a train communication status, trainpos represents a train position status, and trainMA represents a train authority status.

### Claim 3 (depends on 1)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 1, wherein the environmental status variables comprise:
at least one of a section status, a route status, a train communication status, a train position status, and a train authority status.

### Claim 4 (depends on 3)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 3, wherein
the section status comprises an idle status, an occupied status, a cleared status, a train envelope existing status, and a hidden train fault occupied status;
the route status comprise normal, shunting, inactive, in use, unavailable, and presence of other trains;
the train communication status comprises normal communication, disconnection, registration, deregistration, and delay;
the train position status comprises a section, a pre-route position, an approach section, an in-route position, a last section of a route, a post-route position, and preceding train information; and
the train authority status comprises the route.

### Claim 5 (depends on 1)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 1, wherein
in S4, a double deep Q network generates a random number; when the random number is less than a preset value, a route allocation action is randomly selected; and when the random number is greater than or equal to the preset value, the current Q neural network is selected to output the optimal action information ac.

### Claim 6 (depends on 5)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 5, wherein the randomly selecting the route allocation action comprises:
randomly selecting action information act with a probability ε in the Q neural network, exploring the environment with the probability ε, and selecting a current optimal status with a probability (1−ε), i.e.,
ac
t
=
arg
max
ac
Q
⁢
(
S
t
,
ac
;
θ
)
,
wherein ε is a greedy probability; wherein a larger value is used in an early training stage ε to efficiently search for a train operation environment, and a value of ε is gradually reduced as training progress and status continuously transition.

### Claim 7 (depends on 1)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 1, wherein an expression of the reward function R is shown in Equation (5):
R
=
α
1
⁢
r
1
-
α
2
⁢
r
2
+
α
3
⁢
r
3
,
(
5
)
wherein r1, r2 and r3 are a discovery reward, an exploration failure reward, and an irrational allocation reward, respectively, and α1, α2 and α3 are weight coefficients of the rewards;
the discovery reward is configured to improve an exploration capability of the hard-to-reach scenario generator for the hard-to-reach scenarios, and an expression of the discovery reward is shown in Equation (6):
r
1
=
{
a
,
route
⁢
allocation
⁢
error
0
,
others
;
(
6
)
the exploration failure reward is configured to indicate whether a hard-to-reach scenario occurs within a predetermined period, and an expression of the exploration failure reward is shown in Equation (7):
r
2
=
{
p
,
exploration
⁢
failure
0
,
exploration
⁢
success
;
(
7
)
the irrational allocation reward is configured to indicate whether an irrational allocation occurs within a predetermined period, and an expression of the irrational allocation reward is shown in Equation (8):
r
3
=
{
-
q
,
irrational
⁢
allocation
q
,
rational
⁢
allocation
.
(
8
)

### Claim 8 (depends on 1)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 1, wherein
the loss function=the maximum Q value of the target neural network in the St+1 status+the reward value rt of the Q neural network in an St status−the Q value of the Q neural network in the St status.

### Claim 9 (depends on 1)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 1, wherein the Q value deviation Δθ is as follows:
Δ
⁢
θ
=
α
[
y
c
-
Q
⁡
(
S
t
,
ac
;
θ
)
]
⁢
∇
Q
⁡
(
S
,
AC
;
θ
)
,
wherein α is a deep learning rate, S,AC represent a discrete event dynamic system status set and a set of selected action information, respectively, and yt is the reward value of the target neural network in the St+1 status.

### Claim 10 (depends on 9)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 9, wherein the reward value yt of the target neural network in the St+1 status is as follows:
if a current status St is the terminal status, yt=rt, otherwise, yt=the maximum Q value of the target neural network in the St+1 status+rt, i.e.
y
t
=
r
t
+
γ
⁢
Q
⁢
(
S
t
+
1
,
arg
⁢
max
ac
⁢
Q
⁢
(
S
t
,
ac
;
θ
)
;
θ
-
)
,
wherein γ is a discount factor.

### Claim 11 (depends on 1)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 1, wherein
if the St+1 status is not the terminal status, a quadruple {St, ac, rt, St+1} is stored in an experience replay buffer, and the Q neural network selects a plurality of quadruples from the experience replay buffer for training; and the target neural network selects a plurality of quadruples from the experience replay buffer for training.

### Claim 12 (depends on 1)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 1, wherein the updating the target neural network parameter θ− comprises:
copying the network parameter θ of the Q neural network to the target neural network to enable θ−=θ.

### Claim 13 (depends on 1)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 1, wherein
the Q neural network is a fully connected network, wherein a ReLU activation function is adopted among fully connected layers thereof; and the Q neural network provides a bounded output by a tanh activation function.

### Claim 14 (depends on 1)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 1, wherein
the target neural network and the Q neural network are the same in structure.

### Claim 15 (depends on 1)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 1, wherein the first preset condition comprises:
a number of training iterations reaching a first preset number;
and/or a duration of the training cycle reaching a first preset duration;
and/or a train entering an incorrect route.

### Claim 16 (depends on 1)

The reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 1, wherein the second preset condition comprises:
a number of training iterations reaching a second preset number.

### Claim 17 (depends on 1)

A reinforced generation system for hard-to-reach scenarios of a radio block center (RBC) of train autonomous circulation and navigation (TACN), comprising:
the hard-to-reach scenario generator generated by adopting the reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 1; and
the discrete event dynamic system, in communication connection to the hard-to-reach scenario generator.

### Claim 18 (depends on 1)

A radio block center (RBC) test system, comprising:
using the hard-to-reach scenario generator generated by adopting the reinforced generation method for the hard-to-reach scenarios of the RBC of TACN according to claim 1.
