# US 12,602,631 B2 · Intelligent interactive decision-making method for discrete manufacturing system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,631 B2
- Document ID: US12602631B2
- Status: Granted
- Kind: B2
- Application number: 18/290,808
- Filed: 2023-04-04
- Published: 2025-03-20
- Granted: 2026-04-14
- Earliest claimed: 2022-11-30
- Assignees: Nanjing University of Posts and Telecommunciations
- Applicants: Nanjing University of Posts and Telecommunications
- Inventors: Haigen Yang; Donghuang Lin; Mei Wang; Luyang Li; Cong Wang; Jixin Liu; Fanyu Zeng; Yan Ge
- Web page: https://patentgrep.com/patent/US12602631B2

## CPC

- G06Q 10/06311: {Scheduling, planning or task assignment for a person or group}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/092: Reinforcement learning
- G06Q 10/04: Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" (market predictions or forecasting for commercial activities G06Q30/0202)
- G06Q 10/0631: Resource planning, allocation, distributing or scheduling for enterprises or organisations
- G06Q 50/04: Manufacturing
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Disclosed is an intelligent interactive decision-making method for a discrete manufacturing system. The method includes the following steps: step 1, establishing a production scheduling optimization model and strategy for discrete manufacturing for an actual application scene; step 2, training the scheduling strategy with existing production data on the basis of a deep reinforcement learning algorithm, and storing a state having a high reward in a training process in a memory; step 3, updating the state according to prior knowledge in the memory; step 4, inputting the updated state into a deep reinforcement learning network, obtaining a corresponding reward, and updating the memory according to the reward; and step 5, repeating step 4 until model parameters converge, and saving and putting the model into an actual production scene.

## Claims

### Claim 1

An intelligent interactive decision-making method for a discrete manufacturing system, comprising following steps:
step 1, establishing a production scheduling optimization model and strategy for discrete manufacturing for an actual application scene;
step 2, training the scheduling strategy with existing production data on a basis of a deep reinforcement learning algorithm, and storing a state having a high reward in a training process in a memory;
step 3, updating the state according to a prior knowledge in the memory;
step 4, inputting the updated state into a deep reinforcement learning network, obtaining a corresponding reward, and updating the memory according to the reward; and
step 5, repeating step 4 until parameters of the production scheduling optimization model converge, and saving and putting the production scheduling optimization model into an actual production scene,
wherein in step 3, updating the state according to the prior knowledge in the memory specifically comprises: collecting the production data of production cycles different from step 1 from a discrete manufacturing workshop put into production as a training and updating sample, obtaining a high-reward state sm most similar to st in the memory through a similarity computation, obtaining a weighted sum of the state and st, obtaining a new workshop state
s
t
*
,
and using
s
t
*
as input of a neural network R, wherein a specific formula is as follows:
s
t
*
=
α
⁢
s
t
+
β
⁢
s
m
in the formula,
s
t
*
denotes an updated new state, st denotes a workshop state currently input, sm denotes a high-reward state most similar to st from the memory, and α and β denote weight parameters, wherein a selection formula of sm is as follows:
s
m
=
arg
⁢
max
s
i
⁢
(
s
i
·
s
t

s
i

⁢

s
t

)
,
s
i
∈
D
in the formula, si denotes a workshop state recorded in the memory D,
wherein the high-reward state sm is used as the prior knowledge to generate the new workshop state st* with the high reward, such that an iterative convergence speed of the production scheduling optimization model is improved, and training time of the production scheduling optimization model for discrete manufacturing is reduced.

### Claim 2 (depends on 1)

The intelligent interactive decision-making method for the discrete manufacturing system according to claim 1, wherein in step 1, establishing the production scheduling optimization model and strategy for discrete manufacturing for the actual application scene specifically comprises: transforming a discrete manufacturing production problem into a sequential decision-making problem according to an actual scheduling goal, defining the state, an action, the reward, an exploration and a use strategy according to the goal, and determining a maximum product value in a production cycle as the goal, wherein the reward is a total value of products in the production cycle; the state comprises manufacturing information such as a general type of products to be produced in a workshop, a production batch and a processing stage of each type of products, and a processing time and a processing sequence of each product, machine and device state information such as processing devices allocated for products, a device load rate, and normal or faulty devices, and environmental states such as a temperature and a humidity of the workshop; and the action is to adjust the production scheduling strategy comprising a product processing sequence and allocated processing devices; and then selecting a deep reinforcement learning algorithm framework according to the actual application scene, and establishing the production scheduling optimization model for discrete manufacturing.

### Claim 3 (depends on 1)

The intelligent interactive decision-making method for the discrete manufacturing system according to claim 1, wherein in step 2, training the scheduling strategy with existing production data on the basis of the deep reinforcement learning algorithm, and storing the state having the high reward in the training process in the memory specifically comprises: collecting production data of a certain production cycle from a discrete manufacturing workshop put into production as a pre-training sample, selecting the deep reinforcement learning algorithm, and training the production scheduling optimization model for discrete manufacturing with the collected production data, wherein the production data sampled in a current workshop is input into the production scheduling optimization model, and the production scheduling optimization model outputs a decision-making solution for scheduling optimization of a workshop production line; and
defining the memory D having capacity of M and configured to store a state-reward pair (s,r), wherein an action strategy formula used by a workshop state s to obtain the corresponding reward r is as follows:
a
=
{
arg
⁢
min
a
t
⁢
Q
⁡
(
s
t
,
a
t
)
,
ε
random
,
1
-
ε
in the formula, Q(st,at) denotes an action value function, wherein st represents a current manufacturing workshop state, at represents an action scheduling to be used by a production workshop at a current moment, and in the action strategy, an action is randomly selected for the current workshop state st with a probability of 1−ε, and alternatively, a q value of each action of the current state is evaluated according to the action value function Q with a probability of ε, a current optimal action a is selected, an optimal scheduling strategy a is executed for the current workshop state st, the reward rt and a next state st+1 are obtained, and the memory is updated.

### Claim 4 (depends on 3)

The intelligent interactive decision-making method for the discrete manufacturing system according to claim 3, wherein an updating process of the memory specifically comprises:
(a) if the capacity of the memory D is not full and no si(i=1,2,3, . . . , M) similar to st exists, entering a state-reward pair (st,rt), wherein a similarity computation formula is as follows:
Si
⁢
m
i
⁢
t
=
s
i
·
s
t

s
i

⁢

s
t

,
s
i
∈
D
in the formula, Simit represents similarity between st and si, and if Simit>δ, si is considered to be similar to st, wherein δ denotes a threshold for determining the similarity; and
(b) if si similar to st exists in the memory D, selecting the state having a higher reward for replacement, wherein an updating formula is as follows:
(
s
i
,
r
i
)
=
{
(
s
t
,
r
t
)
,
s
t
~
s
i
⁢
and
⁢
r
t
>
r
i
(
s
i
,
r
i
)
,
s
t
~
s
i
⁢
and
⁢
r
t
≤
r
i
in the formula, ri denotes the reward corresponding to the state si, rt denotes the reward corresponding to the state st, and “˜” represents similar; and
finally, initialization of the memory is completed, M workshop states having a highest reward are stored in the memory, and it is indicated that the manufacturing workshop has a higher reward in the M states according to the collected actual production data, which is closer to a goal of production scheduling.

### Claim 5 (depends on 1)

The intelligent interactive decision-making method for the discrete manufacturing system according to claim 1, wherein in step 4, inputting the updated state into the deep reinforcement learning network, obtaining the corresponding reward, and updating the memory according to the reward specifically comprises: inputting the updated workshop state into the production scheduling optimization model for discrete manufacturing, further optimizing an optimal strategy output by a network R, and updating the memory according to the reward corresponding to the state, wherein a formula for obtaining the reward is as follows:
a
=
{
arg
⁢
min
a
t
⁢
Q
⁡
(
s
t
*
,
a
t
)
,
ε
random
,
1
-
ε
in the formula,
Q
⁡
(
s
t
*
,
a
t
)
denotes an action value function, wherein
s
t
*
represents an updated manufacturing workshop state, at represents a scheduling strategy used by a workshop, and in the strategy, an action is randomly selected for a current state st with a probability of 1−ε, and alternatively, a q value of each action of the current state is evaluated according to the network Q with a probability of ε, a current optimal action a is selected, an optimal action a is executed for the current state
s
t
*
,
the reward
r
t
*
and a next state st+1 are obtained, and the memory is updated.

### Claim 6 (depends on 5)

The intelligent interactive decision-making method for the discrete manufacturing system according to claim 5, wherein an updating process of the memory specifically comprises:
(a) if si similar to
s
t
*
exists in the memory D, using an updating formula as follows:
(
s
i
,
r
i
)
=
{
(
s
t
*
,
r
t
*
)
,
s
t
*
~
s
i
⁢
and
⁢
r
t
*
>
r
i
(
s
i
,
r
i
)
,
s
t
*
~
s
i
⁢
and
⁢
r
t
*
≤
r
i
(b) if no si similar to
s
t
*
exists in the memory D, traversing the memory, selecting (smin, rmin) having a lowest reward to replace the state having a higher reward, wherein an updating formula is as follows:
(
s
min
,
r
min
)
=
{
(
s
t
*
,
r
t
*
)
,
r
t
*
>
r
min
(
s
min
,
r
min
)
,
r
t
*
≤
r
min
in the formula, si denotes the state recorded in the memory D.

### Claim 7 (depends on 1)

The intelligent interactive decision-making method for the discrete manufacturing system according to claim 1, wherein in step 5, repeating step 4 until the parameters of the production scheduling optimization model converge, and saving and putting the production scheduling optimization model into the actual production scene specifically comprises: returning to step 3, repeating step 3 and step 4, continuously optimizing the memory D, quickly updating the production scheduling optimization model through interaction between the memory D and the production scheduling optimization model until the parameters of the production scheduling optimization model converge, which indicates that the production scheduling optimization model reaches an optimal decision-making model, putting the production scheduling optimization model into the manufacturing workshop, and intelligently deciding production scheduling of the workshop according to the production scheduling optimization model.