# US 2023/0401390 A1 · Automatic concrete dam defect image description generation method based on graph attention network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0401390 A1
- Document ID: US20230401390A1
- Status: Published application
- Kind: A1
- Application number: 18/327,074
- Filed: 2023-06-01
- Published: 2023-12-14
- Earliest claimed: 2022-06-13
- Assignees: Huaneng Lancang River Hydropower Inc.; Hohai University; Huaneng Group R&D Center Co., Ltd.
- Applicants: Huaneng Lancang River Hydropower Inc.; Hohai University; Huaneng Group R&D Center Co., Ltd.
- Inventors: Hua Zhou; Fudong Chi; Yingchi Mao; Hao Chen; Xu Wan; Huan Zhao; Bohui Pang; Jiyuan Yu; Rui Guo; Guangyao Wu; Shunbo Wang
- Web page: https://patentgrep.com/patent/US20230401390A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06T 7/0004: {Industrial image inspection}
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/42: Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06T 2207/20072: Graph-based image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30132: Masonry; Concrete
- G06T 2207/30184: Infrastructure
- Y02A 10/40: Controlling or monitoring, e.g. of flood or hurricane; Forecasting, e.g. risk assessment or mapping
- Y04S 10/50: Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- Y02A: TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y04S: SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS

## Abstract

An automatic concrete dam defect image description generation method based on graph attention network, including: 1) extract the local grid features and whole image features of the defect image and conduct image coding by using multi-layer convolutional neural network; 2) construct the grid feature interaction graph, and fuse and encode the grid visual features and global image features of the defect image; 3) update and optimize the global and local features through the graph attention network, and fully utilize the improved visual features for defect description. The invention constructs the grid feature interaction graph, updates the node information by using the graph attention network, and realizes the feature extraction task as the graph node classification task. The invention can capture the global image information of the defect image and the potential interaction of local grid features, and the generated description text can accurately and coherently describe the defect information.

## Claims

### Claim 1

An automatic concrete dam defect image description generation method based on graph attention network, characterized by including the following steps:
1) Extract the global features and grid features of the defect image respectively by using the multi-layer convolutional neural network;
2) Construct the grid feature interaction graph, and input the global features and grid features as nodes;
3) Update and optimize the node information in the grid feature interaction graph constructed in Step 2) by using the graph attention network to get the updated global features and grid features;
4) Automatically generate the image description by using the updated global feature and grid feature sequence based on Transformer decoding module.

### Claim 2 (depends on 1)

The automatic concrete dam defect image description generation method based on graph attention network according to claim 1, characterized in that in Step 1), Faster R-CNN model pre-trained on Visual Genome data set is used to extract the global features and grid features, and uses the convolutional layer C5 with the stride of 1 and 1×1 RoIPool with two FC layers as detection heads, and the output of the layer C5 is used as the grid feature of the defect image.

### Claim 3 (depends on 1)

The automatic concrete dam defect image description generation method based on graph attention network according to claim 1, characterized in that in Step 2), the dependency among grid features and the global features are introduced, the grid feature interaction graph is established by means of the global node mechanism, and the grid feature interaction graph is constructed as follows: the global features and grid features obtained in Step 1) are used as the node inputs of the grid feature interaction graph to obtain one global node and multiple local nodes; the global node serves as the virtual center and is connected to all nodes in the graph; the local nodes are connected according to the relative center coordinates of the grid, namely, the value (i,j) of two adjacent grid nodes i and j in the adjacency matrix A is assigned as 1, indicating direct interaction, while the value of non-adjacent nodes is assigned as 0, indicating no interaction; the global nodes collect and distribute general information from local nodes.

### Claim 4 (depends on 1)

The automatic concrete dam defect image description generation method based on graph attention network according to claim 1, characterized in that in Step 1), the grid features and global features of the defect image are extracted for image coding by using the multi-layer convolutional neural network: add the global features of the feature graph on the basis of fusing the grid features, and extract the global features and grid features of the defect image; define the input as a defect image p0=full_image and n fixed size Grids=(p1, p2, . . . , pn), extract the defect features by using Faster R-CNN model pre-trained on the Visual Genome data set, and use the convolutional layer C5 with the stride of 1 and 1×1 RoIPool with two FC layers as detection heads, in which the output of the layer C5 is embedded in IE0:n=CNN(p0:n;ΘCNN) as the extracted defect image; ΘCNN represents the parameter of CNN model, and IE includes global image embedding IEGlobal=IE0 and local image embedding IELocal=[IE1, IE2, . . . , IEn].

### Claim 5 (depends on 1)

The automatic concrete dam defect image description generation method based on graph attention network according to claim 1, characterized in that in Step 3), the nodes of the graph attention network based on grid feature interaction graph correspond to the grid of the defect image according to the grid feature interaction graph and the graph attention network, the feature of the nodes is local image embedding, the edges of the graph correspond to the edges of the grid feature interaction graph, and the multi-head self-attention mechanism is used to fuse and update the defect information of adjacent nodes in the grid feature interaction graph.

### Claim 6 (depends on 1)

The automatic concrete dam defect image description generation method based on graph attention network according to claim 1, characterized in that in Step 4), Transformer decoding module comprises a reference decoding module and an optimized decoding module, the training of the reference decoding module and the optimized decoding module is divided into two stages: cross-entropy loss optimization stage and reinforcement learning stage, in which the cross-entropy loss optimization stage is based on the loss function of the negative log-likelihood estimation, and the reinforcement learning stage is based on the reinforcement learning optimization strategy and takes CIDEr score as a reward function.

### Claim 7 (depends on 1)

The automatic concrete dam defect image description generation method based on graph attention network according to claim 1, characterized in that in Step 3), the steps for updating the nodes by using the graph attention network are as follows:
(3.1) The grid features obtained by defining the multi-layer convolutional neural network is expressed as h=(h1, h2, . . . , hn), hi∈F, where n represents the number of grids, and F is the feature dimension outputted by the CNN hidden layer;
(3.2) The feature vectors of the two grids are connected according to the adjacency matrix A, and self-attention calculation is conducted for each grid through the nonlinear layer of Leakey ReLU function, as shown in Equation (1):
eij=Leakey ReLU(VT[Whi⊕Whj]) (1)
Where eij represents the importance of the features of the grid j to the grid i, V and W are learnable parameter matrices, and ⊕ represents connection;
(3.3) The softmax function is used to normalize all neighborhood grid features of the grid i to obtain the attention coefficient αij so that makes it easy to compare the coefficients between different nodes, as shown in Equation (2):
α
ij
=
soft
⁢
max
j
(
e
ij
)
=
exp
⁡
(
e
ij
)
∑
k
∈
N
i
⁢
exp
⁡
(
e
ik
)
(
2
)
Where Ni represents all the first-order neighborhood node sets of the grid i in the grid feature interaction graph, including i itself;
(3.4) The features of all adjacent grids j to the grid i are weighted and summed with the corresponding weight coefficient αij, and the final output features of each object node are obtained by the nonlinear layer σ, as shown in Equation (3):
h
i
′
=
σ
⁡
(
∑
j
∈
N
i
α
ij
⁢
W
⁢
h
i
)
(
3
)
(3.5) The method constituted by Steps 3.1-3.4 is extended to the multi-head self-attention mechanism.

### Claim 8 (depends on 1)

The automatic concrete dam defect image description generation method based on graph attention network according to claim 1, characterized in that in Step 4), the decoding module based on Transformer automatically generates the image description by using the updated global and grid feature sequences and comprises a reference decoding module and an optimized decoding module, the training of the reference decoding module and the optimized decoding module is divided into two stages: cross-entropy loss optimization stage and reinforcement learning stage, and the specific steps are as follows:
(4.1) In the cross-entropy loss optimization stage, the goal is to make the prediction description consistent with the annotation description; given a decoding module with manual annotation w*1:T and parameter θ, the following loss function is minimized:
L
XE
(
θ
)
=
-
∑
t
=
1
T
log
⁡
(
p
θ
(
w
t
*
|
w
1
:
t
-
1
*
)
)
(
4
)
Where T is the length of the generated descriptive statement, and pθ(w*t|w*1:t-1) represents the probability of decoding module to predict the vector w*t of the tth word based on the annotation sequence w*1:t-1 with the length of t−1;
(4.2) In the actual training, Transformer model is weighted and updated on a batch basis, as shown in Equation (5):
loss
=
-
1
N
⁢
∑
n
=
1
N
L
XE
(
θ
)
(
5
)
Where N is the size of the batch;
(4.3) In the reinforcement learning stage, based on the loss function training of the negative log-likelihood estimation in Steps 4.1-4.2, the self-critical sequence training strategy is adopted to further optimize the Transformer model with CIDEr score as a reward, as shown in Equation (6):
LRL(θ)=−Ew1:Tpθ[r(w1:T)] (6)
Where r(⋅) is the original reward defined based on CIDEr score, and w1:T represents the generated image description sequence with the length of T;
(4.4) The optimization goal in the reinforcement learning stage is to minimize negative expected rewards, and the gradient of average value using the rewards is updated and defined as follows:
b
=
1
k
⁢
(
∑
i
k
r
⁡
(
w
i
:
T
i
)
)
(
7
)
∇
θ
L
RL
(
θ
)
≈
-
1
k
⁢
∑
i
=
1
k
(
(
r
⁡
(
w
1
:
T
i
)
-
b
)
⁢
∇
θ
log
⁢
p
θ
(
w
1
:
T
i
)
)
(
8
)
Where k is the number of image description sequences, w1:Ti is the ith image description sequence, and b is the average value of rewards for all image description sequences.

### Claim 9 (depends on 1)

A computer device, characterized in that the computer device comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor realizes the automatic concrete dam defect image description generation method based on the graph attention network in claim 1 when executing the computer program.

### Claim 10 (depends on 2)

A computer device, characterized in that the computer device comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor realizes the automatic concrete dam defect image description generation method based on the graph attention network in claim 2 when executing the computer program.

### Claim 11 (depends on 3)

A computer device, characterized in that the computer device comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor realizes the automatic concrete dam defect image description generation method based on the graph attention network in claim 3 when executing the computer program.

### Claim 12 (depends on 4)

A computer device, characterized in that the computer device comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor realizes the automatic concrete dam defect image description generation method based on the graph attention network in claim 4 when executing the computer program.

### Claim 13 (depends on 5)

A computer device, characterized in that the computer device comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor realizes the automatic concrete dam defect image description generation method based on the graph attention network in claim 5 when executing the computer program.

### Claim 14 (depends on 6)

A computer device, characterized in that the computer device comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor realizes the automatic concrete dam defect image description generation method based on the graph attention network in claim 6 when executing the computer program.

### Claim 15 (depends on 7)

A computer device, characterized in that the computer device comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor realizes the automatic concrete dam defect image description generation method based on the graph attention network in claim 7 when executing the computer program.

### Claim 16 (depends on 8)

A computer device, characterized in that the computer device comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor realizes the automatic concrete dam defect image description generation method based on the graph attention network in claim 8 when executing the computer program.

### Claim 17 (depends on 1)

A computer readable memory medium, characterized in that the computer readable memory medium stores a computer program that performs the automatic concrete dam defect image description generation method based on the graph attention network in claim 1.

### Claim 18 (depends on 2)

A computer readable memory medium, characterized in that the computer readable memory medium stores a computer program that performs the automatic concrete dam defect image description generation method based on the graph attention network in claim 2.

### Claim 19 (depends on 3)

A computer readable memory medium, characterized in that the computer readable memory medium stores a computer program that performs the automatic concrete dam defect image description generation method based on the graph attention network in claim 3.

### Claim 20 (depends on 4)

A computer readable memory medium, characterized in that the computer readable memory medium stores a computer program that performs the automatic concrete dam defect image description generation method based on the graph attention network in claim 4.
