# US 2021/0279414 A1 · Interpretable label-attentive encoder-decoder parser

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279414 A1
- Document ID: US20210279414A1
- Status: Published application
- Kind: A1
- Application number: 16/810,345
- Filed: 2020-03-05
- Published: 2021-09-09
- Earliest claimed: 2020-03-05
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: KHALIL MRINI; WALlTER CHANG; TRUNG BUI; QUAN TRAN; FRANCK DERNONCOURT
- Web page: https://patentgrep.com/patent/US20210279414A1

## CPC

- G06F 40/211: Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06F 40/216: using statistical methods
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/289: Phrasal analysis, e.g. finite state techniques or chunking
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for parsing natural language sentences using an artificial neural network (ANN) are described. Embodiments of the described systems and methods may generate a plurality of word representation matrices for an input sentence, wherein each of the word representation matrices is based on an input matrix of word vectors, a query vector, a matrix of key vectors, and a matrix of value vectors, and wherein a number of the word representation matrices is based on a number of syntactic categories, compress each of the plurality of word representation matrices to produce a plurality of compressed word representation matrices, concatenate the plurality of compressed word representation matrices to produce an output matrix of word vectors, and identify at least one word from the input sentence corresponding to a syntactic category based on the output matrix of word vectors.

## Claims

### Claim 1

A method for natural language processing, comprising:
generating a plurality of word representation matrices for an input sentence, wherein each of the word representation matrices is based on an input matrix of word vectors, a query vector, a matrix of key vectors, and a matrix of value vectors, and wherein a number of the word representation matrices is based on a number of syntactic categories;
concatenating a plurality of compressed word representation matrices to produce an output matrix of word vectors; and
identifying at least one word from the input sentence corresponding to a syntactic category based on the output matrix of word vectors; and
generating annotation information for the input sentence based on identifying the at least one word corresponding to the syntactic category.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
compressing each of the plurality of word representation matrices to produce the plurality of compressed word representation matrices;

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
multiply the query vector by the matrix of key vectors to produce an attention weight vector;
multiply the attention weight vector by the matrix of value vectors to produce a context vector; and
adding a plurality of repeated versions of the context vector to the input matrix of word vectors to produce a word representation matrix.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
projecting the context vector into a different number of dimensions, wherein the repeated versions of the context vector are based on the projection.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
identifying a plurality of labels corresponding to the syntactic categories, wherein a number of the labels is based on a number of label attention heads; and
generating label interpretation information for the at least one span or words based on a correspondence between the labels and one or more of the label attention heads, wherein the label interpretation information comprises information regarding the significance of each word of the input sentence for identifying the at least one span of words corresponding to the syntactic category.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
the matrix of key vectors is a product of a label-specific key matrix and the input matrix of word vectors, the matrix of value vectors is a product of a label-specific value matrix and the input matrix of word vectors, and the query vector is not a product of the input matrix of word vectors.

### Claim 7

A system for natural language processing, comprising:
an encoder comprising a label attention layer configured to produce an output matrix of word vectors based on an input matrix of word vectors using a plurality of label attention heads corresponding to a number of syntactic categories, wherein each of the label attention heads produces a word representation matrix based on the input matrix of word vectors, a query vector, a matrix of key vectors, and a matrix of value vectors; and
a decoder configured to identify at least one span of words of an input sentence corresponding to a syntactic category based on the output matrix of word vectors.

### Claim 8 (depends on 7)

The system of claim 7, further comprising:
an interpreter configured to generate label interpretation information based on a correspondence between the syntactic categories and the label attention heads.

### Claim 9 (depends on 7)

The system of claim 7, further comprising:
a query sublayer configured to multiply the query vector by a matrix of key vectors to produce an attention weight vector;
an attention sublayer configured to the attention weight vector by a matrix of value vectors to produce a context vector;
a word representation sublayer configured to add a plurality of repeated versions the context vector to an input matrix of word vectors to produce a word representation matrix;
a projection sublayer configured to compress the word representation matrix to produce a compressed word representation matrix; and
a concatenation sublayer configured to concatenate the compressed word representation matrices to produce the output matrix of word vectors.

### Claim 10 (depends on 7)

The system of claim 7, wherein:
the encoder comprises a plurality of layers, wherein an output of each layer prior to a final layer is provided as an input for a subsequent layer, and wherein the label attention layer comprises the final layer.

### Claim 11 (depends on 7)

The system of claim 7, further comprising:
a word embedding function configured to produce the input matrix of word vectors.

### Claim 12 (depends on 11)

The system of claim 11, wherein:
the word embedding function comprises an XLNet word embedding algorithm.

### Claim 13 (depends on 7)

The system of claim 7, wherein:
the decoder comprises a softmax function.

### Claim 14 (depends on 7)

The system of claim 7, wherein:
the decoder comprises a CKY decoder.

### Claim 15

A method for training natural language processing model, comprising:
generating a word representation matrix for each of a plurality of label attention heads, wherein each of the word representation matrices is based on an input matrix of word vectors, a query vector, a label-specific key matrix, and a label-specific value matrix, and wherein a number of the label attention heads is based on a number of syntactic categories;
compressing each of the word representation matrices to produce a plurality of compressed word representation matrices;
concatenating the compressed word representation matrices to produce an output matrix of word vectors;
computing a loss value based at least in part on the output matrix of word vectors; and
updating the label-specific key matrix and the label-specific value matrix based on the loss value.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
multiply the label-specific key matrix and the input matrix of word vectors to produce a matrix of key vectors; and
multiply the label-specific value matrix and the input matrix of word vectors to produce the matrix of value vectors.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
multiply the query vector by the matrix of key vectors to produce an attention weight vector;
multiply the attention weight vector by the matrix of value vectors to produce a context vector; and
adding a plurality of repeated versions of the context vector to the input matrix of word vectors to produce a word representation matrix.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
projecting the context vector into a different number of dimensions, wherein the repeated versions of the context vector are based on the projection.

### Claim 19 (depends on 15)

The method of claim 15, further comprising:
identifying an input sentence;
embedding the input sentence in an embedding space; and
applying at least one self-attention layer to the embedded input sentence to produce the input matrix of word vectors.

### Claim 20 (depends on 15)

The method of claim 15, further comprising:
applying a softmax function and a dropout function, wherein the loss value is computed based on the softmax function and the dropout function.
