US 10,380,236 B1Granted

Machine learning system for annotating unstructured text

AssigneeAmazon Technologies, Inc.

InventorsHrishikesh Vidyadhar Ganu, Rajeev Ramnarain Rastogi, Subhajit Sanyal

Life of this patent

Estimating the term…

Loading patent…

Abstract

Systems and methods are disclosed to implement a machine learning system that is trained to assign annotations to text fragments in an unstructured sequence of text. The system employs a neural model that includes an encoder recurrent neural network (RNN) and a decoder RNN. The input text sequence is encoded by the encoder RNN into successive encoder hidden states. The encoder hidden states are then decoded by the decoder RNN to produce a sequence of annotations for text fragments within the text sequence. In embodiments, the system employs a fixed-attention window during the decoding phase to focus on a subset of encoder hidden states to generate the annotations. In embodiments, the system employs a beam search technique to track a set of candidate annotation sequences before the annotations are outputted. By using a decoder RNN, the neural model is better equipped to capture long-range annotation dependencies in the text sequence.

Claims20

Dependent claims

A computer-implemented method comprising:

receiving an unstructured text sequence comprising a plurality of words;
encoding the plurality of words into a plurality of encoder hidden states of an encoder recurrent neural network (RNN), wherein the encoder RNN is configured to generate a given encoder hidden state based at least in part on a word and a previous encoder hidden state;
decoding one or more of the plurality of encoder hidden states into a plurality of decoder hidden states of a decoder RNN, wherein the decoder RNN is configured to generate a given decoder hidden state based at least in part on a previous decoder hidden state and an output of the previous decoder hidden state;
and
generating a sequence of annotations for individual text fragments in the unstructured text from to the decoder hidden states, wherein the annotations are selected from a set of annotations used to train the encoder RNN and decoder RNN, and generating an annotation in the sequence comprises:
determining a particular encoder hidden state associated with a particular word in the unstructured text sequence that is in a position aligned with the annotation;
determining a fixed-size attention window of a subset of the plurality of encoder hidden states surrounding the position of the particular encoder state;
and
generating the annotation based at least in part on the subset of encoder hidden states in the attention window.

6 more elements

A system, comprising:

one or more hardware processors with associated memory, implementing a machine learning system configured to:
receive an unstructured text sequence comprising a plurality of words;
encode the plurality of words into a plurality of encoder hidden states of an encoder recurrent neural network (RNN), wherein the encoder RNN is configured to generate a given encoder hidden state based at least in part on a word and a previous encoder hidden state;
decode one or more of the plurality of encoder hidden states into a plurality of decoder hidden states of a decoder RNN, wherein to generate a given decoder hidden state based at least in part on a previous decoder hidden state and an output of the previous decoder hidden state;
and
generate a sequence of annotations for individual text fragments in the unstructured text from to the decoder hidden states, wherein the annotations are selected from a set of annotations used to train the encoder RNN and decoder RNN, and to generate an annotation in the sequence, the decoder RNN is configured to:
determine a particular encoder hidden state associated with a particular word in the unstructured text sequence that is in a position aligned with the annotation;
determine a fixed-size attention window of a subset of the plurality of encoder hidden states surrounding the position of the particular encoder state;
and
generate the annotation based at least in part on the subset of encoder hidden states in the attention window.

7 more elements

A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors of a machine learning system, cause the machine learning system to:

receive an unstructured text sequence comprising a plurality of words;
encode the plurality of words into a plurality of encoder hidden states of an encoder recurrent neural network (RNN), wherein the encoder RNN is configured to generate a given encoder hidden state based at least in part on a word and a previous encoder hidden state;
decode one or more of the plurality of encoder hidden states into a plurality of decoder hidden states of a decoder RNN, wherein the decoder RNN is configured to generate a given decoder hidden state based at least in part on a previous decoder hidden state and an output of the previous decoder hidden state;
and
generate a sequence of annotations for individual text fragments in the unstructured text from to the decoder hidden states, wherein the annotations are selected from a set of annotations used to train the encoder RNN and decoder RNN, and to generate an annotation in the sequence, the program instructions when executed on the one or more processors cause the machine learning system to:
determine a particular encoder hidden state associated with a particular word in the unstructured text sequence that is in a position aligned with the annotation;
determine a fixed size attention window of a subset of the plurality of encoder hidden states surrounding the position of the particular encoder state;
and
generate the annotation based at least in part on the subset of encoder hidden states in the attention window.

6 more elements

Dependencies are inferred from the claim text.