# US 10,482,176 B2 · System and method for quality evaluation of collaborative text inputs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,176 B2
- Document ID: US10482176B2
- Status: Granted
- Kind: B2
- Application number: 15/920,243
- Filed: 2018-03-13
- Published: 2019-04-18
- Granted: 2019-11-19
- Earliest claimed: 2017-10-17
- Assignees: Tata Consultancy Services Limited
- Applicants: Tata Consultancy Services Limited
- Inventors: Manasi Smarth Patwardhan; Kanika Kalra; Mandar Shrikant Kulkarni; Shirish Subhash Karande
- Web page: https://patentgrep.com/patent/US10482176B2

## CPC

- G06F 40/289: Phrasal analysis, e.g. finite state techniques or chunking
- G06F 40/166: Editing, e.g. inserting or deleting
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06Q 10/06395: {Quality analysis or management}
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

This disclosure relates generally to quality evaluation of collaborative text input, and more particularly to system and method for quality evaluation of collaborative text inputs using Long Short Term Memory (LSTM) networks. In one embodiment, the method includes receiving an input data associated with a task to be accomplished collaboratively and sequentially by a plurality of contributors. The input data includes task-wise data sequence of contributor's post-edit submissions. A plurality of features are extracted from the input data. Based on the plurality of features, a plurality of input sequences are constructed. The input sequences include a plurality of concatenated feature vectors, where each of the concatenated feature vectors includes a post-edit feature vector and a contributor representation feature vector. The input sequences are modelled as a LSTM network, where the LSTM network is utilized to train a binary classifier for quality evaluation of the post-edit submission.

## Claims

### Claim 1

A processor-implemented method for quality evaluation of collaborative text input, comprising:
receiving, via one or more hardware processors, an input data associated with a task to be accomplished collaboratively and sequentially by a plurality of contributors, the input data comprising task-wise data sequence of contributor's post-edit submissions;
extracting a plurality of features from the input data, via the one or more hardware processors;
constructing, via the one or more hardware processors, a plurality of input sequences based on the plurality of features, an input sequence of the plurality of input sequences comprising a plurality of concatenated feature vectors, each of the concatenated feature vectors comprising a post-edit feature vector and a contributor representation feature vector; and
modelling, via the one or more hardware processors, the plurality of input sequences as a Long Short Term Memory (LSTM) network, wherein the LSTM network is utilized to train a first binary classifier for quality evaluation of the post-edit submission.

### Claim 2 (depends on 1)

The method of claim 1, wherein the post-edit submission is determined to be improved over prior post-edit submissions if the first binary classifier determines a binary class as 1, and post-edit submission is determined not to be improved if the first binary classifier determines the binary class as 0.

### Claim 3 (depends on 1)

The method of claim 1, wherein the task comprises a translation of text, for each post-edit submission, the post-edit feature vector comprises an edit vector of a current submission, and wherein the post-edit feature vector comprises of ratios of word-wise inserts, deletions and substitutions made by a contributor to a selected parent submission with total number of words, word length of a source sentence associated with the task, and quantized and normalized time taken by the contributor to perform the task.

### Claim 4 (depends on 1)

The method of claim 1, wherein for every post-edit submission, the contributor representation feature vector comprises one of one-hot vector encoding of dimensions of total number of contributors and learned contributor embedding by feeding the one-hot-vectors associated with the contributors to a neural network (NN) with a single linear hidden layer.

### Claim 5 (depends on 4)

The method of claim 4, further comprising learning the contributor embedding during the training of the LSTM network.

### Claim 6 (depends on 1)

The method of claim 1, further comprising training a second binary classifier, by the LSTM network to predict a probability of a contributor improving upon prior submissions.

### Claim 7 (depends on 6)

The method of claim 6, wherein predicting the probability comprises:
mapping a history of prior post-edit submissions into an intermediate embedding, the history of prior post-edit submissions obtained from the input data;
applying output of the LSTM networks and current contributor embedding to a neural network (NN), wherein the output of the LSTM networks comprises the history embedding at the last time stamp; and
predicting the probability of a contributor improving upon the prior submissions based on the output of the NN.

### Claim 8 (depends on 1)

The method of claim 1, further comprising training an encoder-decoder architecture to predict sequence of ranks for each of the plurality of sequence of submissions based on the plurality of sequences of the concatenated feature vectors.

### Claim 9 (depends on 8)

The method of claim 8, wherein training the encoder-decoder architecture comprises:
applying the sequence of concatenated feature vectors into the encoder-decoder architecture, the encoder-decoder architecture predicts a sequence of binary labels generated from selected expert provided ranks of the post-edit submissions as ordinal labels;
obtaining, from the encoder, the embedding of the last hidden state which represents the history of post-edit;
feeding the embedding to the decoder which learns the sequence of binary labels for each time step to provide a binary sequence; and
calculating ranks based on the binary sequence obtained from the encoder-decoder.

### Claim 10

A system for quality evaluation of collaborative text input, the system comprising:
at least one memory storing instructions; and
one or more hardware processors coupled to said at least one memory, wherein said one or more hardware processors are configured by said instructions to:
receive an input data associated with a task to be accomplished collaboratively and sequentially by a plurality of contributors, the input data comprising task-wise data sequence of contributor's post-edit submissions;
extract a plurality of features from the input data;
construct a plurality of input sequences based on the plurality of features, an input sequence of the plurality of input sequences comprising a plurality of concatenated feature vectors, each of the concatenated feature vectors comprising a post-edit feature vector and a contributor representation feature vector; and
model the plurality of input sequences as a Long Short Term Memory (LSTM) network, wherein the LSTM network is utilized to train a first binary classifier for quality evaluation of the post-edit submission.

### Claim 11 (depends on 10)

The system of claim 10, wherein the one or more hardware processors are configured by the instructions to determine if the post-edit submission is improved over prior post-edit submissions when the first binary classifier determines a binary class as 1, and determines the post-edit submission not to be improved when the first binary classifier determines the binary class as 0.

### Claim 12 (depends on 10)

The system of claim 10, wherein the task comprises a translation of text, for every post-edit submission, the post-edit feature vector comprises an edit vector of a current submission, and wherein the post-edit feature vector comprises of ratios of word-wise inserts, deletions and substitutions made by a contributor to a selected parent submission with total number of words; word length of a source sentence, associated with the task, and quantized and normalized time taken by the contributor to perform the task.

### Claim 13 (depends on 10)

The system of claim 10, wherein for every post-edit submission, the contributor representation feature vector comprises one of one-hot vector encoding of dimensions of total number of contributors and learned contributor embedding by feeding the one-hot-vectors associated with the contributors to a neural network (NN) with a single linear hidden layer.

### Claim 14 (depends on 13)

The system of claim 13, wherein the one or more hardware processors are configured by the instructions to learn the contributor embedding during the training of the LSTM network.

### Claim 15 (depends on 10)

The system of claim 10, wherein the one or more hardware processors are configured by the instructions to train a second binary classifier, by the LSTM network to predict a probability of a contributor improving upon prior submissions.

### Claim 16 (depends on 15)

The system of claim 15, wherein to predict the probability, the one or more hardware processors are configured by the instructions to:
map a history of prior post-edit submissions into an intermediate embedding, the history of prior post-edit submissions obtained from the input data;
apply output of the LSTM networks and current contributor embedding to a neural network (NN), wherein the output of the LSTM networks comprises the history embedding at the last time stamp; and
predict the probability of a contributor improving upon the prior submissions based on the output of the NN.

### Claim 17 (depends on 10)

The system of claim 10, wherein the one or more hardware processors are configured by the instructions to train an encoder-decoder architecture to predict sequence of ranks for each of the plurality of sequence of submissions based on the plurality of sequences of the concatenated feature vectors.

### Claim 18 (depends on 17)

The system of claim 17, wherein to train the encoder-decoder architecture, the one or more hardware processors are configured by the instructions to:
apply the sequence of concatenated feature vectors into the encoder-decoder architecture, the encoder-decoder architecture predicts a sequence of binary labels generated from selected expert provided ranks of the post-edit submissions as ordinal labels;
obtain, from the encoder, the embedding of the last hidden state which represents the history of post-edit;
feed the embedding to the decoder which learns the sequence of binary labels for each time step to provide a binary sequence; and
calculate ranks based on the binary sequence obtained from the encoder-decoder.

### Claim 19

A non-transitory computer-readable medium having embodied thereon a computer program for executing a method for quality evaluation of collaborative text input, the method comprising:
receiving, via one or more hardware processors, an input data associated with a task to be accomplished collaboratively and sequentially by a plurality of contributors, the input data comprising task-wise data sequence of contributor's post-edit submissions;
extracting a plurality of features from the input data, via the one or more hardware processors;
constructing, via the one or more hardware processors, a plurality of input sequences based on the plurality of features, an input sequence of the plurality of input sequences comprising a plurality of concatenated feature vectors, each of the concatenated feature vectors comprising a post-edit feature vector and a contributor representation feature vector; and
modelling, via the one or more hardware processors, the plurality of input sequences as a Long Short Term Memory (LSTM) network, wherein the LSTM network is utilized to train a first binary classifier for quality evaluation of the post-edit submission.