US 2018/0121405 A1Published application

System and method of annotating utterances based on tags assigned by unmanaged crowds

AssigneeVoiceBox Technologies, Inc.

InventorsSpencer John ROTHWELL, Daniela BRAGA, Ahmad Khamis ELSHENAWY, Stephen Steele CARTER

Life of this patent

Estimating the term…

Loading patent…

Abstract

A system and method of tagging utterances with Named Entity Recognition (“NER”) labels using unmanaged crowds is provided. The system may generate various annotation jobs in which a user, among a crowd, is asked to tag which parts of an utterance, if any, relate to various entities associated with a domain. For a given domain that is associated with a number of entities that exceeds a threshold N value, multiple batches of jobs (each batch having jobs that have a limited number of entities for tagging) may be used to tag a given utterance from that domain. This reduces the cognitive load imposed on a user, and prevents the user from having to tag more than N entities. As such, a domain with a large number of entities may be tagged efficiently by crowd participants without overloading each crowd participant with too many entities to tag.

Claims20

Dependent claims

A computer implemented method of tagging utterances with Named Entity Recognition (“NER”) labels using an unmanaged crowd, the method being implemented in an end user device having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the end user device to perform the method, the method comprising:

obtaining, by the computer system, a plurality of utterances relating to a domain, the domain being associated with a plurality of entities, each entity relating to a category of information in the domain;
generating, by the computer system, a first annotation job configured to request that at least a first portion of the utterance be assigned to one of a first set of entities, from among the plurality of entities, wherein a number of the first set of entities does not exceed a maximum number such that cognitive load imposed on a user to whom the first annotation job is provided is controlled;
generating, by the computer system, a second annotation job configured to request that at least a second portion of the utterance be assigned to one of a second set of entities, from among the plurality of entities, wherein:
a number of the second set of entities does not exceed the maximum number such that cognitive load imposed on a user to whom the second annotation job is provided is controlled, the first portion and the second portion are the same or different and the first set of entities is different than the second set of entities, and the user to whom the first annotation job is provided is the same or different from the user to whom the second annotation job is provided;
causing, by the computer system, the first annotation job and the second annotation to be deployed to the unmanaged crowd;
and
receiving, by the computer system, a plurality of annotations provided by the unmanaged crowd, the plurality of annotation comprising a first annotation relating to the first annotation job and a second annotation relating to the second annotation job.

4 more elements

A system of tagging utterances with Named Entity Recognition (“NER”) labels using unmanaged crowds, the comprising:

an end user device having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the end user device to:
obtain a plurality of utterances relating to a domain, the domain being associated with a plurality of entities, each entity relating to a category of information in the domain;
generate a first annotation job configured to request that at least a first portion of the utterance be assigned to one of a first set of entities, from among the plurality of entities, wherein a number of the first set of entities does not exceed a maximum number such that cognitive load imposed on a user to whom the first annotation job is provided is controlled;
generate a second annotation job configured to request that at least a second portion of the utterance be assigned to one of a second set of entities, from among the plurality of entities, wherein:
a number of the second set of entities does not exceed the maximum number such that cognitive load imposed on a user to whom the second annotation job is provided is controlled, the first portion and the second portion are the same or different and the first set of entities is different than the second set of entities, and the user to whom the first annotation job is provided is the same or different from the user to whom the second annotation job is provided;
cause the first annotation job and the second annotation to be deployed to the unmanaged crowd;
and
receive a plurality of annotations provided by the unmanaged crowd, the plurality of annotation comprising a first annotation relating to the first annotation job and a second annotation relating to the second annotation job.

5 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/675,908Filed 14 Aug 2017
Priority: 7 Sep 2015
Published: 3 May 2018
Term ends: Not estimatedThe source records don’t support an estimate.

Classification5 codes

G06F 40/169
Editing, e.g. inserting or deleting: annotation, e.g. comment data or footnotes
G06F 40/117
Formatting, i.e. changing of presentation of documents: tagging; Marking up ; Designating a block; Setting of attributes
G06F 40/295
Phrasal analysis, e.g. finite state techniques or chunking: named entity recognition
G06F 40/30
Handling natural language data: semantic analysis

Family

Application62/215,116Us provisional application · filed 7 Sep 2015
Application15/257,084Continuation · filed 6 Sep 2016
Application15/675,908Continuation
US 10,394,944 B2Related patent
US 2017/0068651 A1Published application
US 9,734,138 B2Related patent
US 2018/0121405 A1This patent

From the US filing history. Foreign family members aren’t listed.