Corpus cleaning method and corpus entry system
AssigneeUbtech Robotics Corp Ltd
InventorsLI MA, Youjun Xiong
Life of this patent
Estimating the term…
Abstract
The present disclosure provides a corpus cleaning method and a corpus entry system. The method includes: obtaining an input utterance; generating a predicted value of an information amount of each word in the input utterance according to the context of the input utterance using a pre-trained general model; and determining redundant words according to the predicted value of the information amount of each word, and determining whether to remove the redundant words from the input utterance. In such a manner, the objectivity and accuracy of corpus cleaning can be improved.
Claims20
A computer-implemented corpus cleaning method, comprising executing on a processor steps of:
- obtaining an input utterance, wherein the input utterance is obtained from computerized text;
- generating a predicted value of an information amount of each word in the input utterance according to the context of the input utterance using a pre-trained general model;
- and
- determining redundant words according to the predicted value of the information amount of each word, and determining whether to remove the redundant words from the input utterance.
A corpus entry system, comprising:
- a memory;
- a processor;
- and
- one or more computer programs stored in the memory and executable on the processor, wherein the one or more computer programs comprise:
- instructions for obtaining an input utterance, wherein the input utterance is obtained from computerized text;
- instructions for generating a predicted value of an information amount of each word in the input utterance according to the context of the input utterance using a pre-trained general model;
- and
- instructions for determining redundant words according to the predicted value of the information amount of each word, and determining whether to remove the redundant words from the input utterance.
Dependencies are inferred from the claim text.