US 9,002,843 B2Granted

System and method for extraction of off-topic part from conversation

AssigneeInternational Business Machines Corporation

InventorsNobuyasu Itoh, Masafumi Nishimura, Yuto Yamaguchi

Life of this patent

Estimating the term…

Loading patent…

Abstract

A system and method extract off-topic parts from a conversation. The system includes a first corpus including documents of a plurality of fields; a second corpus including only documents of a field to which the conversation belongs; a determination means for determination as a lower limit subject word a word for which IDF value for the first corpus and IDF value for the second corpus are each below a first certain threshold value; a score calculation part for calculation as a score a TF-IDF value for each word included in the second corpus; a clipping part, for sequential cutting out of intervals from text data that are contents of the conversation; and an extraction part for extraction as an off-topic part an interval where average value of the score of words included in the clipped interval is larger than a second certain threshold value.

Claims9

Dependent claims

A system for extraction of an off-topic part from a conversation, the system comprising:

a memory;
a processing unit connected to the memory;
a first corpus stored in the memory, the first corpus including documents of a plurality of fields;
a second corpus stored in the memory, the second corpus including only documents of a field to which said conversation belongs;
a determination means stored in the memory, the determination means interoperates with the processing unit for determination of, as a lower limit subject word, a word for which IDF value for said first corpus, wherein the IDF value in each corpus is found according to the following formula:
IDF
⁡
(
w
)
=
log
⁡
(
D

DF
⁡
(
w
)
)
, where D indicates the number of documents contained in each corpus, and DF(w) indicates the number of documents that include a word w within the documents contained in each corpus, and IDF value for said second corpus are each below a first certain threshold value for each word included in said second corpus;
a score calculation part stored in the memory, the score calculation part interoperates with the processing unit for calculation as a score a TF-IDF value, the TF-IDF value being determined based on the product of the term frequency of appearance in a target document and the log of the inverse of the proportion of document frequency of appearance of the term, for each word included in said second corpus, said score calculation part using a constant setting a lower limit rather than a TF-IDF value for said lower limit subject word;
a clipping part stored in the memory, the clipping part interoperates with the processing unit, the clipping part, while displacing a window of a certain length sequentially over text data comprising words of said conversation acquired by speech recognition, for sequential cutting out of clipped intervals subject to processing from text data comprising words that are contents of said conversation;
and
an extraction part stored in the memory, the extraction part interoperates with the processing unit for extraction of, as an off-topic part of a conversation, a clipped interval where an average value of score of words included in the clipped interval is larger than a second certain threshold value.

24 more elements

An off-topic part extraction method for use with a computer for calculation processing to extract an off-topic part from a conversation:

wherein said computer is capable of accessing a first corpus that includes documents of a plurality of fields, and is capable of accessing a second corpus that includes only documents of a field to which said conversation belongs;
and
wherein said off-topic part extraction method comprises the steps of:
determining by said computer as a lower limit subject word, for each word included in said second corpus, a word for which IDF value in said first corpus, wherein the IDF value in each corpus is found according to the following formula:
IDF
⁡
(
w
)
=
log
⁡
(
D

DF
⁡
(
w
)
)
, where D indicates the number of documents contained in each corpus, and DF(w) indicates the number of documents that include a word w within the documents contained in each corpus, and IDF value in said second corpus each becomes less than or equal to a first certain threshold value;
calculating by said computer as a score a TF-IDF value, the TF-IDF value being determined based on the product of the term frequency of appearance in a target document and the log of the inverse of the proportion of document frequency of appearance of the term, for each word included in said second corpus, using a constant setting a lower limit rather than a TF-IDF value for said lower limit subject word, and storing said score in a dictionary memory unit;
cutting out sequentially by said computer, from text data comprising words that are contents of said conversation, clipped intervals subject to processing while displacing a window of a certain length;
and
referencing, by said computer, information stored in said dictionary memory unit, and finding an average value of a score of words included in a clipped interval, and extracting as an off-topic part of a conversation a clipped interval where the average value is larger than a second certain threshold value.

23 more elements

A computer program product for extraction of an off-topic part from a conversation between a customer and a person in charge of business, the computer program product comprising:

a non-transitory computer readable medium;
and
computer program instructions for executing the following method:
determining, by a computer that is capable of accessing a first corpus including documents of a plurality of fields and that is capable of accessing a second corpus including only documents of a field to which said conversation belongs, as a lower limit subject word, for each word included in said second corpus, a word for which IDF value in said first corpus, wherein the IDF value in each corpus is found according to the following formula:
IDF
⁡
(
w
)
=
log
⁡
(
D

DF
⁡
(
w
)
)
, where D indicates the number of documents contained in each corpus, and DF(w) indicates the number of documents that include a word w within the documents contained in each corpus, and IDF value in said second corpus each becomes less than or equal to a first certain threshold value, said second corpus being formed from a customer-use corpus that is a collection of speech parts of a customer and an in-charge person-use corpus that is a collection of speech parts of the person in charge of business;
calculating as a score an RF-IDF value, the RF-IDF value being determined based on the product of a proportion of appearance frequency CF(w) of said word w in said customer-use corpus relative to appearance frequency SF(w) of said word w in said in-charge person-use corpus and the log of the inverse of the proportion of document frequency of appearance of the term, for each word included in said second corpus, using a constant setting a lower limit rather than an RF-IDF value for said lower limit subject word, and storing said score in a dictionary memory unit;
cutting out sequentially intervals subject to processing while displacing a window of a certain length from text data comprising words that are contents of said conversation;
and
referencing of information stored in said dictionary memory unit, and finding an average value of said score of words included in said clipped interval, and extracting as an off-topic part an interval where said average value is larger than a second certain threshold value, and wherein said proportion of appearance frequency CF(w) of said word w in said customer-use corpus relative to appearance frequency SF(w) of said word w in said in-charge person-use corpus is found according to the following formula (CF(w)+1)/(CF(w)+SF(w)).

23 more elements

Dependencies are inferred from the claim text.