# US 10,373,060 B2 · Answer scoring by using structured resources to generate paraphrases

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,373,060 B2
- Document ID: US10373060B2
- Status: Granted
- Kind: B2
- Application number: 14/886,023
- Filed: 2015-10-17
- Published: 2017-04-20
- Granted: 2019-08-06
- Earliest claimed: 2015-10-17
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Stephen A. Boxwell; Jared M. Smythe
- Web page: https://patentgrep.com/patent/US10373060B2

## CPC

- G06N 5/047: Pattern matching networks; Rete networks
- G06F 16/345: {Summarisation for human users}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 5/02: Knowledge representation; Symbolic representation
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An approach is provided in which a knowledge manager creates a pattern set that includes paraphrases and corresponding paraphrase scores. The paraphrase scores are based on a set of first candidate answers obtained from querying a first set of resource data. The knowledge manager performs a search, which is based on the paraphrases and a user question, on a second set of resource data and identifies a set of second candidate answers. In turn, the knowledge manager scores the set of second candidate answers based on the paraphrase scores corresponding to the paraphrases utilized to identify the set of second candidate answers.

## Claims

### Claim 1

A method implemented by an information handling system that includes a memory and a processor, the method comprising:
creating a pattern set that comprises a plurality of paraphrases and a plurality of paraphrase scores, wherein the plurality of paraphrase scores are based on a plurality of query answers obtained from querying a first set of resource data, the creating further comprising:
generating a first one of the plurality of paraphrases and a first query utilizing a first set of training data;
using the first query to query the first set of resource data and produce a first query answer, wherein the first query answer is included in the plurality of query answers;
generating a first one of the plurality of paraphrase scores based upon comparing the first query answer against a training answer included in the first set of training data; and
assigning the first paraphrase score to the first paraphrase based on the first paraphrase being generated from the first set of training data;
utilizing the plurality of paraphrases and a user question to perform a search on a second set of resource data, wherein the search results in a set of candidate answers corresponding to the user question;
in response to determining that the first paraphrase was utilized to identify the selected candidate answer in the second set of resource data, assigning the first paraphrase score to a selected one of the candidate answers; and
providing the selected candidate answer to a user based on the assigned score.

### Claim 2 (depends on 1)

The method of claim 1 wherein the first set of training data is selected from the group consisting of a training question/answer pair and a training statement.

### Claim 3 (depends on 2)

The method of claim 2 further comprising:
retrieving a second set of training data;
creating a second one of the plurality of paraphrases and a second query based on the second set of training data;
comparing the second query to the first query; and
in response to determining that the second query correlates to the first query, adding the second paraphrase to the pattern set.

### Claim 4 (depends on 1)

The method of claim 1 wherein the pattern set includes a plurality of first syntactic patterns, the method further comprising:
generating a second syntactic pattern based on the user question;
creating a plurality of new questions utilizing the plurality of paraphrases in response to matching one of the plurality of first syntactic patterns to the second syntactic pattern; and
utilizing the plurality of new questions to perform the search on the second set of resource data.

### Claim 5 (depends on 4)

The method of claim 4 wherein the creating of the plurality of new questions further comprises:
selecting one or more terms in the user question; and
replacing one or more variables in the plurality of paraphrases with the selected one or more terms.

### Claim 6 (depends on 1)

The method of claim 1 wherein the first set of resource data is structured data, and wherein the second set of resource data is unstructured data.

### Claim 7

An information handling system comprising:
one or more processors;
a memory coupled to at least one of the processors; and
a set of computer program instructions stored in the memory and executed by at least one of the processors in order to perform actions of:
creating a pattern set that comprises a plurality of paraphrases and a plurality of paraphrase scores, wherein the plurality of paraphrase scores are based on a plurality of query answers obtained from querying a first set of resource data, the creating further comprising:
generating a first one of the plurality of paraphrases and a first query utilizing a first set of training data;
using the first query to query the first set of resource data and produce a first query answer, wherein the first query answer is included in the plurality of query answers;
generating a first one of the plurality of paraphrase scores based upon comparing the first query answer against a training answer included in the first set of training data; and
assigning the first paraphrase score to the first paraphrase based on the first paraphrase being generated from the first set of training data;
utilizing the plurality of paraphrases and a user question to perform a search on a second set of resource data, wherein the search results in a set of candidate answers corresponding to the user question;
in response to determining that the first paraphrase was utilized to identify the selected candidate answer in the second set of resource data, assigning the first paraphrase score to a selected one of the candidate answers; and
providing the selected candidate answer to a user based on the assigned score.

### Claim 8 (depends on 7)

The information handling system of claim 7 wherein the first set of training data is selected from the group consisting of a training question/answer pair and a training statement.

### Claim 9 (depends on 8)

The information handling system of claim 8 wherein at least one of the one or more processors perform additional actions comprising:
retrieving a second set of training data;
creating a second one of the plurality of paraphrases and a second query based on the second set of training data;
comparing the second query to the first query; and
in response to determining that the second query correlates to the first query, adding the second paraphrase to the pattern set.

### Claim 10 (depends on 7)

The information handling system of claim 7 wherein the pattern set includes a plurality of first syntactic patterns, and wherein at least one of the one or more processors perform additional actions comprising:
generating a second syntactic pattern based on the user question;
creating a plurality of new questions utilizing the plurality of paraphrases in response to matching one of the plurality of first syntactic patterns to the second syntactic pattern; and
utilizing the plurality of new questions to perform the search on the second set of resource data.

### Claim 11 (depends on 10)

The information handling system of claim 10 wherein at least one of the one or more processors perform additional actions comprising:
selecting one or more terms in the user question; and
replacing one or more variables in the plurality of paraphrases with the selected one or more terms.

### Claim 12 (depends on 7)

The information handling system of claim 7 wherein the first set of resource data is structured data, and wherein the second set of resource data is unstructured data.

### Claim 13

A computer program product stored in a computer readable storage medium, comprising computer program code that, when executed by an information handling system, causes the information handling system to perform actions comprising:
creating a pattern set that comprises a plurality of paraphrases and a plurality of paraphrase scores, wherein the plurality of paraphrase scores are based on a plurality of query answers obtained from querying a first set of resource data, the creating further comprising:
generating a first one of the plurality of paraphrases and a first query utilizing a first set of training data;
using the first query to query the first set of resource data and produce a first query answer, wherein the first query answer is included in the plurality of query answers;
generating a first one of the plurality of paraphrase scores based upon comparing the first query answer against a training answer included in the first set of training data; and
assigning the first paraphrase score to the first paraphrase based on the first paraphrase being generated from the first set of training data;
utilizing the plurality of paraphrases and a user question to perform a search on a second set of resource data, wherein the search results in a set of candidate answers corresponding to the user question;
in response to determining that the first paraphrase was utilized to identify the selected candidate answer in the second set of resource data, assigning the first paraphrase score to a selected one of the candidate answers; and
providing the selected candidate answer to a user based on the assigned score.

### Claim 14 (depends on 13)

The computer program product of claim 13 wherein the first set of training data is selected from the group consisting of a training question/answer pair and a training statement.

### Claim 15 (depends on 14)

The computer program product of claim 14 wherein the information handling system performs additional actions comprising:
retrieving a second set of training data;
creating a second one of the plurality of paraphrases and a second query based on the second set of training data;
comparing the second query to the first query; and
in response to determining that the second query correlates to the first query, adding the second paraphrase to the pattern set.

### Claim 16 (depends on 13)

The computer program product of claim 13 wherein the pattern set includes a plurality of first syntactic patterns, and wherein the information handling system performs additional actions comprising:
generating a second syntactic pattern based on the user question;
creating a plurality of new questions utilizing the plurality of paraphrases in response to matching one of the plurality of first syntactic patterns to the second syntactic pattern; and
utilizing the plurality of new questions to perform the search on the second set of resource data.

### Claim 17 (depends on 16)

The computer program product of claim 16 wherein the information handling system performs additional actions comprising:
selecting one or more terms in the user question; and
replacing one or more variables in the plurality of paraphrases with the selected one or more terms.
