# US 10,318,885 B2 · Cognitive system virtual corpus training and utilization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,318,885 B2
- Document ID: US10318885B2
- Status: Granted
- Kind: B2
- Application number: 14/854,108
- Filed: 2015-09-15
- Published: 2017-03-16
- Granted: 2019-06-11
- Earliest claimed: 2015-09-15
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Joseph N. Kozhaya; Christopher M. Madison; Sridhar Sudarsan
- Web page: https://patentgrep.com/patent/US10318885B2

## CPC

- G06N 20/00: Machine learning
- G06F 16/3331: Query processing
- G06F 16/93: Document management systems
- G06N 5/02: Knowledge representation; Symbolic representation
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Mechanisms are provided for implementing a virtual corpus engine that receives an inquiry to be processed and analyzes the inquiry to extract one or more features of the inquiry. The virtual corpus engine selects a weight matrix associated with a virtual corpus based on the extracted one or more features of the inquiry. The virtual corpus comprises a plurality of actual corpora of information. The weight matrix comprises a separate weight value for each actual corpus in the plurality of actual corpora. The virtual corpus engine processes the inquiry using a set of selected actual corpora selected from the plurality of actual corpora based on the weight values in the weight matrix and receives results of the processing of the inquiry using the set of selected actual corpora. The virtual corpus engine outputs the results of the processing of the inquiry.

## Claims

### Claim 1

A method, in a data processing system comprising a processor and a memory having instructions which are executed by the processor and specifically configure the data processing system to implement a virtual corpus engine, the method comprising:
receiving, by a cognitive computing system associated with the virtual corpus engine of the data processing system, a natural language input from a requestor device;
performing, by the cognitive computing system, natural language processing on the natural language input to generate an inquiry to be processed by the virtual corpus engine;
analyzing, by the virtual corpus engine, the inquiry to extract one or more features of the inquiry;
selecting, by the virtual corpus engine, a weight matrix associated with a virtual corpus based on the extracted one or more features of the inquiry, wherein the virtual corpus is a logical data structure that comprises a plurality of actual corpora of information stored as a plurality of content data structures in one or more data storage devices, and wherein the weight matrix comprises a separate weight value for each actual corpus in the plurality of actual corpora;
selecting, by the virtual corpus engine, a set of actual corpora from the plurality of actual corpora based on the weight values in the weight matrix;
processing, by the virtual corpus engine, the inquiry using the selected set of actual corpora;
receiving, by the virtual corpus engine, results of the processing of the inquiry using the selected set of actual corpora;
processing, by hypothesis generation and evaluation logic of the cognitive computing system, the results of processing the inquiry using the set of selected actual corpora to generate a response to the natural language input; and
outputting, by the cognitive computing system the response to the natural language input to the requestor device.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more features of the inquiry comprise a topic or domain of the inquiry, and wherein selecting the weight matrix associated with the virtual corpus based on the extracted one or more features of the inquiry comprises selecting the weight matrix associated with the virtual corpus based on the topic or domain of the inquiry.

### Claim 3 (depends on 1)

The method of claim 1, wherein the one or more features of the inquiry comprise a topic or domain of the inquiry, and wherein selecting a weight matrix associated with the virtual corpus based on the extracted one or more features of the inquiry comprises:
determining if there is a weight matrix already defined for the topic or domain; and
in response to determining that a weight matrix has not already been defined for the topic or domain, creating a new weight matrix associated with the topic or domain and initializing the weight matrix to an initial set of weight values for each of the actual corpora in the plurality of actual corpora.

### Claim 4 (depends on 1)

The method of claim 1, wherein processing the inquiry using the selected set of actual corpora comprises submitting the inquiry to each of the actual corpora in the selected set of actual corpora and receiving results from each of the corpora in the selected set of actual corpora.

### Claim 5 (depends on 4)

The method of claim 4, wherein outputting the results of the processing comprises:
outputting the results from each of the corpora in the selected set of actual corpora;
receiving, for each result in the results from each of the corpora in the selected set of actual corpora, user feedback rating a correctness of the results obtained from each of the corpora in the selected set of actual corpora; and
modifying, for each corpus in the selected set of actual corpora, a weight value in the selected weight matrix, associated with the corpus in the selected set of actual corpora based on the user feedback associated with the results received from the corpus.

### Claim 6 (depends on 4)

The method of claim 4, wherein outputting the results of the processing comprises:
outputting the results from each of the corpora in the selected set of actual corpora;
comparing each result in the results from each of the corpora in the selected set of actual corpora to a ground truth data structure to identify whether the result matches a correct answer to the inquiry; and
modifying, for each corpus in the selected set of actual corpora, a weight value in the selected weight matrix, associated with the corpus in the selected set of actual corpora based on results of the comparison.

### Claim 7 (depends on 1)

The method of claim 1, wherein there are a plurality of weight matrices, each weight matrix being associated with a different topic or domain.

### Claim 8 (depends on 1)

The method of claim 1, wherein processing, by the virtual corpus engine, the inquiry using the selected set of actual corpora comprises:
evaluating weight values associated with each actual corpus in the plurality of actual corpora against one or more threshold weight values; and
selecting actual corpora from the plurality of actual corpora based on whether or not a weight value associated with the actual corpus meets or exceeds a threshold weight value in the one or more threshold weight values.

### Claim 9 (depends on 8)

The method of claim 8, wherein the one or more threshold weight values comprises a first threshold weight value indicating a trusted corpus and a second threshold weight value indicating an untrusted corpus.

### Claim 10 (depends on 9)

The method of claim 9, wherein selecting actual corpora from the plurality of actual corpora comprises selecting actual corpora whose associated weight values in the weight matrix are equal to or above the second threshold.

### Claim 11

A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program is executed on a computing device and specifically configures the computing device to implement a virtual corpus engine and a cognitive computing system that operates to:
receive, by the cognitive computing system, a natural language input from a requestor device;
perform, by the cognitive computing system, natural language processing on the natural language input to generate an inquiry to be processed by the virtual corpus engine;
analyze, by the virtual corpus engine, the inquiry to extract one or more features of the inquiry;
select, by the virtual corpus engine, a weight matrix associated with a virtual corpus based on the extracted one or more features of the inquiry, wherein the virtual corpus is a logical data structure that comprises a plurality of actual corpora of information stored as a plurality of content data structures in one or more data storage devices, and wherein the weight matrix comprises a separate weight value for each actual corpus in the plurality of actual corpora;
select, by the virtual corpus engine, a set of actual corpora from the plurality of actual corpora based on the weight values in the weight matrix;
process, by the virtual corpus engine, the inquiry using the selected set of actual corpora;
receive, by the virtual corpus engine, results of the processing of the inquiry using the selected set of actual corpora;
process, by hypothesis generation and evaluation logic of the cognitive computing system, the results of processing the inquiry using the selected set of actual corpora to generate a response to the natural language input; and
output, by the cognitive computing system, the response to the natural language input to the requestor device.

### Claim 12 (depends on 11)

The computer program product of claim 11, wherein the one or more features of the inquiry comprise a topic or domain of the inquiry, and wherein selecting the weight matrix associated with the virtual corpus based on the extracted one or more features of the inquiry comprises selecting the weight matrix associated with the virtual corpus based on the topic or domain of the inquiry.

### Claim 13 (depends on 11)

The computer program product of claim 11, wherein the one or more features of the inquiry comprise a topic or domain of the inquiry, and wherein selecting a weight matrix associated with the virtual corpus based on the extracted one or more features of the inquiry comprises:
determining if there is a weight matrix already defined for the topic or domain; and
in response to determining that a weight matrix has not already been defined for the topic or domain, creating a new weight matrix associated with the topic or domain and initializing the weight matrix to an initial set of weight values for each of the actual corpora in the plurality of actual corpora.

### Claim 14 (depends on 11)

The computer program product of claim 11, wherein processing the inquiry using the selected set of actual corpora comprises submitting the inquiry to each of the actual corpora in the selected set of actual corpora and receiving results from each of the corpora in the selected set of actual corpora.

### Claim 15 (depends on 14)

The computer program product of claim 14, wherein outputting the results of the processing comprises:
outputting the results from each of the corpora in the selected set of actual corpora;
receiving, for each result in the results from each of the corpora in the selected set of actual corpora, user feedback rating a correctness of the results obtained from each of the corpora in the selected set of actual corpora; and
modifying, for each corpus in the selected set of actual corpora, a weight value in the selected weight matrix, associated with the corpus in the selected set of actual corpora based on the user feedback associated with the results received from the corpus.

### Claim 16 (depends on 14)

The computer program product of claim 14, wherein outputting the results of the processing comprises:
outputting the results from each of the corpora in the selected set of actual corpora;
comparing each result in the results from each of the corpora in the selected set of actual corpora to a ground truth data structure to identify whether the result matches a correct answer to the inquiry; and
modifying, for each corpus in the selected set of actual corpora, a weight value in the selected weight matrix, associated with the corpus in the selected set of actual corpora based on results of the comparison.

### Claim 17 (depends on 11)

The computer program product of claim 11, wherein there are a plurality of weight matrices, each weight matrix being associated with a different topic or domain.

### Claim 18 (depends on 11)

The computer program product of claim 11, wherein processing the inquiry using the selected set of actual corpora comprises:
evaluating weight values associated with each actual corpus in the plurality of actual corpora against one or more threshold weight values; and
selecting actual corpora from the plurality of actual corpora based on whether or not a weight value associated with the actual corpus meets or exceeds a threshold weight value in the one or more threshold weight values.

### Claim 19 (depends on 18)

The computer program product of claim 18, wherein the one or more threshold weight values comprises a first threshold weight value indicating a trusted corpus and a second threshold weight value indicating an untrusted corpus.

### Claim 20

An apparatus comprising:
at least one processor; and
at least one memory coupled to the at least one processor, wherein the at least one memory comprises instructions which are executed by the at least one processor and specifically configure the apparatus to implement a virtual corpus engine and a cognitive computing system that operate to:
receive, by the cognitive computing system, a natural language input from a requestor device;
perform, by the cognitive computing system, natural language processing on the natural language input to generate an inquiry to be processed by the virtual corpus engine;
analyze, by the virtual corpus engine, the inquiry to extract one or more features of the inquiry;
select, by the virtual corpus engine, a weight matrix associated with a virtual corpus based on the extracted one or more features of the inquiry, wherein the virtual corpus is a logical data structure that comprises a plurality of actual corpora of information stored as a plurality of content data structures in one or more data storage devices, and wherein the weight matrix comprises a separate weight value for each actual corpus in the plurality of actual corpora;
select, by the virtual corpus engine, a set of actual corpora from the plurality of actual corpora based on the weight values in the weight matrix;
process, by the virtual corpus engine, the inquiry using the selected set of actual corpora;
receive, by the virtual corpus engine, results of the processing of the inquiry using the selected set of actual corpora;
process, by hypothesis generation and evaluation logic of the cognitive computing system, the results of processing the inquiry using the selected set of actual corpora to generate a response to the natural language input; and
output, by the cognitive computing system, the response to the natural language input to the requestor device.