# US 2020/0394262 A1 · Natural Language Processing and Candidate Response Evaluation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394262 A1
- Document ID: US20200394262A1
- Status: Published application
- Kind: A1
- Application number: 16/439,220
- Filed: 2019-06-12
- Published: 2020-12-17
- Earliest claimed: 2019-06-12
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Stephen A. Boxwell; Keith G. Frost; Kyle M. Brake; Stanley J. Vernier
- Web page: https://patentgrep.com/patent/US20200394262A1

## CPC

- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 16/3329: Natural language query formulation
- G06F 16/3344: {using natural language analysis}
- G06F 16/90344: {by using string matching techniques}
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06N 5/04: Inference or reasoning models
- G06N 5/046: Forward inferencing; Production systems
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Embodiments relate to an intelligent computer platform to identify and evaluate candidate passages response data in natural language form. A passage is analyzed by applying input tokens against passage tokens. A structure representing the analyzed passage is populated with of matching input and passage tokens. The input and passage tokens are used to assess the leveraged structure. First and second counts of token matching criteria are determined, including a first count of matching token entries and a second count of evaluated token entries. An alignment of the passage to a candidate question is calculated and a ratio of the first and second counts assessed. Data from the passage is returned with the ratio.

## Claims

### Claim 1

A computer implemented method comprising:
representing a natural language passage as a plurality of natural language tokens and analyzing the passage in natural language form, including applying one or more natural language input tokens against one or more passage tokens represented as natural language passage tokens;
generating a structure representing the analyzed passage, the structure including one or more entries identifying matching input and passage tokens;
representing the one or more input tokens in compact form;
representing the one or more passage tokens in compact form;
leveraging the structure and iteratively associating the one or more input tokens and matching passage tokens as an index representation;
assessing the leveraged structure with the compact representation of the tokens with a compact operation, the assessment determining a first count of matching token entries qualified by a closeness criteria and a second count of evaluated token entries qualified by the closeness criteria;
calculating an alignment of the passage to a candidate question represented by the one or more input tokens, including assessing a ratio of the first count and the second count as a confidence value; and
a candidate answer from the passage returned with the confidence value.

### Claim 2 (depends on 1)

The method of claim 1, wherein the compact form includes representing the one or more input tokens as a first bitfield and the one or more passage tokens as a second bitfield.

### Claim 3 (depends on 2)

The method of claim 2, further comprising identifying an index in the structure with matching tokens, and representing the identified index as a third bitfield.

### Claim 4 (depends on 3)

The method of claim 3, further comprising performing a first bitwise operation using the first and second bitfields, including expanding bits set in the third bitfield, wherein the expansion includes setting all bits within a range that match a first criteria.

### Claim 5 (depends on 4)

The method of claim 4, further comprising performing a second bitwise operation using the third bitfield, including discounting when only a single token is found in a multiple token search.

### Claim 6 (depends on 3)

The method of claim 3, further comprising applying a closeness criteria as a qualifier in identifying matching input and passage tokens, and projecting the closeness criteria onto one or more bitfield representations.

### Claim 7

A computer program product for identifying and evaluating candidate passages responses, the computer program product comprising a computer readable storage medium having program code embodied therewith, the program code executable by a processor to:
represent a natural language passage as a plurality of natural language tokens and analyze the passage in natural language form, including apply one or more natural language input tokens against one or more passage tokens represented as natural language passage tokens;
generate a structure representing the analyzed passage, the structure including one or more entries identifying matching input and passage tokens;
represent the one or more input tokens in compact form and to represent the one or more passage tokens in compact form;
leverage the structure and iteratively associate the one or more input tokens and matching passage tokens as an index representation;
determine a first count of matching token entries qualified by a closeness criteria and a second count of evaluated token entries qualified by the closeness criteria; and
calculate an alignment of the passage to a candidate question represented by the one or more input tokens, including assess a ratio of the first count and the second count as a confidence value; and
a candidate answer from the passage returned with the confidence value.

### Claim 8 (depends on 7)

The computer program product of claim 7, wherein the compact form includes representation of the one or more input tokens as a first bitfield and the one or more passage tokens as a second bitfield.

### Claim 9 (depends on 8)

The computer program product of claim 8, further comprising program code to identify an index in the structure with matching tokens, and represent the identified index as a third bitfield.

### Claim 10 (depends on 9)

The computer program product of claim 9, further comprising program code to perform a first bitwise operation using the first and second bitfields, including expand bits set in the third bitfield, wherein the expansion includes setting all bits within a range that match a first criteria.

### Claim 11 (depends on 10)

The computer program product of claim 10, further comprising program code to perform a second bitwise operation using the third bitfield, including discount when only a single token is found in a multiple token search.

### Claim 12 (depends on 9)

The computer program product of claim 9, further comprising program code to apply a closeness criteria as a qualifier in identifying matching input and passage tokens, and project the closeness criteria onto one or more bitfield representations.

### Claim 13

A computer system comprising:
a processing unit operatively coupled to memory;
an artificial intelligence (AI) platform in communication with the processing unit, the AI platform to identify and evaluate candidate passage responses, including:
a natural language (NL) manager to represent a natural language passage as a plurality of natural language tokens and analyze the passage in natural language form, including apply one or more natural language input tokens against one or more passage tokens represented as natural language passage tokens;
a structure manager operatively coupled to the NL manager, the structure manager to generate a structure representing the analyzed passage, the structure including one or more entries identifying matching input and passage tokens;
a token manager to represent the one or more input tokens in compact form and to represent the one or more passage tokens in compact form;
the token manager to leverage the structure and iteratively associate the one or more input tokens and matching passage tokens as an index representation;
the token manager to determine a first count of matching token entries qualified by a closeness criteria and a second count of evaluated token entries qualified by the closeness criteria; and
a director to calculate an alignment of the passage to a candidate question represented by the one or more input tokens, including assess a ratio of the first count and the second count as a confidence value; and
a candidate answer from the passage returned with the confidence value.

### Claim 14 (depends on 13)

The system of claim 13, wherein the compact form includes representation of the one or more input tokens as a first bitfield and the one or more passage tokens as a second bitfield.

### Claim 15 (depends on 14)

The system of claim 14, further comprising the structure manager to identify an index in the structure with matching tokens, and represent the identified index as a third bitfield.

### Claim 16 (depends on 15)

The system of claim 15, further comprising the director to perform a first bitwise operation using the first and second bitfields, including expanding bits set in the third bitfield, wherein the expansion includes setting all bits within a range that match a first criteria.

### Claim 17 (depends on 16)

The system of claim 16, further comprising the director to perform a second bitwise operation using the third bitfield, including discount when only a single token is found in a multiple token search.

### Claim 18 (depends on 15)

The system of claim 15, further comprising the director to apply a closeness criteria as a qualifier in identifying matching input and passage tokens, and project the closeness criteria onto one or more bitfield representations.

### Claim 19

A computer implemented method comprising:
analyzing a passage in natural language form, including identifying one or more passage tokens matching one or more input tokens;
generating a structure representing the analyzed passage, the structure including one or more entries identifying matching input and passage tokens;
representing the one or more input tokens and the one or more passage tokens in compact form;
leveraging the structure and iteratively associating the one or more input tokens and matching passage tokens as an index representation;
assessing the leveraged structure with the compact representation of the tokens with a compact operation, and the assessment determining a first count of matching token entries qualified by a closeness criteria and a second count of evaluated token entries qualified by the closeness criteria;
calculating an alignment of the passage to the one or more input tokens, including assessing a ratio of the first count and the second count as a confidence value; and
data from the passage qualified by the one or more input tokens is returned with the confidence value.

### Claim 20 (depends on 19)

The method of claim 19, further comprising identifying an index in the structure with matching tokens, and representing the identified index as a third bitfield.

### Claim 21 (depends on 20)

The method of claim 20, wherein the compact operation is a bitwise operation, and further comprising performing a first bitwise operation using the compact representation of the input and passage tokens, including expanding bits set in the third bitfield, wherein the expansion includes setting all bits within a range that match a first criteria.

### Claim 22 (depends on 21)

The method of claim 21, further comprising performing a second bitwise operation using the third bitfield, including discounting when only a single token is found in a multiple token search.

### Claim 23

A computer program product for identifying and evaluating candidate passages responses, the computer program product comprising a computer readable storage medium having program code embodied therewith, the program code executable by a processor to:
analyze a passage in natural language form, including identifying one or more passage tokens matching one or more input tokens;
generate a structure representing the analyzed passage, the structure including one or more entries identifying matching input and passage tokens;
represent the one or more input tokens and one or more passage tokens in compact form;
leverage the structure and iteratively associating the one or more input tokens and matching passage tokens as an index representation;
assess the leveraged structure with the compact representation of the token with a compact operation, and the assessment determining a first count of matching token entries qualified by a closeness criteria and a second count of evaluated token entries qualified by the closeness criteria;
calculate an alignment of the passage to the one or more input tokens, including assessing a ratio of the first count and the second count as a confidence value; and
data from the passage qualified by the one or more input tokens is returned with the confidence value.

### Claim 24 (depends on 23)

The method of claim 23, further comprising identifying an index in the structure with matching tokens, and representing the identified index as a bitfield.

### Claim 25 (depends on 24)

The method of claim 24, wherein the compact operation is a bitwise operation, and further comprising performing the bitwise operation, the bitwise operation selected from the group consisting of: a first bitwise operation using the compact representation of the input and passage tokens and a representation of the structure, and a second bitwise operation using the representation of the structure and discounting when only a single token is found in a multiple token search.
