# US 2022/0012411 A1 · Performance characteristics of cartridge artifacts over text pattern constructs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0012411 A1
- Document ID: US20220012411A1
- Status: Published application
- Kind: A1
- Application number: 16/925,537
- Filed: 2020-07-10
- Published: 2022-01-13
- Earliest claimed: 2020-07-10
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: ISHRAT FATMA; Sandhya Nayak; Scott Carrier
- Web page: https://patentgrep.com/patent/US20220012411A1

## CPC

- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06F 40/279: Recognition of textual entities
- G06F 16/3344: {using natural language analysis}
- G06F 16/355: Creation or modification of classes or clusters
- G06F 40/177: of tables; using ruled lines
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Embodiments of the present invention are directed to evaluating the performance characteristics of annotator configurations against text pattern constructs in unstructured text. In a non-limiting embodiment of the invention, unstructured text is received by a processor. A text pattern construct is identified in the unstructured text and a first performance characteristic of an annotator is determined based on the text pattern construct. The text pattern construct is converted to a natural language text and a second performance characteristic of the annotator is determined based on the natural language text. A delta is determined between the first performance characteristic and the second performance characteristic. An alternative annotator configuration is identified for a portion of the unstructured text comprising the text pattern construct.

## Claims

### Claim 1

A computer-implemented method for evaluating the performance characteristics of annotator configurations against text pattern constructs in unstructured text, the computer-implemented method comprising:
receiving, by a processor, unstructured text;
identifying, by the processor, a text pattern construct in the unstructured text;
determining, by the processor, a first performance characteristic of an annotator based on the text pattern construct;
converting, by the processor, the text pattern construct to a natural language text;
determining, by the processor, a second performance characteristic of the annotator based on the natural language text;
determining, by the processor, a delta between the first performance characteristic and the second performance characteristic; and
identifying, by the processor, an alternative annotator configuration for a portion of the unstructured text comprising the text pattern construct.

### Claim 2 (depends on 1)

The method of claim 1, wherein the alternative annotator configuration comprises an improved annotation accuracy with respect to the text pattern construct.

### Claim 3 (depends on 1)

The method of claim 1, wherein the text pattern construct comprises at least one of a comma-separated list, a bullet point, and a table.

### Claim 4 (depends on 1)

The method of claim 1, wherein receiving the unstructured text comprises receiving, from a client, a natural language processing (NLP) request comprising a healthcare document comprising the unstructured text.

### Claim 5 (depends on 4)

The method of claim 4 further comprising generating an alert to the client, the alert identifying the alternative annotator configuration.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first performance characteristic comprises a first annotation output and the second performance characteristic comprises a second annotation output.

### Claim 7 (depends on 1)

The method of claim 1, wherein identifying the alternative annotator configuration comprises searching an annotation database for annotator performance metadata against the text pattern construct.

### Claim 8 (depends on 7)

The method of claim 7, wherein identifying the alternative annotator configuration further comprises selecting the alternative annotator configuration from a set of available annotator configurations within the annotation database that comprises a highest annotation accuracy with respect to the text pattern construct.

### Claim 9

A system comprising:
one or more processors for executing computer readable instructions, the computer readable instructions controlling the one or more processors to perform operations comprising:
receiving unstructured text;
identifying a text pattern construct in the unstructured text;
determining a first performance characteristic of an annotator based on the text pattern construct;
converting the text pattern construct to a natural language text;
determining a second performance characteristic of the annotator based on the natural language text;
determining a delta between the first performance characteristic and the second performance characteristic; and
identifying an alternative annotator configuration for a portion of the unstructured text comprising the text pattern construct.

### Claim 10 (depends on 9)

The system of claim 9, wherein the alternative annotator configuration comprises an improved annotation accuracy with respect to the text pattern construct.

### Claim 11 (depends on 9)

The system of claim 9, wherein the text pattern construct comprises at least one of a comma-separated list, a bullet point, and a table.

### Claim 12 (depends on 9)

The system of claim 9, wherein receiving the unstructured text comprises receiving, from a client, a natural language processing (NLP) request comprising a healthcare document comprising the unstructured text.

### Claim 13 (depends on 12)

The system of claim 12 further comprising generating an alert to the client, the alert identifying the alternative annotator configuration.

### Claim 14 (depends on 9)

The system of claim 9, wherein the first performance characteristic comprises a first annotation output and the second performance characteristic comprises a second annotation output.

### Claim 15 (depends on 9)

The system of claim 9, wherein identifying the alternative annotator configuration comprises searching an annotation database for annotator performance metadata against the text pattern construct.

### Claim 16 (depends on 15)

The system of claim 15, wherein identifying the alternative annotator configuration further comprises selecting the alternative annotator configuration from a set of available annotator configurations within the annotation database that comprises a highest annotation accuracy with respect to the text pattern construct.

### Claim 17

A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by one or more processors to cause the one or more processors to perform operations comprising:
receiving unstructured text;
identifying a text pattern construct in the unstructured text;
determining a first performance characteristic of an annotator based on the text pattern construct;
converting the text pattern construct to a natural language text;
determining a second performance characteristic of the annotator based on the natural language text;
determining a delta between the first performance characteristic and the second performance characteristic; and
identifying an alternative annotator configuration for a portion of the unstructured text comprising the text pattern construct.

### Claim 18 (depends on 17)

The computer program product of claim 17, wherein the first performance characteristic comprises a first annotation output and the second performance characteristic comprises a second annotation output.

### Claim 19 (depends on 17)

The computer program product of claim 17, wherein identifying the alternative annotator configuration comprises searching an annotation database for annotator performance metadata against the text pattern construct.

### Claim 20 (depends on 19)

The computer program product of claim 19, wherein identifying the alternative annotator configuration further comprises selecting the alternative annotator configuration from a set of available annotator configurations within the annotation database that comprises a highest annotation accuracy with respect to the text pattern construct.

### Claim 21

A computer-implemented method comprising:
receiving, by a processor, a natural language processing (NLP) request comprising unstructured text;
identifying, by the processor, a set of text pattern constructs in the unstructured text, the set of text pattern constructs comprising a first text pattern construct and a second text pattern construct;
determining, by the processor, a first annotator configuration for a first portion of the unstructured text comprising the first text pattern construct;
determining, by the processor, a second annotator configuration for a second portion of the unstructured text comprising the second text pattern construct; and
generating, by the processor, an NLP response for the unstructured text, the NLP response comprising a first annotation of the first portion of the unstructured text by the first annotator configuration and a second annotation of the second portion of the unstructured text by the second annotator configuration.

### Claim 22 (depends on 21)

The method of claim 21, wherein determining the first annotator configuration and the second annotator configuration comprises searching an annotation database for annotator performance metadata against the first text pattern construct and the second text pattern construct.

### Claim 23 (depends on 21)

The method of claim 21, wherein the first text pattern construct comprises one of a comma-separated list, a bullet point, and a table, wherein the second text pattern construct comprises one of a comma-separated list, a bullet point, and a table, and wherein the first text pattern construct and the second text pattern construct are not the same.

### Claim 24

A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by one or more processors to cause the one or more processors to perform operations comprising:
receiving, by a processor, a natural language processing (NLP) request comprising unstructured text;
identifying, by the processor, a set of text pattern constructs in the unstructured text, the set of text pattern constructs comprising a first text pattern construct and a second text pattern construct;
determining, by the processor, a first annotator configuration for a first portion of the unstructured text comprising the first text pattern construct;
determining, by the processor, a second annotator configuration for a second portion of the unstructured text comprising the second text pattern construct; and
generating, by the processor, an NLP response for the unstructured text, the NLP response comprising a first annotation of the first portion of the unstructured text by the first annotator configuration and a second annotation of the second portion of the unstructured text by the second annotator configuration.

### Claim 25 (depends on 24)

The computer program product of claim 24, wherein determining the first annotator configuration and the second annotator configuration comprises searching an annotation database for annotator performance metadata against the first text pattern construct and the second text pattern construct.
