# US 2012/0253783 A1 · Optimization of natural language processing system based on conditional output quality at risk

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0253783 A1
- Document ID: US20120253783A1
- Status: Published application
- Kind: A1
- Application number: 13/073,255
- Filed: 2011-03-28
- Published: 2012-10-04
- Earliest claimed: 2011-03-28
- Assignees: International Business Machines Corporation
- Inventors: Vittorio Castelli; David Nahamoo; Bing Zhao
- Web page: https://patentgrep.com/patent/US20120253783A1

## CPC

- G06F 40/44: Statistical methods, e.g. probability models
- G06F 40/51: Translation evaluation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Techniques are disclosed for optimizing results output by a natural language processing system. For example, a method comprises optimizing one or more parameters of a natural language processing system so as to improve a measure of quality of an output of the natural language processing system for a first type of data processed by the natural language processing system while maintaining a given measure of quality of an output of the natural language processing system for a second type of data processed by the natural language processing system. For example, the first type of data may have a substantive complexity that is greater than that of the second type of data. Thus, when the natural language processing system is a machine translation system, use of a conditional value at risk metric for the translation quality provides for a high quality output of the machine translation system for data of a high substantive complexity (for sophisticated users) while maintaining an average quality output for average data (for average users).

## Claims

### Claim 1

A method, comprising:
optimizing one or more parameters of a natural language processing system so as to improve a measure of quality of an output of the natural language processing system for a first type of data processed by the natural language processing system while maintaining a given measure of quality of an output of the natural language processing system for a second type of data processed by the natural language processing system.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first type of data has a substantive complexity that is greater than a substantive complexity of the second type of data.

### Claim 3 (depends on 2)

The method of claim 2, wherein the first type of data is presented to the natural language processing system by a first type of user and the second type of data is presented to the natural language processing system by a second type of user.

### Claim 4 (depends on 1)

The method of claim 1, wherein the optimizing step further comprises computing a conditional value at risk metric.

### Claim 5 (depends on 4)

The method of claim 4, wherein the conditional value at risk metric represents a level of risk aversion for one or more users of the natural language processing system associated with the quality of the output of the natural language processing system.

### Claim 6 (depends on 4)

The method of claim 4, wherein the conditional value at risk computing step further comprises obtaining one or more input data sets.

### Claim 7 (depends on 6)

The method of claim 6, wherein the conditional value at risk computing step further comprises computing initial processing results from the one or more input data sets.

### Claim 8 (depends on 7)

The method of claim 7, wherein the initial processing results computing step further comprises computing one or more error metric scores for the one or more input data sets.

### Claim 9 (depends on 8)

The method of claim 8, wherein the initial processing results computing step further comprises ranking and sorting the one or more input data sets based on the one or more error scores.

### Claim 10 (depends on 9)

The method of claim 9, wherein the initial processing results computing step further comprises computing tail scores based on given percentiles for the one or more ranked and sorted input data sets.

### Claim 11 (depends on 7)

The method of claim 7, wherein the initial processing results computing step further comprises obtaining one or more baseline output data sets from the natural language processing system for the one or more input data sets.

### Claim 12 (depends on 11)

The method of claim 11, wherein the initial processing results computing step further comprises obtaining user feedback on the one or more baseline output data sets.

### Claim 13 (depends on 12)

The method of claim 12, wherein the initial processing results computing step further comprises ranking the one or more baseline output data sets based on the user feedback for given percentiles.

### Claim 14 (depends on 7)

The method of claim 7, wherein the conditional value at risk computing step further comprises computing a risk index from at least a portion of the initial processing results.

### Claim 15 (depends on 14)

The method of claim 14, wherein the conditional value at risk computing step further comprises generating one or more scores based on the risk index.

### Claim 16 (depends on 15)

The method of claim 15, wherein the optimizing step further comprises using the one or more scores to improve the measure of quality of the output of the natural language processing system for subsequently presented data of the first type while maintaining the given measure of quality of the output for subsequently presented data of the second type.

### Claim 17 (depends on 1)

The method of claim 1, wherein the optimizing step further comprises:
computing a distribution of the quality of the output across sophisticated and other users of the natural language processing system;
computing a conditional quality at risk from the distribution; and
utilizing a lower-dimension line search to optimize the one or more parameters of the natural language processing system for the computed conditional quality at risk.

### Claim 18 (depends on 1)

The method of claim 1, wherein the natural language processing system comprises a machine translation system, and the output is a translation of an input document into a different language.

### Claim 19

Apparatus, comprising:
a memory; and
a processor device operatively coupled to the memory and configured to optimize one or more parameters of a natural language processing system so as to improve a measure of quality of an output of the natural language processing system for a first type of data processed by the natural language processing system while maintaining a given measure of quality of an output of the natural language processing system for a second type of data processed by the natural language processing system.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the first type of data has a substantive complexity that is greater than a substantive complexity of the second type of data.

### Claim 21 (depends on 20)

The apparatus of claim 20, wherein the first type of data is presented to the natural language processing system by a first type of user and the second type of data is presented to the natural language processing system by a second type of user.

### Claim 22 (depends on 19)

The apparatus of claim 19, wherein the processor device is further configured to perform the optimization by computing a conditional value at risk metric.

### Claim 23 (depends on 22)

The apparatus of claim 22, wherein the conditional value at risk metric represents a level of risk aversion for one or more users of the natural language processing system associated with the quality of the output of the natural language processing system.

### Claim 24 (depends on 19)

The apparatus of claim 19, wherein the natural language processing system comprises a machine translation system, and the output is a translation of an input document into a different language.

### Claim 25

An article of manufacture comprising a computer readable storage medium having tangibly embodied thereon computer readable program code which, when executed, causes a processor device to optimize one or more parameters of a natural language processing system so as to improve a measure of quality of an output of the natural language processing system for a first type of data processed by the natural language processing system while maintaining a given measure of quality of an output of the natural language processing system for a second type of data processed by the natural language processing system.
