# US 2013/0158982 A1 · Computer-Implemented Systems and Methods for Content Scoring of Spoken Responses

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0158982 A1
- Document ID: US20130158982A1
- Status: Published application
- Kind: A1
- Application number: 13/688,306
- Filed: 2012-11-29
- Published: 2013-06-20
- Earliest claimed: 2011-11-29
- Assignees: Educational Testing Service
- Inventors: Klaus Zechner; Keelan Evanini; Lei Chen; Shasha Xie; Wenting Xiong; Fei Huang; Jana Sukkarieh; Miao Chen
- Web page: https://patentgrep.com/patent/US20130158982A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G09B 19/06: Foreign languages (with audible presentation of material to be studied G09B5/04)
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 2015/088: {Word spotting}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Systems and methods are provided for scoring a non-scripted speech sample. A system includes one or more data processors and one or more computer-readable mediums. The computer-readable mediums are encoded with a non-scripted speech sample data structure, where the non-scripted speech sample data structure includes: a speech sample identifier that identifies a non-scripted speech sample, a content feature extracted from the non-scripted speech sample, and a content-based speech score for the non-scripted speech sample. The computer-readable mediums further include instructions for commanding the one or more data processors to extract the content feature from a set of words automatically recognized in the non-scripted speech sample and to score the non-scripted speech sample by providing the extracted content feature to a scoring model to generate the content-based speech score.

## Claims

### Claim 1

A computer-implemented method of scoring a non-scripted speech sample, comprising:
extracting, using a processing system, a content feature from a set of words automatically recognized in the non-scripted speech sample; and
scoring, using the processing system, the non-scripted speech sample by providing the extracted content feature to a content scoring model to generate a content-based speech score.

### Claim 2 (depends on 1)

The method of claim 1, wherein the content scoring model is built by:
transcribing a set of speech samples from a training corpus;
assigning a score to each speech sample;
partitioning the set of transcribed speech samples of the training corpus into sub-sets, with each of the sub-sets containing speech samples with identical scores; and
building score-level training vectors for each sub-set of responses.

### Claim 3 (depends on 2)

The method of claim 2, wherein the speech samples of the training corpus are transcribed by human transcribers.

### Claim 4 (depends on 2)

The method of claim 2, wherein the speech samples of the training corpus are transcribed by an automated speech recognizer.

### Claim 5 (depends on 2)

The method of claim 2, wherein the score for each speech sample is assigned by a human rater.

### Claim 6 (depends on 2)

The method of claim 2, wherein the score for each speech sample is assigned by an automatic process.

### Claim 7 (depends on 2)

The method of claim 2, wherein the elements of a score-level training vector are a sub-set of words occurring in a plurality of responses of the training corpus with a same score, and where each word is associated with a numerical weight, reflecting a relative importance of that word in the sub-set of responses at the score point associated with the score-level training vector.

### Claim 8 (depends on 1)

The method of claim 1, wherein the content feature of a speech sample is an evaluation vector comprising a sub-set of the words recognized by the automated speech recognition and an associated numerical weight for each word that reflects a relative importance of the word.

### Claim 9 (depends on 8)

The method of claim 8, wherein a similarity metric is used to compare the evaluation vector of a received speech sample with each score-level training vector of the content scoring model to obtain the content score of the speech sample.

### Claim 10 (depends on 9)

The method of claim 9, wherein the similarity metric is an average of a sum of inner products between the evaluation vector and the plurality of score-level training vectors of a score point.

### Claim 11 (depends on 10)

The method of claim 10, wherein a score corresponding to the score point of score-level training vectors with the highest similarity to the evaluation vector is assigned as the score for the speech sample.

### Claim 12 (depends on 10)

The method of claim 10, wherein the similarity between the evaluation vector and the score-level training vectors with the highest score is assigned as the score for the speech sample.

### Claim 13 (depends on 10)

The method of claim 10, wherein the content score of a speech sample is based on a multiple regression model for similarities with each plurality of score-level vectors, and wherein weights of the multiple regression model are trained based on held-out data.

### Claim 14 (depends on 1)

The method of claim 1, wherein the content scoring model includes a content vector analysis model, a latent semantic analysis model, or a pointwise mutual information model.

### Claim 15 (depends on 14)

The method of claim 14, wherein the content vector model comprises vectors of words and associated numerical weights, indicating relative importance of words of the vectors.

### Claim 16 (depends on 14)

The method of claim 14, wherein the latent semantic analysis model comprises singular-value decomposed vectors of words and associated numerical weights, indicating relative importance of words of the vectors.

### Claim 17 (depends on 14)

The method of claim 14, wherein the pointwise mutual information model comprises vectors of words and associated numerical weights, indicating the relative importance of words of the vectors based on a training corpus.

### Claim 18 (depends on 2)

The method of claim 2, wherein building the score-level training vectors further comprises:
comparing a particular word from the set of transcribed words to an ontology data store, wherein the ontology data store comprises a plurality of content hierarchies, wherein a particular content hierarchy includes a particular concept identifier, and wherein the particular content hierarchy further includes a plurality of words associated with the particular concept identifier; and
using the particular concept identifier as the element in the score-level training vector when the particular transcribed word matches one of the plurality of words associated with the particular concept identifier.

### Claim 19 (depends on 8)

The method of claim 8, wherein building the evaluation vector further comprises:
comparing a particular word from the set of recognized words to an ontology data store, wherein the ontology data store comprises a plurality of content hierarchies, wherein a particular content hierarchy includes a particular concept identifier, and wherein the particular content hierarchy further includes a plurality of words associated with the particular concept identifier; and
using the particular concept identifier as the element in the evaluation vector when the particular recognized word matches one of the plurality of words associated with the particular concept identifier

### Claim 20 (depends on 8)

The method of claim 8, wherein scoring the non-scripted speech sample includes identifying whether the concept identifier of a word in the recognized speech sample is present in the score-level training vector associated with a particular score point;
wherein when the concept identifier is not present:
determining a similarity score for the concept identifier based on a recognized word in the speech sample and each of the concept identifiers in a score-level training vector associated with the particular score point; and:
assigning the average of the weights associated with the concept identifiers that have a highest n similarity scores as a weight for the concept identifier not found in the score-level training vector.

### Claim 21 (depends on 20)

The method of claim 20, wherein the similarity score is based on an edge-based similarity measurement for the concept identifier based on a recognized word of the speech sample and the concept identifiers of the score-level training vector associated with the particular score point.

### Claim 22 (depends on 8)

The method of claim 8, wherein extracting the content feature further comprises:
generating a set of expected answers that are relevant to the prompt;
automatically extracting possible candidate answer phrases from the recognized word sequence;
comparing a particular recognized phrase to the set of expected answers.

### Claim 23 (depends on 22)

The method of claim 22 wherein a phrase chunker or a named entity recognizer is used to extract possible candidate answer phrases.

### Claim 24 (depends on 22)

The method of claim 22, wherein comparing a particular recognized phrase to the set of expected answers further comprises:
querying a lexical database with word co-occurrence statistics;
computing a similarity metric between a particular recognized phrase and each of the expected answers using the co-occurrence statistics;
computing a content feature using a weighted sum of the similarity metrics.

### Claim 25 (depends on 8)

The method of claim 8, wherein extracting the content feature further comprises:
using a dimensionality reduction method to process the automatically recognized words.

### Claim 26 (depends on 25)

The method of claim 25 wherein Latent Semantic Analysis or a Hidden Markov Model are used as the dimensionality reduction method.

### Claim 27 (depends on 8)

The method of claim 8, wherein the content feature is generated by a process comprising:
segmenting the speech sample into multiple parts;
extracting information from each part, based on the words recognized;
combining the information extracted on each segment to a sub-feature; and
combining all sub-features to generate the content feature for the speech sample.

### Claim 28 (depends on 27)

The method of claim 27, wherein the segmentation is performed using a Hidden Markov Model trained on annotated data.

### Claim 29 (depends on 27)

The method of claim 27, wherein the information extracted for each part includes measures of distance to adjacent segments, keywords matching a training vector of keywords, or sequences of words matching sequences from a training corpus.

### Claim 30 (depends on 1)

The method of claim 1 further comprising:
receiving the non-scripted speech sample; and
recognizing the set of words using automatic speech recognition, wherein a confidence metric is associated with each recognized word, and wherein the scoring of the non-scripted speech sample is based on the confidence metric.

### Claim 31

A computer-implemented system for scoring a non-scripted speech sample, comprising:
one or more data processors;
one or more computer-readable mediums encoded with:
a non-scripted speech sample data structure, wherein the non-scripted speech sample data structure includes:
a speech sample identifier that identifies a non-scripted speech sample;
a content feature extracted from the non-scripted speech sample; and
a content-based speech score for the non-scripted speech sample;
instructions for commanding the one or more data processors to extract the content feature from a set of words automatically recognized in the non-scripted speech sample and to score the non-scripted speech sample by providing the extracted content feature to a scoring model to generate the content-based speech score.

### Claim 32 (depends on 30)

The system of claim 30, wherein the data structure is a data base.