# US 10,504,522 B2 · System and method for validating natural language content using crowdsourced validation jobs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,504,522 B2
- Document ID: US10504522B2
- Status: Granted
- Kind: B2
- Application number: 15/924,863
- Filed: 2018-03-19
- Published: 2018-09-27
- Granted: 2019-12-10
- Earliest claimed: 2015-09-07
- Assignees: VoiceBox Technologies, Inc.
- Applicants: Voicebox Technologies Corporation
- Inventors: Spencer John Rothwell; Daniela Braga; Ahmad Khamis Elshenawy; Stephen Steele Carter
- Web page: https://patentgrep.com/patent/US10504522B2

## CPC

- G06Q 30/02: Marketing; Price estimation or determination; Fundraising
- G06F 40/226: Validation
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06Q 10/40: Business processes related to social networking or social networking services
- G10L 15/01: Assessment or evaluation of speech recognition systems
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Systems and methods of validating transcriptions of natural language content using crowdsourced validation jobs are provided herein. In various implementations, a transcription pair comprising natural language content and text corresponding to a transcription of the natural language content may be gathered. A group of validation devices may be selected for reviewing the transcription pair. A crowdsourced validation job may be created for the group of validation devices. The crowdsourced validation job may be provided to the group of validation devices. One or more votes representing whether or not the text accurately represents the natural language content may be received from the group of validation devices. Based on the one or more votes received, the transcription pair may be stored in a validated transcription library, which may be used to process end-user voice data.

## Claims

### Claim 1

A computer-implemented method, the method being implemented in a computer system having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, program the computer system to perform the method, the method comprising:
obtaining, by the computer system, a transcription pair comprising natural language content and text, wherein the natural language content comprises audio content received from one or more audio input components, and wherein the text corresponds to a transcription of the natural language content;
creating, by the computer system, a first crowdsourced validation job to be performed at one or more first validation devices, the first crowdsourced validation job comprising first instructions for a crowd user to provide an indication of an accuracy of the transcription of the natural language content;
causing, by the computer system, the transcription pair and the first crowdsourced validation job to be provided to the one or more first validation devices;
receiving, by the computer system, at least a first response from at least a first validation device from among the one or more first validation devices, wherein the first response includes a first indication of an accuracy of the transcription of the natural language content; and
storing, by the computer system, the transcription pair in a validated transcription library based, at least in part, on the first response.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
assigning, by the computer system, a validation score to the transcription pair based, at least in part, on the first response received from the first validation device with respect to the transcription pair, wherein the transcription pair is stored in the validated transcription library based on the validation score.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
identifying, by the computer system, a confidence score of each of the one or more first validation devices, the confidence score representing confidence in the response from the each of the one or more first validation devices; and
updating, by the computer system, the validation score with respect to the transcription pair using the confidence score.

### Claim 4 (depends on 2)

The method of claim 2, further comprising storing the transcription pair in the validated transcription library if the validation score of the transcription pair exceeds a validation threshold.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
determining, by the computer system, whether or not the one or more first validation devices agreed the text is an accurate transcription of the natural language content; and
determining, by the computer system, whether to provide the transcription pair to one or more second validation devices.

### Claim 6 (depends on 5)

The method of claim 5, wherein if the one or more first validation devices did not agree the text is an accurate transcription of the natural language content, the method further comprises:
causing, by the computer system, the transcription pair to be provided to the one or more second validation devices;
receiving, by the computer system, at least a second response from at least a second validation device from among the one or more second validation devices, wherein the second response includes a second indication of an accuracy of the transcription of the natural language content; and
storing, by the computer system, the transcription pair in the validated transcription library based, at least in part, on the second response.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first instructions configure the one or more first validation devices to provide the natural language content and the text in a survey on a mobile application on the one or more first validation devices.

### Claim 8 (depends on 7)

The method of claim 7, wherein the survey includes one or more tasks configured to assist engagement of first validators associated with the one or more first validation devices when responding.

### Claim 9 (depends on 7)

The method of claim 7, wherein the survey includes one or more of: test questions, captions that are not machine-readable, and audio that is not understandable to machines.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
identifying, by the computer system, a plurality of transcription devices performing crowdsourced transcription jobs; and
selecting, by the computer system, the one or more first validation devices from the plurality of transcription devices.

### Claim 11 (depends on 1)

The method of claim 1, wherein the one or more first validation devices is two validation devices.

### Claim 12 (depends on 1)

The method of claim 1, further comprising using the transcription pair from the validated transcription library to transcribe end-user voice data in real-time.

### Claim 13 (depends on 1)

The method of claim 1, wherein causing the first crowdsourced validation job to be provided to the one or more first validation devices comprises:
causing, by the computer system, the first crowdsourced validation job to be provided to the one or more first validation devices in a validator application.

### Claim 14 (depends on 1)

The method of claim 1, wherein causing the first crowdsourced validation job to be provided to the one or more first validation devices comprises:
causing, by the computer system, a message that includes a direct selectable link to an interface to be provided to the one or more first validation devices, wherein the interface includes the first crowdsourced validation job.

### Claim 15

A system comprising:
one or more physical processors programmed with one or more computer program instructions which, when executed, program the one or more physical processors to:
gather a transcription pair comprising natural language content and text, wherein the natural language content comprises audio content received from one or more audio input components, and wherein the text corresponds to a transcription of the natural language content;
create a first crowdsourced validation job to be performed at one or more first validation devices, the first crowdsourced validation job comprising first instructions for a crowd user to provide an indication of an accuracy of the transcription of the natural language content;
cause the transcription pair and the first crowdsourced validation job to be provided to the one or more first validation devices;
receive at least a first response from at least a first validation device from among the one or more first validation devices, wherein the first response includes a first indication of an accuracy of the transcription of the natural language content; and
store the transcription pair in a validated transcription library based, at least in part, on the first response.

### Claim 16 (depends on 15)

The system of claim 15, wherein the one or more physical processors are further programmed to:
assign a validation score to the transcription pair based, at least in part, on the first response received from the first validation device with respect to the transcription pair, wherein the transcription pair is stored in the validated transcription library based on the validation score.

### Claim 17 (depends on 16)

The system of claim 16, wherein the one or more physical processors are further programmed to:
identify a confidence score of each of the one or more first validation devices, the confidence score representing confidence in the response from the each of the one or more first validation devices; and
update the validation score with respect to the transcription pair using the confidence score.

### Claim 18 (depends on 16)

The system of claim 16, further comprising storing the transcription pair in the validated transcription library if the validation score of the transcription pair exceeds a validation threshold.

### Claim 19 (depends on 15)

The system of claim 15, wherein the one or more physical processors are further programmed to:
determine whether or not the one or more first validation devices agreed the text is an accurate transcription of the natural language content; and
determine whether to provide the transcription pair to one or more second validation devices.

### Claim 20 (depends on 19)

The system of claim 19, wherein if the one or more first validation devices did not agree the text is an accurate transcription of the natural language content, the one or more processors are further programmed to:
cause the transcription pair to be provided to the one or more second validation devices;
receive at least a second response from at least a second validation device from among the one or more second validation devices, wherein the second response includes a second indication of an accuracy of the transcription of the natural language content; and
store the transcription pair in the validated transcription library based, at least in part, on the second response.

### Claim 21 (depends on 15)

The system of claim 15, wherein the first instructions configure the one or more first validation devices to provide the natural language content and the text in a survey on a mobile application on the one or more first validation devices.

### Claim 22 (depends on 21)

The system of claim 21, wherein the survey includes one or more tasks configured to assist engagement of first validators associated with the one or more first validation devices when responding.

### Claim 23 (depends on 21)

The system of claim 21, wherein the survey includes one or more of: test questions, captions that are not machine-readable, and audio that is not understandable to machines.

### Claim 24 (depends on 15)

The system of claim 15, wherein the one or more physical processors are further programmed to:
identify a plurality of transcription devices performing crowdsourced transcription jobs; and
select the one or more first validation devices from the plurality of transcription devices.

### Claim 25 (depends on 15)

The system of claim 15, wherein the one or more first validation devices is two validation devices.

### Claim 26 (depends on 15)

The system of claim 15, further comprising using the transcription pair from the validated transcription library to transcribe end-user voice data in real-time.

### Claim 27 (depends on 15)

The system of claim 15, wherein to cause the first crowdsourced validation job to be provided to the one or more first validation devices, the one or more physical processors are further programmed to:
cause the first crowdsourced validation job to be provided to the one or more first validation devices in a validator application.

### Claim 28 (depends on 15)

The system of claim 15, wherein to cause the first crowdsourced validation job to be provided to the one or more first validation devices, the one or more physical processors are further programmed to:
cause a message that includes a direct selectable link to an interface to be provided to the one or more first validation devices, wherein the interface includes the first crowdsourced validation job.