# US 11,114,088 B2 · Adaptive self-trained computer engines with associated databases and methods of use thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,114,088 B2
- Document ID: US11114088B2
- Status: Granted
- Kind: B2
- Application number: 15/952,802
- Filed: 2018-04-13
- Published: 2018-10-18
- Granted: 2021-09-07
- Earliest claimed: 2017-04-03
- Assignees: Green Key Technologies, Inc.
- Applicants: Green Key Technologies, Inc.
- Inventors: Tejas Shastry; Anthony Tassone; Patrick Kuca; Svyatoslav Vergun
- Web page: https://patentgrep.com/patent/US11114088B2

## CPC

- G10L 15/063: {Training}
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/08: Speech classification or search
- G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 15/28: Constructional details of speech recognition systems
- G10L 17/04: Training, enrolment or model building
- G10L 25/18: the extracted parameters being spectral information of each sub-band
- G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units}
- G10L 2015/088: {Word spotting}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

In some embodiments, the present invention provides for an exemplary computer system which includes at least the following components: an adaptive self-trained computer engine programmed, during a training stage, to electronically receive an initial speech audio data generated by a microphone of a computing device; dynamically segment the initial speech audio data and the corresponding initial text into a plurality of user phonemes; dynamically associate a plurality of first timestamps with the plurality of user-specific subject-specific phonemes; and, during a transcription stage, electronically receive to-be-transcribed speech audio data of at least one user; dynamically split the to-be transcribed speech audio data into a plurality of to-be-transcribed speech audio segments; dynamically assigning each timestamped to-be-transcribed speech audio segment to a particular core of the multi-core processor; and dynamically transcribing, in parallel, the plurality of timestamped to-be-transcribed speech audio segments based on the user-specific subject-specific speech training model.

## Claims

### Claim 1

A computer system, comprising:
a processor operable:
to differentiate between multiple speakers in an audio stream of speech based in part on frequencies in the audio stream, wherein said audio stream of speech comprises audible speech of at least a first speaker and a second speaker,
to convert the audio stream into text, and
to generate time stamps in the audio stream to associate the text with the audio stream; and
a machine learning module implemented by one or more processors:
to access pre-learned phonemes,
to identify the first speaker in the audio stream based on the pre-learned phonemes,
to locate a portion of the text associated with the first speaker based on the time stamps,
to segment the text associated with the first speaker into text phonemes,
to correct the text associated with the first speaker in real-time by comparing the text phonemes with phonetically-similar letter pairs of the pre-learned phonemes and applying one or more filters to the text to generate a clean transcript, and
to execute a transaction based on the clean transcript.

### Claim 2 (depends on 1)

The computer system of claim 1, wherein:
the machine learning module, during a training stage, is further operable:
to process speech of the first speaker,
to learn phonemes of the first speaker,
to build a profile of the first speaker in a database, and
to store the phonemes of the first speaker as the pre-learned phonemes of the first speaker in the profile of the first speaker of the database.

### Claim 3 (depends on 2)

The computer system of claim 2, wherein:
the processor is a multi-core processor operable to segment the audio stream into a plurality of speech portions, to process the speech portions through different cores in the multi-core processor at about a same time.

### Claim 4 (depends on 2)

The computer system of claim 2, further comprising:
a user interface configured with text inputs,
wherein the machine learning module is further operable to auto-fill the corrected text associated with the first speaker in at least one of the text inputs of the user interface as the first speaker speaks to the computing system.

### Claim 5 (depends on 4)

The computer system of claim 4, wherein:
the machine learning module is further operable to process a correction audibly made by the first speaker, and to auto-fill the audibly made correction in the at least one text input of the user interface.

### Claim 6 (depends on 2)

The computer system of claim 2, wherein:
at least a portion of the pre-learned phonemes comprises subject specific phonemes.

### Claim 7 (depends on 2)

The computer system of claim 2, wherein:
the machine learning module is further operable to remove duplicated words in the text associated with the first speaker.

### Claim 8

A method, comprising:
differentiating between multiple speakers in an audio stream of speech based in part on frequencies in the audio stream, wherein said audio stream of speech comprises audible speech of at least a first speaker and a second speaker;
converting the audio stream into text;
generating time stamps in the audio stream to associate the text with the audio stream;
accessing pre-learned phonemes;
identifying the first speaker in the audio stream based on the pre-learned phonemes;
locating a portion of the text associated with the first speaker based on the time stamps;
segmenting the text associated with the first speaker into text phonemes;
correcting the text associated with the first speaker in real-time by comparing the text phonemes with phonetically-similar letter pairs of the pre-learned phonemes and applying one or more filters to the text to generate a clean transcript; and
executing a transaction based on the clean transcript.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
during a training stage, processing speech of the first speaker;
learning phonemes of the first speaker;
building a profile of the first speaker in a database; and
storing the phonemes of the first speaker as the pre-learned phonemes of the first speaker in the profile of the first speaker of the database.

### Claim 10 (depends on 8)

The method of claim 8, further comprising:
segmenting the audio stream into a plurality of speech portions; and
processing the speech portions through different cores in a multi-core processor at about a same time.

### Claim 11 (depends on 8)

The method of claim 8, further comprising:
auto-filling the corrected text associated with the first speaker in at least one of the text inputs of a user interface as the first speaker speaks.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
processing a correction audibly made by the first speaker; and
auto-filling the audibly made correction in the at least one text input of the user interface.

### Claim 13 (depends on 8)

The method of claim 8, wherein:
at least a portion of the pre-learned phonemes comprises subject specific phonemes.

### Claim 14 (depends on 8)

The method of claim 8, further comprising:
removing duplicated words in the text associated with the first speaker.

### Claim 15

A non-transitory computer readable medium comprising instructions that, when executed by a multi-core processor, direct the multi-core processor to:
differentiate between multiple speakers in an audio stream of speech based in part on frequencies in the audio stream, wherein said audio stream of speech comprises audible speech of at least a first speaker and a second speaker;
convert the audio stream into text;
generate time stamps in the audio stream to associate the text with the audio stream;
access pre-learned phonemes; identify the first speaker in the audio stream based on the pre-learned phonemes;
locate a portion of the text associated with the first speaker based on the time stamps;
segment the text associated with the first speaker into text phonemes;
correct the text associated with the first speaker in real-time by comparing the text phonemes with phonetically-similar letter pairs of the pre-learned phonemes and applying one or more filters to the text to generate a clean transcript; and
execute a transaction based on the clean transcript.

### Claim 16 (depends on 15)

The computer readable medium of claim 15, further comprising instructions that direct the multi-core processor to:
during a training stage, process speech of the first speaker;
learn phonemes of the first speaker;
build a profile of the first speaker in a database; and
store the phonemes of the first speaker as the pre-learned phonemes of the first speaker in the profile of the first speaker of the database.

### Claim 17 (depends on 15)

The computer readable medium of claim 15, further comprising instructions that direct the multi-core processor to:
segment the audio stream into a plurality of speech portions; and
process the speech portions through different cores of the multi-core processor at about a same time.

### Claim 18 (depends on 15)

The computer readable medium of claim 15, further comprising instructions that direct the multi-core processor to:
auto-fill the corrected text associated with the first speaker in at least one of the text inputs of a user interface as the first speaker speaks.

### Claim 19 (depends on 18)

The computer readable medium of claim 18, further comprising instructions that direct the multi-core processor to:
process a correction audibly made by the first speaker; and
auto-fill the audibly made correction in the at least one text input of the user interface.

### Claim 20 (depends on 16)

The computer readable medium of claim 16, further comprising instructions that direct the multi-core processor to:
remove duplicated words in the text associated with the first speaker.