# US 10,242,669 B1 · Enhanced transcription of audio data with punctuation markings based on silence durations > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,242,669 B1 - Document ID: US10242669B1 - Status: Granted - Kind: B1 - Application number: 16/057,627 - Filed: 2018-08-07 - Granted: 2019-03-26 - Earliest claimed: 2018-08-07 - Assignees: Repnow Inc. - Applicants: Repnow Inc. - Inventors: Michael Sandler; Fredrick Korfin; Alexander Boone; Stanfell Boone - Web page: https://patentgrep.com/patent/US10242669B1 ## CPC - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G10L 15/19: Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G10L 25/48: specially adapted for particular use - G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) - G10L 2015/088: {Word spotting} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A method and process for using subsequent analysis of an audio recording to improve the punctuation of a transcribed audio file that has been processed by a transcription (speech to text) service or process. Features are disclosed for assessing audio data and transcription information to enhance the punctuation or capitalization of the words detected therein. The timing information received from a service transcribing the audio may be compared with audio analysis of the original audio data to identify words likely to be associated with punctuation or capitalization. The features generate a transcript with improved semantic and syntactic accuracy over the transcript received from the transcription service. ## Claims ### Claim 1 A computer-implemented method comprising: under control of one or more processing devices, receiving an audio file including audio data for spoken words; transmitting, to a transcription device, the audio file; receiving, from the transcription device, transcription information, wherein the transcription information includes: (i) a transcript of the spoken words, and (ii) word timings identifying a position where the transcription device identified each of the spoken words in the audio data; detecting an audio silence in the audio data; associating the audio silence with a word timing for a spoken word included in the spoken words based on a matching of a start time of the audio silence with an end time of the spoken word, wherein the start time of the audio silence is determined from the audio data, and wherein the end time of the spoken word is included in the word timings received from the transcription device; removing punctuation characters from the transcript of the spoken words; and generating a punctuated transcript by including a punctuation mark adjacent to the spoken word associated with the audio silence. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein detecting the audio silence comprises: identifying portions of the audio data having an amplitude below a threshold, wherein one of the portions includes the audio silence; and determining that a length of the one of the portions is greater than or equal to a silence duration threshold. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, further comprising: storing, in a data store, a look up table associating punctuation marks with silence durations; and identifying the punctuation mark from the look up table based at least in part on the length of the one of the portions being greater than or equal to the silence duration threshold. ### Claim 4 (depends on 2) The computer-implemented method of claim 2, further comprising: generating a speech rate for the audio data for the spoken words, wherein the speech rate identifies a quantity of words spoken over a period of time; and generating the silence duration threshold based at least on the speech rate in relation to at least one of: (i) a baseline speech rate, and (ii) a baseline silence length. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the punctuation mark is included in a set of punctuation marks, and wherein the computer-implemented method further comprises: detecting a keyword in the transcript; and identifying the punctuation mark from the set of punctuation marks based at least in part on the keyword. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the punctuation mark is included in a set of punctuation marks, and wherein the computer-implemented method further comprises: storing, in a data store, a set of unpunctuated words; and prior to generating the punctuated transcript, determining that the spoken word is not included in the set of unpunctuated words. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, further comprising: associating the audio silence with a second word timing for a second spoken word included in the spoken words; generating a second punctuated transcript by including the punctuation mark with the second spoken word associated with the audio silence; receiving, from a natural language processing system, first intent data identifying a first semantic intent for the punctuated transcript; receiving, from the natural language processing system, second intent data identifying a second semantic intent for the second punctuated transcript; and determining that the first intent data more closely reflects a target semantic intent than the second intent data. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, comprising: detecting an audio characteristic for a portion of the audio data associated with the spoken word; and identifying the punctuation mark for the spoken word based at least in part on a comparison of the audio characteristic with the audio characteristic for at least one other portion of the audio data. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein the audio characteristic comprises at least one of: pitch, speed, or tone of the audio data. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, further comprising adjusting capitalization of a word adjacent to the punctuation mark in the punctuated transcript, wherein the punctuation mark comprises at least one of a period or a question mark. ### Claim 11 A system comprising: a data store storing specific computer-executable instructions; and a computing device in communication with the data store, the computing device including a processor that executes the specific computer-executable instructions to cause the system to at least: receive an audio file including audio data for spoken words; transmit, to a transcription device, the audio file; receive, from the transcription device, transcription information, wherein the transcription information includes: (i) a transcript of the spoken words, and (ii) word timings identifying a position where the transcription device identified each of the spoken words in the audio data; detect an audio silence in the audio data; associate the audio silence with a word timing for a spoken word included in the spoken words based on a matching of a start time of the audio silence with an end time of the spoken word, wherein the start time of the audio silence is determined from the audio data, and wherein the end time of the spoken word is included in the word timings received from the transcription device; remove punctuation characters from the transcript of the spoken words; and generate a punctuated transcript by including a punctuation mark adjacent to the spoken word associated with the audio silence. ### Claim 12 (depends on 11) The system of claim 11, wherein the processor executes the specific computer-executable instructions to cause the system to detect the audio silence by at least: identifying portions of the audio data having an amplitude below a threshold, wherein one of the portions includes the audio silence; and determining that a length of the one of the portions is greater than or equal to a silence duration threshold. ### Claim 13 (depends on 12) The system of claim 12, wherein the processor executes the specific computer-executable instructions to further cause the system to: store, in a data store, a look up table associating punctuation marks with silence durations; identify the punctuation mark from the look up table based at least in part on the length of the one of the portions being greater than or equal to the silence duration threshold. ### Claim 14 (depends on 12) The system of claim 12, wherein the processor executes the specific computer-executable instructions to further cause the system to: generate a speech rate for the audio data for the spoken words, wherein the speech rate identifies a quantity of words spoken over a period of time; and generate the silence duration threshold based at least on the speech rate in relation to at least one of: (i) a baseline speech rate, and (ii) a baseline silence detection threshold length. ### Claim 15 (depends on 11) The system of claim 11, wherein the punctuation mark is included in a set of punctuation marks, and wherein the processor executes the specific computer-executable instructions to further cause the system to: detect a keyword in the transcript; and identify the punctuation mark from the set of punctuation marks based at least in part on the keyword. ### Claim 16 (depends on 11) The system of claim 11, wherein the punctuation mark is included in a set of punctuation marks, and wherein the processor executes the specific computer-executable instructions to further cause the system to: store, in a data store, a set of unpunctuated words; and prior to generating the punctuated transcript, determine that the spoken word is not included in the set of unpunctuated words. ### Claim 17 (depends on 11) The system of claim 11, wherein the processor executes the specific computer-executable instructions to further cause the system to: associate the audio silence with a second word timing for a second spoken word included in the spoken words; generate a second punctuated transcript by including the punctuation mark prior to the second spoken word associated with the audio silence; receive, from a natural language processing system, first intent data identifying a first semantic intent for the punctuated transcript; receive, from the natural language processing system, second intent data identifying a second semantic intent for the second punctuated transcript; and determine that the first intent data more closely reflects a target semantic intent than the second intent data. ### Claim 18 (depends on 11) The system of claim 11, wherein the processor executes the specific computer-executable instructions to further cause the system to: detect an audio characteristic for a portion of the audio data associated with the spoken word; and identify the punctuation mark for the spoken word based at least in part on a comparison of the audio characteristic with the audio characteristic for at least one other portion of the audio data. ### Claim 19 (depends on 18) The system of claim 18, wherein the audio characteristic comprises at least one of: pitch, speed, or tone of the audio data. ### Claim 20 (depends on 11) The system of claim 11 wherein the processor executes the specific computer-executable instructions to further cause the system to detect language of the spoken words based at least in part on a source of the audio file, and wherein generating the punctuated transcript comprises: identifying a punctuation rule for the language of the spoken words; including the punctuation mark in the punctuated transcript according to the punctuation rule for the language. ### Claim 21 A computer-implemented method comprising: under control of one or more processing devices, receiving an audio file including audio data for spoken words; transmitting, to a transcription device, the audio file; receiving, from the transcription device, transcription information, wherein the transcription information includes: (i) a transcript of the spoken words, and (ii) word timings identifying a position where the transcription device identified each of the spoken words in the audio data; detecting a non-speech portion of the audio data based at least in part on at least one of: amplitude, frequency, pitch, speed, or tone of the non-speech portion of audio data in relation to a non-speech detection threshold; associating the non-speech portion with a word timing for a spoken word included in the spoken words based on a matching of a start time of the non-speech portion with an end time of the spoken word, wherein the start time of the non-speech portion is determined from the audio data, and wherein the end time of the spoken word is included in the word timings received from the transcription device; removing punctuation characters from the transcript of the spoken words; and generating a punctuated transcript by including a punctuation mark adjacent to the spoken word associated with the non-speech portion. ### Claim 22 (depends on 21) The computer-implemented method of claim 21, wherein the punctuation mark is included in a set of punctuation marks, and wherein the computer-implemented method further comprises: detecting a keyword in the transcript; and identifying the punctuation mark from the set of punctuation marks based at least in part on the keyword. ### Claim 23 (depends on 21) The computer-implemented method of claim 21, wherein the punctuation mark is included in a set of punctuation marks, and wherein the computer-implemented method further comprises: storing, in a data store, a set of unpunctuated words; and prior to generating the punctuated transcript, determining that the spoken word is not included in the set of unpunctuated words. ### Claim 24 (depends on 21) The computer-implemented method of claim 21, further comprising: associating the non-speech portion with a second word timing for a second spoken word included in the spoken words; generating a second punctuated transcript by including the punctuation mark with the second spoken word associated with the non-speech portion; receiving, from a natural language processing system, first intent data identifying a first semantic intent for the punctuated transcript; receiving, from the natural language processing system, second intent data identifying a second semantic intent for the second punctuated transcript; and determining that the first intent data more closely reflects a target semantic intent than the second intent data.