# US 11,443,734 B2 · System and method for combining phonetic and automatic speech recognition search > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,734 B2 - Document ID: US11443734B2 - Status: Granted - Kind: B2 - Application number: 16/550,700 - Filed: 2019-08-26 - Published: 2021-03-04 - Granted: 2022-09-13 - Earliest claimed: 2019-08-26 - Assignees: Nice Ltd. - Applicants: Nice Ltd. - Inventors: William Mark Finlay; Robert William Morris; Peter S. Cardillo; Maria Michaela Kunin - Web page: https://patentgrep.com/patent/US11443734B2 ## CPC - G06F 16/685: {using automatically derived transcript of audio data, e.g. lyrics (speech recognition G10L15/00)} - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/08: Speech classification or search - G10L 13/00: Speech synthesis; Text to speech systems - G10L 15/04: Segmentation; Word boundary detection - G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A text search query including one or more words may be received. An ASR index created for an audio recording may be searched over using the query to produce ASR search results including words, each word associated with a confidence score. For each of the words in the ASR search results associated with a confidence score below a threshold (and in some cases having one or more preceding words in the ASR index and one or more subsequent words in the ASR index), a phonetic representation of the audio recording may be searched for the word having the confidence score below the threshold, where it occurs in the audio recording, possibly after the one or more preceding words and in the audio recording before the one or more subsequent words, to produce phonetic search results. Search results may be returned include ASR and phonetic results. ## Claims ### Claim 1 A method for searching an audio recording using text, the method comprising: accepting a text search query; converting the text search query to a phonetic representation of the text search query; searching over an automatic speech recognition (ASR) index created for an audio file using the text search query to produce ASR search results wherein the ASR index comprises textual representations of words, each textual representation associated with a confidence score; for one or more words in the ASR search results associated with a confidence score below a threshold, searching over a portion of the phonetic representation of the audio file using the phonetic representation of the text search query to produce phonetic search results, wherein the phonetic representation is extended by a window comprised of at least one word before and at least one word after the one or more words; and returning as search results ASR search results and phonetic search results wherein the searching further comprises: processing each word in the ASR transcript sequentially; starting a new time segment to be used with phonetic processing when an ASR word is encountered with confidence below a threshold, wherein the new time segment start is based on at least one of: start time of the current low confidence ASR word, start or end time of the previous high confidence ASR word; start or end time of the N previous confidence words; and fixed time window added to or subtracted from any of the aforementioned values; continuing processing each word in the ASR and incrementing the phonetic portion until an ASR transcript word is found with a confidence score equal to or above the threshold or until the end of the ASR transcript is reached; setting the end of the of the time sequence and adding a master list of media segments requiring a phonetic index; and continuing to process each word until the entire ASR transcript has been processed. ### Claim 2 (depends on 1) The method of claim 1, wherein the phonetic representation represents portions of the audio file corresponding to low confidence scores. ### Claim 3 (depends on 1) The method of claim 1 wherein the phonetic representation and ASR index are comprised 1, within a composite index, and wherein the phonetic representation represents only portions of the audio file comprising words associated with a confidence score below the threshold. ### Claim 4 (depends on 1) The method of claim 1 wherein the phonetic representation and ASR index are comprised within a composite index, and wherein the phonetic representation represents portions of the audio file comprising words associated with a confidence score below the threshold and an overlap portion including words associated with a confidence score not below the threshold. ### Claim 5 (depends on 1) The method of claim 1, wherein the confidence score indicates the confidence that the word accurately represents the corresponding word in the audio recording. ### Claim 6 (depends on 3) The method of claim 3, wherein the search results comprise a location in the audio recording corresponding to the text search query. ### Claim 7 A method for searching an audio recording using text, the method comprising: accepting a text search query comprising a plurality of words; searching over an automatic speech recognition (ASR) index created for an audio recording using the text search query to produce ASR search results, the ASR search results comprising words, each word associated with a confidence score; for one or more words comprised in the ASR search results associated with a confidence score below a threshold and having one or more preceding words in the ASR index and one or more subsequent words in the ASR index, searching over a portion of the phonetic representation of the audio recording for the word associated with a confidence score below the threshold where it occurs in the audio recording after the one or more preceding words and in the audio recording before the one or more subsequent words, to produce phonetic search results, wherein the phonetic representation is extended by a window comprised of at least one word before and at least one word after the word position; and returning as search results ASR search results and phonetic search results; the searching further comprising: processing each word in the ASR transcript sequentially; starting a new time segment to be used with phonetic processing when an ASR word is encountered with confidence below a threshold, wherein the new time segment start is based on at least one of: start time of the current low confidence ASR word, start or end time of the previous high confidence ASR word; start or end time of the N previous confidence words; and fixed time window added to or subtracted from any of the aforementioned values; continuing processing each word in the ASR and incrementing the phonetic portion until an ASR transcript word is found with a confidence score equal to or above the threshold or until the end of the ASR transcript is reached; setting the end of the of the time sequence and adding a master list of media segments requiring a phonetic index; and continuing to process each word until the entire ASR transcript has been processed. ### Claim 8 (depends on 7) The method of claim 7, comprising searching over a phonetic representation of the audio file before the end of a preceding word and after the beginning of a subsequent word, to produce phonetic search results. ### Claim 9 (depends on 7) The method of claim 7 wherein the phonemic representation and ASR index are comprised within a composite index, and wherein the phonetic representation represents only portions of the audio file comprising words associated with a confidence score below the threshold. ### Claim 10 (depends on 7) The method of claim 7 wherein the phonetic representation and ASR index are comprised within a composite index, and wherein the phonetic representation represents portions of the audio file comprising words associated with a confidence score below the threshold and an overlap portion including words associated with a confidence score not below the threshold. ### Claim 11 (depends on 7) The method of claim 7, wherein the confidence score indicates the confidence that the word accurately represents the corresponding word in the audio recording. ### Claim 12 (depends on 7) The method of claim 7, wherein the search results comprise a location in the audio recording corresponding to the text search query. ### Claim 13 (depends on 7) The method of claim 7, wherein searching over the ASR index comprises: converting the text search query to a phoneme representation; and using the phoneme representation to access a phoneme sequence lookup table, to return an index to the ASR index. ### Claim 14 A system for searching an audio recording using text, the system comprising: a memory; and a controller configured to: accept a text search query comprising a plurality of words; search over an automatic speech recognition (ASR) index created for an audio recording using the text search query to produce ASR search results, the ASR search results comprising words, each word associated with a confidence score; for one or more words comprised in the ASR search results associated with a confidence score below a threshold and having one or more preceding words in the ASR index and one or more subsequent words in the ASR index, search over a portion of the phonetic representation of the audio recording for the word associated with a confidence score below the threshold where it occurs in the audio recording after the one or more preceding words and in the audio recording before the one or more subsequent words, to produce phonetic search results, wherein the phonetic representation is extended by a window comprised of at least one word before and at least one word after the position; and returning as search results ASR search results and phonetic search results; the searching further comprising: processing each word in the ASR transcript sequentially; starting a new time segment to be used with phonetic processing when an ASR word is encountered with confidence below a threshold, wherein the new time segment start is based on at least one of: start time of the current low confidence ASR word, start or end time of the previous high confidence ASR word; start or end time of the N previous confidence words; and fixed time window added to or subtracted from any of the aforementioned values; continuing processing each word in the ASR and incrementing the phonetic portion until an ASR transcript word is found with a confidence score equal to or above the threshold or until the end of the ASR transcript is reached; setting the end of the of the time sequence and adding a master list of media segments requiring a phonetic index; and continuing to process each word until the entire ASR transcript has been processed. ### Claim 15 (depends on 14) The system of claim 14, wherein the controller is configured to search over a phonetic representation of the audio file before the end of a preceding word and after the beginning of a subsequent word, to produce phonetic search results. ### Claim 16 (depends on 14) The system of claim 14 wherein the phonetic representation and ASR index are comprised within a composite index, and wherein the phonetic representation represents only portions of the audio file comprising words associated with a confidence score below the threshold. ### Claim 17 (depends on 14) The system of claim 14 wherein the phonetic representation and ASR index are comprised within a composite index, and wherein the phonetic representation represents portions of the audio file comprising words associated with a confidence score below the threshold and an overlap portion including words associated with a confidence score not below the threshold. ### Claim 18 (depends on 14) The system of claim 14, wherein the confidence score indicates the confidence that the word accurately represents the corresponding word in the audio recording. ### Claim 19 (depends on 14) The system of claim 14, wherein the search results comprise a location in the audio recording corresponding to the text search query. ### Claim 20 (depends on 14) The system of claim 14, wherein searching over the ASR index comprises: converting the text search query to a phoneme representation; and using the phoneme representation to access a phoneme sequence lookup table, to return an index to the ASR index.