# US 2017/0256262 A1 · System and Method for Speech-to-Text Conversion > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2017/0256262 A1 - Document ID: US20170256262A1 - Status: Published application - Kind: A1 - Application number: 15/070,827 - Filed: 2016-03-15 - Published: 2017-09-07 - Earliest claimed: 2016-03-02 - Assignees: Wipro Limited - Applicants: Wipro Limited - Inventors: Manjunath RAMACHANDRA; Priyanshu SHARMA - Web page: https://patentgrep.com/patent/US20170256262A1 ## CPC - G10L 15/25: using position of the lips, movement of the lips or face analysis - G06F 40/197: Version control (for software G06F8/71) - G10L 15/14: using statistical models, e.g. Hidden Markov Models [HMMs] (G10L15/18 takes precedence) - G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract This disclosure relates generally to speech recognition, and more particularly to system and method for speech-to-text conversion using audio as well as video input. In one embodiment, a method is provided for performing speech to text conversion. The method comprises receiving an audio data and a video data of a user while the user is speaking, generating a first raw text based on the audio data via one or more audio-to-text conversion algorithms, generating a second raw text based on the video data via one or more video-to-text conversion algorithms, determining one or more errors by comparing the first raw text and the second raw text, and correcting the one or more errors by applying one or more rules. The one or more rules employ at least one of a domain specific word database, a context of conversation, and a prior communication history. ## Claims ### Claim 1 A method for performing speech to text conversion, the method comprising: receiving, via a processor, an audio data and a video data of a user while the user is speaking; generating, via the processor, a first raw text based on the audio data via one or more audio-to-text conversion algorithms; generating, via the processor, a second raw text based on the video data via one or more video-to-text conversion algorithms; determining, via the processor, one or more errors by comparing the first raw text and the second raw text; and correcting, via the processor, the one or more errors by applying one or more rules, wherein the one or more rules employ at least one of a domain specific word database, a context of conversation, and a prior communication history. ### Claim 2 (depends on 1) The method of claim 1, wherein receiving the video data comprises receiving a movement of a mouth region of the user while the user is speaking, wherein the movement of the mouth region comprises at least one of a movement of one or more articulators, a movement of facial muscles, and a movement of vocal cords. ### Claim 3 (depends on 1) The method of claim 1, wherein generating the first raw text comprises: extracting a sequence of phonemes from the audio data; and mapping the sequence of phonemes to a corresponding text using a phoneme-to-text database and a word database. ### Claim 4 (depends on 3) The method of claim 3, wherein the phoneme-to-text database is customized for the user during a training phase. ### Claim 5 (depends on 1) The method of claim 1, wherein generating the second raw text comprises: extracting a sequence of visemes from the video data; and mapping the sequence of visemes to a corresponding text using a viseme-to-text database and a domain specific word database. ### Claim 6 (depends on 5) The method of claim 5, wherein the viseme-to-text database is customized for the user during a training phase. ### Claim 7 (depends on 1) The method of claim 1, wherein determining the one or more errors comprises comparing a sequence of phonemes in the first raw text with a corresponding sequence of visemes in the second raw text for one or more mismatches. ### Claim 8 (depends on 1) The method of claim 1, wherein correcting the one or more errors comprises applying the one or more rules in a pre-defined order. ### Claim 9 (depends on 1) The method of claim 1, further comprising generating a final text based on the first raw text or the second raw text, and the correction. ### Claim 10 A system for performing speech to text conversion, the system comprising: at least one processor; and a computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: receiving an audio data and a video data of a user while the user is speaking; generating a first raw text based on the audio data via one or more audio-to-text conversion algorithms; generating a second raw text based on the video data via one or more video-to-text conversion algorithms; determining one or more errors by comparing the first raw text and the second raw text; correcting the one or more errors by applying one or more rules, wherein the one or more rules employ at least one of a domain specific word database, a context of conversation, and a prior communication history. ### Claim 11 (depends on 10) The system of claim 10, wherein receiving the video data comprises receiving a movement of a mouth region of the user while the user is speaking, wherein the movement of the mouth region comprises at least one of a movement of one or more articulators, a movement of facial muscles, and a movement of vocal cords. ### Claim 12 (depends on 10) The system of claim 10, wherein generating the first raw text comprises: extracting a sequence of phonemes from the audio data; and mapping the sequence of phonemes to a corresponding text using a phoneme-to-text database and a word database. ### Claim 13 (depends on 12) The system of claim 12, wherein the phoneme-to-text database is customized for the user during a training phase. ### Claim 14 (depends on 10) The system of claim 10, wherein generating the second raw text comprises: extracting a sequence of visemes from the video data; and mapping the sequence of visemes to a corresponding text using a viseme-to-text database and a domain specific word database. ### Claim 15 (depends on 14) The system of claim 14, wherein the viseme-to-text database is customized for the user during a training phase. ### Claim 16 (depends on 10) The system of claim 10, wherein determining the one or more errors comprises comparing a sequence of phonemes in the first raw text with a corresponding sequence of visemes in the second raw text for one or more mismatches. ### Claim 17 (depends on 10) The system of claim 10, wherein correcting the one or more errors comprises applying the one or more rules in a pre-defined order. ### Claim 18 (depends on 10) The system of claim 10, wherein the operations further comprise generating a final text based on the first raw text or the second raw text, and the correction. ### Claim 19 A non-transitory computer-readable medium storing computer-executable instructions for: receiving an audio data and a video data of a user while the user is speaking; generating a first raw text based on the audio data via one or more audio-to-text conversion algorithms; generating a second raw text based on the video data via one or more video-to-text conversion algorithms; determining one or more errors by comparing the first raw text and the second raw text; correcting the one or more errors by applying one or more rules, wherein the one or more rules employ at least one of a domain specific word database, a context of conversation, and a prior communication history. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein determining the one or more errors comprises comparing a sequence of phonemes in the first raw text with a corresponding sequence of visemes in the second raw text for one or more mismatches, and wherein correcting the one or more errors comprises applying the one or more rules in a pre-defined order.