# US 2017/0287481 A1 · System and method to insert visual subtitles in videos > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2017/0287481 A1 - Document ID: US20170287481A1 - Status: Published application - Kind: A1 - Application number: 15/473,029 - Filed: 2017-03-29 - Published: 2017-10-05 - Earliest claimed: 2016-03-31 - Assignees: Tata Consultancy Services Limited - Applicants: Tata Consultancy Services Limited - Inventors: Chitralekha Bhat; Sunil Kumar Kopparapu; Ashish Panda - Web page: https://patentgrep.com/patent/US20170287481A1 ## CPC - G10L 15/04: Segmentation; Word boundary detection - G10L 15/183: using context dependencies, e.g. language models - G10L 15/25: using position of the lips, movement of the lips or face analysis - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 21/06: Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids (G10L15/26 takes precedence) - G10L 21/10: Transforming into visible information - G10L 25/57: for processing of video signals - G10L 25/63: for estimating an emotional state - G10L 25/81: for discriminating voice from music - G11B 27/036: Insert-editing - G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units} - G10L 2021/065: {Aids for the handicapped in understanding} - G10L 2021/105: {Synthesis of the lips movements from speech, e.g. for talking heads} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) ## Abstract A system and method to insert visual subtitles in videos is described. The method comprises segmenting an input video signal to extract the speech segments and music segments. Next, a speaker representation is associated for each speech segment corresponding to a speaker visible in the frame. Further, speech segments are analysed to compute the phones and the duration of each phone. The phones are mapped to a corresponding viseme and a viseme based language model is created with a corresponding score. Most relevant viseme is selected for the speech segments by computing a total viseme score. Further, a speaker representation sequence is created such that phones and emotions in the speech segments are represented as reconstructed lip movements and eyebrow movements. The speaker representation sequence is then integrated with the music segments and super imposed on the input video signal to create subtitles. ## Claims ### Claim 1 A computer implemented system for creation of visually subtitled videos, wherein said system comprises: a memory storing instructions; a memory coupled to the processor, wherein the memory has a plurality of modules stored therein that are executable by the processor, the plurality of modules comprising: a segmenting module to segment an audio signal with at least one speaker in a video frame from an input video signal, and a music segment from the input video signal; an analysis module to determine the most relevant viseme from the segmented audio signal; a synthesizer module to generate a speaker representation sequence of the segmented audio signal; and an integration module to integrate the segmented audio signal and viseme sequence to generate visual subtitles. ### Claim 2 (depends on 1) The system as claimed in claim 1, wherein the segmenting module further comprises creating a speaker representation by separating speech segments for at least one speaker visible in the video frame, and separating music segments from an input video frame. ### Claim 3 (depends on 2) The system as claimed in claim 2, wherein the speaker representation further comprises extracting a speaker face of the at least one speaker visible in the video frame and associating the speaker face with the speech segment. ### Claim 4 (depends on 3) The system as claimed in claim 3, wherein the speaker representation for the speaker not visible in the video frame comprises creating a generic speaker face, wherein gender of the speaker is identified from the speech segment. ### Claim 5 (depends on 1) The system as claimed in claim 1, wherein the analysis module further comprises: mapping a plurality of recognized phones to a single viseme or to multiple visemes using a language model; determining a likelihood score of the recognised phones; determining a viseme based language model score (VLM) for the at least one mapped visemes; and computing a total viseme score for the at least one mapped viseme. ### Claim 6 (depends on 5) The system as claimed in claim 5, wherein the computing a total viseme score further comprises: determining the sum of the product of the likelihood score and the VLM score when more than one phone map to the same viseme or determining the product of the likelihood score and VLM score when one phone maps to one single viseme; and selecting the most relevant viseme further comprises comparing the total viseme score and selecting the viseme with the highest total score. ### Claim 7 (depends on 1) The system as claimed in claim 1, wherein the synthesizer module further comprises: removing the lip region and mouth region of the speaker representation; and reconstructing a lip movement and a eyebrow movement sequence using a series of facial animation points executed through an interpolation simulation that is corresponding to the most relevant viseme for the computed time duration of the viseme. ### Claim 8 (depends on 1) The system as claimed in claim 1, wherein the integration module further comprises integrating the music segment and speaker representation sequence in time synchrony with the input video signal. ### Claim 9 A computer implemented method for creation of visually subtitled videos, wherein the method comprises: segmenting at least one audio signal from an input video frame of a video signal by an audio segmenting module; analysing the segmented audio signal for the input video frame to select the most relevant viseme by an analysis module; generating a speaker representation sequence subsequent to audio segmentation by a synthesizer module; and integrating the video signal with the speaker representation sequence to generate visual subtitles by an integration module. ### Claim 10 (depends on 9) The method as claimed in claim 9, wherein the segmenting of the at least one audio signal from an input video frame further comprises: separating speech segments for at least one speaker visible in the video frame, and separating music segments from an input video frame; and associating the at least one acoustic model to a speaker representation. ### Claim 11 (depends on 10) The method as claimed in claim 10, wherein the speaker representation comprises extracting a speaker face of the at least one speaker visible in the video frame and associating the speaker face with the speech segment. ### Claim 12 (depends on 10) The method as claimed in claim 10, wherein the speaker representation for the speaker not visible in the video frame comprises creating a generic speaker face, wherein gender of the speaker is identified from the speech segment. ### Claim 13 (depends on 9) The method as claimed in claim 9, wherein the selecting the most relevant viseme further comprises: mapping a plurality of recognized phones to a single viseme or to multiple visemes using a language model; and determining a likelihood score of the recognised phones; determining a viseme based language model score (VLM) for the at least one mapped visemes; and computing a total viseme score for the at least one mapped viseme. ### Claim 14 (depends on 13) The method as claimed in claim 13, wherein the computing a total viseme score further comprises determining the sum of the product of the likelihood score and the VLM score when more than one phones map to the same viseme or determining the product of the likelihood score and VLM score when one phone maps to one single viseme. ### Claim 15 (depends on 13) The method as claimed in claim 13, wherein selecting the most relevant viseme further comprises comparing the total viseme score and selecting the viseme with the highest total score. ### Claim 16 (depends on 9) The method as claimed in claim 9, wherein generating the speaker representation sequence further comprises removing the lip region and mouth region of the speaker representation and reconstructing a lip movement and an eyebrow movement sequence. ### Claim 17 (depends on 16) The method as claimed in claim 16, wherein generating the speaker representation sequence further comprises a series of facial animation points to form a lip shape or an eyebrow shape, executed through an interpolation simulation that is corresponding to the most relevant viseme for the computed time duration of the viseme. ### Claim 18 The method as claimed in 17 wherein the eyebrow movement and the lip movement corresponds to emotion detected in the speech segment of the segmented audio. ### Claim 19 (depends on 9) The method as claimed in claim 9, wherein the integrating the video signal further comprises integrating the viseme sequence and notational representation of the music segment in time synchrony with the input video signal. ### Claim 20 One or more non-transitory machine readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors causes: segmenting at least one audio signal from an input video frame of a video signal by an audio segmenting module; analysing the segmented audio signal for the input video frame to select the most relevant viseme by an analysis module; generating a speaker representation sequence subsequent to audio segmentation by a synthesizer module; and integrating the video signal with the speaker representation sequence to generate visual subtitles by an integration module.