Audio signal processing device and method for synchronizing speech and text by using machine learning model

A machine learning device for generating a neural network model configured to synchronize an audio signal and text with a speech signal, the audio signal comprising speech and the text corresponding to the speech, and the device comprising a processor,

• wherein the processor is configured to:
• obtain first audio pronunciation information corresponding to the speech, the first audio pronunciation information being divided with regard to multiple frames included in the audio signal;
• convert the text to a code, and obtain first text pronunciation information based on the converted code, wherein the first text pronunciation information is divided with regard to multiple segments and the converted code is an international phonetic alphabet (IPA);
• obtain a similarity between second audio pronunciation information, which is a feature extracted from each of the multiple frames of the first audio pronunciation information, and second text pronunciation information, which is a feature extracted from each of the multiple segments of the first text pronunciation information;
• based on the similarity, obtain a time stamp indicating a time at which each of the multiple segments of the text corresponding to each of the multiple segments of the second text pronunciation information corresponds to an audio signal;
• and
• learn to synchronize the text with the audio signal according to the information indicating the similarity,
• wherein the neural network model has a UNet structure,
• wherein the UNet comprises a CNN and an RNN, and
• wherein, within a layer included in the UNet, the CNN performs analysis on a local similarity region, and the RNN performs analysis on a global similarity region.