Information processing device, information processing method, computer program, learning device, remote conference system, and support device

An information processing method comprising:

• a speech-to-unit conversion step of generating an acoustic unit from a speech waveform;
• and
• a unit-to-speech conversion step of reconstructing a speech waveform from an acoustic unit.

A computer program that is described in a computer-readable format to allow a computer to function as:

• a speech-to-unit encoder that generates an acoustic unit from a speech waveform;
• and
• a unit-to-speech decoder that reconstructs a speech waveform from an acoustic unit.

A learning device that learns a speech-to-unit encoder that generates an acoustic unit from a speech waveform,

• the learning device being configured to learn the speech-to-unit encoder to generate an acoustic unit common to a normal speech and a whisper, the acoustic unit being a latent expression in which a difference between the normal speech and the whisper is absorbed, by self-supervised learning of a Masked Language Model type using a normal speech and a whisper in which a part of an input is masked and the masked part is estimated from other related information.

A learning device that learns a unit-to-speech decoder that reconstructs a speech waveform from an acoustic unit,

• the learning device being configured to learn the unit-to-speech decoder using a first loss function based on a difference between a mel-spectrogram generated by the unit-to-speech decoder using an acoustic unit generated from a target speech using a frozen model and a mel-spectrogram generated from the target speech.