Systems and methods for automatically generating sound event subtitles
AssigneeNetflix, Inc.
InventorsYadong Wang, Shilpa Jois Rao
Life of this patent
Estimating the term…
Abstract
The disclosed computer-implemented method may include systems and methods for automatically generating sound event subtitles for digital videos. For example, the systems and methods described herein can automatically generate subtitles for sound events within a digital video soundtrack that includes sounds other than speech. Additionally, the systems and methods described herein can automatically generate sound event subtitles as part of an automatic and comprehensive approach that generates subtitles for all sounds within a soundtrack of a digital video—thereby avoiding the need for any manual inputs as part of the subtitling process.
Claims20
A computer-implemented method comprising:
- identifying an input audio waveform from a soundtrack of a digital video;
- applying a sound event classification model to the input audio waveform to:
- detect one or more sound events within the input audio waveform, each of the one or more sound events comprising sounds other than speech;
- and
- generate sound event classification labels and sound event time boundaries for the one or more sound events;
- translating the sound event classification labels into digital video display metadata describing the one or more sound events;
- and
- maintaining the digital video display metadata in association with the digital video for playback within the digital video corresponding to the sound event time boundaries by adding the digital video display metadata to a subtitle file associated with the digital video at an insertion point that is determined based on the sound event time boundaries.
A system comprising:
- at least one physical processor;
- and
- physical memory comprising computer-executable instructions that, when executed by the at least one physical processor, cause the at least one physical processor to perform acts comprising:
- identifying an input audio waveform from a soundtrack of a digital video;
- applying a sound event classification model to the input audio waveform to:
- detect one or more sound events within the input audio waveform, each of the one or more sound events comprising sounds other than speech;
- and
- generate sound event classification labels and sound event time boundaries for the one or more sound events;
- translating the sound event classification labels into digital video display metadata describing the one or more sound events;
- and
- maintaining the digital video display metadata in association with the digital video for playback within the digital video corresponding to the sound event time boundaries by adding the digital video display metadata to a subtitle file associated with the digital video at an insertion point that is determined based on the sound event time boundaries.
A computer-implemented method comprising:
- generating an input audio waveform from a soundtrack of a digital video;
- applying, in parallel:
- a sound event classification model to the input audio waveform to generate sound event classification labels;
- and
- a speech music activation detection model to the input audio waveform to generate waveform segments comprising speech and waveform segments comprising music;
- applying:
- a music classification model to the waveform segments comprising music to generate music classification predictions for the waveform segments comprising music;
- and
- generating a subtitle file associated with the digital video comprising one or more of the sound event classification labels and the music classification predictions.
Dependencies are inferred from the claim text.