US 2022/0189173 A1Published application

Generating highlight video from video and text inputs

AssigneeBaidu USA LLC

InventorsXin ZHOU, Le KANG, Zhiyu CHENG, Hao TIAN, Daming LU, Dapeng LI, Jingya XUN, Jeff WANG, Xi CHEN, Xing LI

Life of this patent

Estimating the term…

Loading patent…

Abstract

Presented herein are systems, methods, and datasets for automatically and precisely generating highlight or summary videos of content. In one or more embodiments, the inputs comprise a text (e.g., an article) of the key event(s) (e.g., a goal, a player action, etc.) in an activity (e.g., a game, a concert, etc.) and a video or videos of the activity. In one or more embodiments, the output is a short video of an event or events in the text, in which the video may include commentary about the highlighted events and/or other audio (e.g., music), which may also be automatically synthesized.

Claims20

Dependent claims

A computer-implemented method comprising:

given an input text comprising a reference to an event in an activity,
parsing the input text, using a text parsing module, to identify the event referenced in the input text;
and
using a text-to-speech (TTS) module to convert the input text into TTS-generated audio;
given an input video of at least part of the activity and the identified event:
performing time anchoring to correlate runtime of the input video with runtime of the activity;
generating an initial clip from the input video that includes the event by using timing information parsed from the input text, from an additional source related to the activity, or both and correlated time obtained by time anchoring to identify an approximate time of when the event occurred during the activity;
extracting features from the initial video clip;
obtaining a final time value of the event in the initial video clip using the extracted features and trained neural network model;
responsive to a runtime of the initial video clip being inconsistent with a runtime of the TTS-generated audio, generating a final video clip by editing the initial video clip to have a runtime consistent with the runtime of the TTS-generated audio;
and
responsive to the runtime of the initial video clip being consistent with the runtime of the TTS-generated audio, using the initial video clip as the final video clip;
and
combining the TTS-generated audio with the final video clip to generate an event highlight video.

11 more elements

A system comprising:

one or more processors;
and
a non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
given an input text comprising a reference to an event in an activity,
parsing the input text, using a text parsing module, to identify the event referenced in the input text;
and
using a text-to-speech (TTS) module to convert the input text into TTS-generated audio;
given an input video of at least part of the activity and the identified event:
performing time anchoring to correlate runtime of the input video with runtime of the activity;
generating an initial clip from the input video that includes the event by using timing information parsed from the input text, from an additional source related to the activity, or both and correlated time obtained by time anchoring to identify an approximate time of when the event occurred during the activity;
extracting features from the initial video clip;
obtaining a final time value of the event in the initial video clip using the extracted features and trained neural network model;
responsive to a runtime of the initial video clip being inconsistent with a runtime of the TTS-generated audio, generating a final video clip by editing the initial video clip to have a runtime consistent with the runtime of the TTS-generated audio;
and
responsive to the runtime of the initial video clip being consistent with the runtime of the TTS-generated audio, using the initial video clip as the final video clip;
and
combining the TTS-generated audio with the final video clip to generate an event highlight video.

14 more elements

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising:

given an input text comprising a reference to an event in an activity,
parsing the input text, using a text parsing module, to identify the event referenced in the input text;
and
using a text-to-speech (TTS) module to convert the input text into TTS-generated audio;
given an input video of at least part of the activity and the identified event:
performing time anchoring to correlate runtime of the input video with runtime of the activity;
generating an initial clip from the input video that includes the event by using timing information parsed from the input text, from an additional source related to the activity, or both and correlated time obtained by time anchoring to identify an approximate time of when the event occurred during the activity;
extracting features from the initial video clip;
obtaining a final time value of the event in the initial video clip using the extracted features and trained neural network model;
responsive to a runtime of the initial video clip being inconsistent with a runtime of the TTS-generated audio, generating a final video clip by editing the initial video clip to have a runtime consistent with the runtime of the TTS-generated audio;
and
responsive to the runtime of the initial video clip being consistent with the runtime of the TTS-generated audio, using the initial video clip as the final video clip;
and
combining the TTS-generated audio with the final video clip to generate an event highlight video.

11 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/533,769Filed 23 Nov 2021
Priority: 13 Dec 2020
Published: 16 Jun 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification21 codes

G06V 20/47
Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames: detecting features for…
G06F 16/7844
Using metadata automatically derived from the content: using original textual content or text extracted from visual content or transcript of audio…
G06F 16/7867
Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually: using information manually…
G06F 40/205
Natural language analysis: parsing

Family

Application63/124,832Us provisional application · filed 13 Dec 2020
Application17/393,373Continuation in part · filed 3 Aug 2021
Application17/533,769Continuation in part
US 11,769,327 B2Related patent
US 12,223,720 B2Related patent
US 2022/0188550 A1Published application
US 2022/0189173 A1This patent

From the US filing history. Foreign family members aren’t listed.