# US 12,223,720 B2 · Generating highlight video from video and text inputs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,720 B2
- Document ID: US12223720B2
- Status: Granted
- Kind: B2
- Application number: 17/533,769
- Filed: 2021-11-23
- Published: 2022-06-16
- Granted: 2025-02-11
- Earliest claimed: 2020-12-13
- Assignees: Baidu USA LLC
- Applicants: Baidu USA, LLC
- Inventors: Xin Zhou; Le Kang; Zhiyu Cheng; Hao Tian; Daming Lu; Dapeng Li; Jingya Xun; Jeff Wang; Xi Chen; Xing Li
- Web page: https://patentgrep.com/patent/US12223720B2

## CPC

- G06V 20/47: {Detecting features for summarising video content}
- G06F 16/7844: {using original textual content or text extracted from visual content or transcript of audio data}
- G06F 16/7867: {using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings}
- G06F 40/205: Parsing
- G06F 40/30: Semantic analysis
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06V 10/82: using neural networks
- G06V 20/44: {Event detection}
- G06V 30/153: {using recognition of characters or words}
- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G06F 40/279: Recognition of textual entities
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 25/57: for processing of video signals
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Presented herein are systems, methods, and datasets for automatically and precisely generating highlight or summary videos of content. In one or more embodiments, the inputs comprise a text (e.g., an article) of the key event(s) (e.g., a goal, a player action, etc.) in an activity (e.g., a game, a concert, etc.) and a video or videos of the activity. In one or more embodiments, the output is a short video of an event or events in the text, in which the video may include commentary about the highlighted events and/or other audio (e.g., music), which may also be automatically synthesized.

## Claims

### Claim 1

A computer-implemented method comprising:
given an input text comprising a reference to an event in an activity,
parsing the input text, using a text parsing module, to identify the event referenced in the input text; and
using a text-to-speech (TTS) module to convert the input text into TTS-generated audio;
given an input video that comprises the event:
obtaining a time mapping by performing time anchoring to correlate runtime of the input video with runtime of the activity;
generating an initial video clip from the input video that includes the event by using timing information related to the activity and the time mapping obtained by time anchoring to identify an approximate time in the input video of when the event occurred;
extracting features from the initial video clip;
obtaining a final time value of the event in the initial video clip using the extracted features and one or more trained neural network models;
responsive to a runtime of the initial video clip being inconsistent with a runtime of the TTS-generated audio, generating a final video clip by editing the initial video clip to have a runtime consistent with the runtime of the TTS-generated audio; and
responsive to the runtime of the initial video clip being consistent with the runtime of the TTS-generated audio, using the initial video clip as the final video clip; and
combining the TTS-generated audio with the final video clip to generate an event highlight video.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1 wherein the step of obtaining a time mapping by performing time anchoring to correlate runtime of the input video with runtime of the activity comprises:
using optical character recognition on a set of video frames of the input video to read time of a clock displayed in the input video;
given the recognized time of the clock, generating a set of time anchors, which comprises a time of the activity and a set of temporal shifts, if any; and
generating the time mapping that maps time of the clock with the input video runtime using at least some of the time anchors in the set of time anchors.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2 wherein the step of generating an initial video clip from the input video that includes the event by using timing information related to the activity and the time mapping obtained by time anchoring to identify an approximate time in the input video of when the event occurred comprises:
parsing data from metadata to obtain an approximate time for the event; and
using the approximate time for the event and the time mapping to generate the initial video clip, which includes the event.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1 wherein the steps of extracting features from the initial video clip and obtaining a final time value of the event in the initial video clip using the extracted features and one or more trained neural network models comprise:
performing feature extraction on the initial video clip using a set of two or more models; and
obtaining the final time value of the event in the initial video clip using an ensemble neural network model that receives input related to the features from the set of two or more models and outputs the final time value.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4 wherein the set of two or more models comprises:
a neural network model that extracts video features from the initial video clip;
a multimodal features neural network model that utilized video and audio information in the initial video clip to generate multimodal-based features; and
an audio feature extractor that generates a feature for the initial video clip based upon audio levels in the initial video clip.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1 further comprising:
generating a dataset comprising one or more entities related to the activity; and
using the dataset to filter text parsed from the input text to help identify which text parsed from the input text comprises information about the event.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1 further comprising:
given a supplemental text about the activity or event,
parsing the supplemental text, using a text parsing module, to identify information related to the event referenced in the supplemental text; and
using the information extracted from the supplemental text to help identify a time of occurrence of the event.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1 wherein the text parsing module is neural network-based, rules-based, or both.

### Claim 9

A system comprising:
one or more processors; and
a non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
given an input text comprising a reference to an event in an activity,
parsing the input text, using a text parsing module, to identify the event referenced in the input text; and
using a text-to-speech (TTS) module to convert the input text into TTS-generated audio;
given an input video that comprises the event:
obtaining a time mapping by performing time anchoring to correlate runtime of the input video with runtime of the activity;
generating an initial video clip from the input video that includes the event by using timing information related to the activity and the time mapping obtained by time anchoring to identify an approximate time in the input video of when the event occurred;
extracting features from the initial video clip;
obtaining a final time value of the event in the initial video clip using the extracted features and one or more trained neural network models;
responsive to a runtime of the initial video clip being inconsistent with a runtime of the TTS-generated audio, generating a final video clip by editing the initial video clip to have a runtime consistent with the runtime of the TTS-generated audio; and
responsive to the runtime of the initial video clip being consistent with the runtime of the TTS-generated audio, using the initial video clip as the final video clip; and
combining the TTS-generated audio with the final video clip to generate an event highlight video.

### Claim 10 (depends on 9)

The system of claim 9 wherein the step of obtaining a time mapping by performing time anchoring to correlate runtime of the input video with runtime of the activity comprises:
using optical character recognition on a set of video frames of the input video to read time of a clock displayed in the input video;
given the recognized time of the clock, generating a set of time anchors, which comprises a time of the activity and a set of temporal shifts, if any; and
generating the time mapping that maps time of the clock with the input video runtime using at least some of the time anchors in the set of time anchors.

### Claim 11 (depends on 10)

The system of claim 10 wherein the step of generating an initial video clip from the input video that includes the event by using timing information related to the activity and the time mapping obtained by time anchoring to identify an approximate time in the input video of when the event occurred comprises:
parsing data from metadata to obtain an approximate time for the event; and
using the approximate time for the event and the time mapping to generate the initial video clip, which includes the event.

### Claim 12 (depends on 9)

The system of claim 9 wherein the steps of extracting features from the initial video clip and obtaining a final time value of the event in the initial video clip using the extracted features and one or more trained neural network models comprise:
performing feature extraction on the initial video clip using a set of two or more models; and
obtaining the final time value of the event in the initial video clip using an ensemble neural network model that receives input related to the features from the set of two or more models and outputs the final time value.

### Claim 13 (depends on 12)

The system of claim 12 wherein the set of two or more models comprises:
a neural network model that extracts video features from the initial video clip;
a multimodal features neural network model that utilized video and audio information in the initial video clip to generate multimodal-based features; and
an audio feature extractor that generates a feature for the initial video clip based upon audio levels in the initial video clip.

### Claim 14 (depends on 9)

The system of claim 9 wherein the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
generating a dataset comprising one or more entities related to the activity; and
using the dataset to filter text parsed from the input text to help identify which text parsed from the input text comprises information about the event.

### Claim 15 (depends on 9)

The system of claim 9 wherein the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
given a supplemental text about the activity or event,
parsing the supplemental text, using a text parsing module, to identify information related to the event referenced in the supplemental text; and
using the information extracted from the supplemental text to help identify a time of occurrence of the event.

### Claim 16

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising:
given an input text comprising a reference to an event in an activity,
parsing the input text, using a text parsing module, to identify the event referenced in the input text; and
using a text-to-speech (TTS) module to convert the input text into TTS-generated audio;
given an input video that comprises the event:
obtaining a time mapping by performing time anchoring to correlate runtime of the input video with runtime of the activity;
generating an initial video clip from the input video that includes the event by using timing information related to the activity and the time mapping obtained by time anchoring to identify an approximate time in the input video of when the event occurred;
extracting features from the initial video clip;
obtaining a final time value of the event in the initial video clip using the extracted features and one or more trained neural network models;
responsive to a runtime of the initial video clip being inconsistent with a runtime of the TTS-generated audio, generating a final video clip by editing the initial video clip to have a runtime consistent with the runtime of the TTS-generated audio; and
responsive to the runtime of the initial video clip being consistent with the runtime of the TTS-generated audio, using the initial video clip as the final video clip; and
combining the TTS-generated audio with the final video clip to generate an event highlight video.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium or media of claim 16 wherein the step of obtaining a time mapping by performing time anchoring to correlate runtime of the input video with runtime of the activity comprises:
using optical character recognition on a set of video frames of the input video to read time of a clock displayed in the input video;
given the recognized time of the clock, generating a set of time anchors, which comprises a time of the activity and a set of temporal shifts, if any; and
generating the time mapping that maps time of the clock with the input video runtime using at least some of the time anchors in the set of time anchors.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium or media of claim 17 wherein the step of generating an initial video clip from the input video that includes the event by using timing information related to the activity and the time mapping obtained by time anchoring to identify an approximate time in the input video of when the event occurred during the activity comprises:
parsing data from metadata to obtain an approximate time for the event; and
using the approximate time for the event and the time mapping to generate the initial video clip, which includes the event.

### Claim 19 (depends on 16)

The non-transitory computer-readable medium or media of claim 16 wherein the steps of extracting features from the initial video clip and obtaining a final time value of the event in the initial video clip using the extracted features and one or more trained neural network models comprise:
performing feature extraction on the initial video clip using a set of two or more models; and
obtaining the final time value of the event in the initial video clip using an ensemble neural network model that receives input related to the features from the set of two or more models and outputs the final time value.

### Claim 20 (depends on 16)

The non-transitory computer-readable medium or media of claim 16 further comprising one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising:
given a supplemental text about the activity or event,
parsing the supplemental text, using a text parsing module, to identify information related to the event referenced in the supplemental text; and
using the information extracted from the supplemental text to help identify a time of occurrence of the event.