# US 2021/0081497 A1 · Cognitively generating information from videos

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081497 A1
- Document ID: US20210081497A1
- Status: Published application
- Kind: A1
- Application number: 16/571,451
- Filed: 2019-09-16
- Published: 2021-03-18
- Earliest claimed: 2019-09-16
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Wei Qin; Jing Jing Zhang; Xi Juan Men; Xiaoli Duan; Yue Chen; Dong Jun Zong
- Web page: https://patentgrep.com/patent/US20210081497A1

## CPC

- G06F 40/35: Discourse or dialogue representation
- G06F 40/295: Named entity recognition
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/174: {Facial expression recognition}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)}
- H04N 21/8405: represented by keywords
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

In an approach for detecting key messages for a video, a processor builds a role model based on data from one or more data sources, with an identification feature of each role in a video. A processor samples a plurality of frames of the video. A processor identifies a key object presented in the plurality of frames. The key object is a role in the video. A processor recognizes a movement scenario associated with the role. A processor dynamically updates the role model based on the movement scenario. A processor identifies a role name based on the movement scenario. A processor generates a description script associated with the movement scenario for the role. A processor outputs the description script.

## Claims

### Claim 1

A computer-implemented method comprising:
building, by one or more processors, a role model, based on data from one or more data sources, with an identification feature of each role in a video, the role model comprising a machine learning model trained to recognize each role by each respective identification feature;
sampling, by one or more processors, a plurality of frames of the video;
identifying, by one or more processors, a key object presented in the plurality of frames, the key object being a role in the video;
recognizing, by one or more processors, a movement scenario associated with the role, including:
generating a frame description for each frame through a first neural network,
removing information from each frame that is not associated with the role in the frame,
upon removing the information that is not associated with the role, entering the plurality of frames, in a timeline sequence, to a second neural network, and
generating the movement scenario associated with the role in the timeline sequence, the movement scenario being a text description associated with the movement;
dynamically updating, by one or more processors, the role model based on the movement scenario;
identifying, by one or more processors, a role name based on the movement scenario;
generating, by one or more processors, a description script associated with the movement scenario for the role; and
outputting, by one or more processors, the description script.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the identification feature is a facial feature.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the first neural network is a convolutional neural network.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the second neural network is a recurrent neural network.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein dynamically updating the role model is based on conversation scripts generated by using natural language processing techniques.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein identifying the role names is based on conversation scripts generated by using natural language processing techniques.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein outputting the description script includes outputting the description script by audio.

### Claim 8

A computer program product comprising:
one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions comprising:
program instructions to build a role model, based on data from one or more data sources, with an identification feature of each role in a video, the role model comprising a machine learning model trained to recognize each role by each respective identification feature;
program instructions to sample a plurality of frames of the video;
program instructions to identify a key object presented in the plurality of frames, the key object being a role in the video;
program instructions to recognize a movement scenario associated with the role, including: program instructions, stored on the one or more computer-readable storage media,
to generate a frame description for each frame through a first neural network,
to remove information from each frame that is not associated with the role in the frame,
upon removing the information that is not associated with the role, to enter the plurality of frames, in a timeline sequence, to a second neural network, and
to generate the movement scenario associated with the role in the timeline sequence, the movement scenario being a text description associated with the movement;
program instructions to dynamically update the role model based on the movement scenario;
program instructions to identify a role name based on the movement scenario;
program instructions to a description script associated with the movement scenario for the role; and
program instructions to output the description script.

### Claim 9 (depends on 8)

The computer program product of claim 8, wherein the identification feature is a facial feature.

### Claim 10 (depends on 8)

The computer program product of claim 8, wherein the first neural network is a convolutional neural network.

### Claim 11 (depends on 8)

The computer program product of claim 8, wherein the second neural network is a recurrent neural network.

### Claim 12 (depends on 8)

The computer program product of claim 8, wherein program instructions to dynamically update the role model are based on conversation scripts generated by using natural language processing techniques.

### Claim 13 (depends on 8)

The computer program product of claim 8, wherein program instructions to identify role names are based on conversation scripts generated by using natural language processing techniques.

### Claim 14 (depends on 8)

The computer program product of claim 8, wherein program instructions to output the description script include program instructions to output the description script by audio.

### Claim 15

A computer system comprising:
one or more computer processors, one or more computer readable storage media, and program instructions stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising:
program instructions to build a role model, based on data from one or more data sources, with an identification feature of each role in a video, the role model comprising a machine learning model trained to recognize each role by each respective identification feature;
program instructions to sample a plurality of frames of the video;
program instructions to identify a key object presented in the plurality of frames, the key object being a role in the video;
program instructions to recognize a movement scenario associated with the role, including: program instructions, stored on the one or more computer-readable storage media,
to generate a frame description for each frame through a first neural network,
to remove information from each frame that is not associated with the role in the frame,
upon removing the information that is not associated with the role, to enter the plurality of frames, in a timeline sequence, to a second neural network, and
to generate the movement scenario associated with the role in the timeline sequence, the movement scenario being a text description associated with the movement;
program instructions to dynamically update the role model based on the movement scenario;
program instructions to identify a role name based on the movement scenario;
program instructions to a description script associated with the movement scenario for the role; and
program instructions to output the description script.

### Claim 16 (depends on 15)

The computer system of claim 15, wherein the identification feature is a facial feature.

### Claim 17 (depends on 15)

The computer system of claim 15, wherein the first neural network is a convolutional neural network.

### Claim 18 (depends on 15)

The computer system of claim 15, wherein the second neural network is a recurrent neural network.

### Claim 19 (depends on 15)

The computer system of claim 15, wherein program instructions to dynamically update the role model are based on conversation scripts generated by using natural language processing techniques.

### Claim 20 (depends on 15)

The computer system of claim 15, wherein program instructions to output the description script include program instructions to output the description script by audio.
