# US 2022/0012500 A1 · Device and method for generating summary video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0012500 A1
- Document ID: US20220012500A1
- Status: Published application
- Kind: A1
- Application number: 17/372,276
- Filed: 2021-07-09
- Published: 2022-01-13
- Earliest claimed: 2020-07-09
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Junkyu HAN; Pius LEE; Hyunuk TAK
- Web page: https://patentgrep.com/patent/US20220012500A1

## CPC

- G06V 20/47: {Detecting features for summarising video content}
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G10L 25/57: for processing of video signals
- G10L 25/63: for estimating an emotional state
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method for generating a summary video includes generating a user emotion graph of a user watching a first video. The method also includes obtaining a character emotion graph for a second video, by analyzing an emotion of a character in a second video that is a target of summarization. The method further includes obtaining an object emotion graph for an object in the second video, based on an object appearing in the second video. Additionally the method includes obtaining an image emotion graph for the second video, based on the character emotion graph and the object emotion graph. The method also includes selecting at least one first scene in the second video by comparing the user emotion graph with the image emotion graph. The method further includes generating the summary video of the second video, based on the at least one first scene.

## Claims

### Claim 1

A method, performed by a device, of generating a summary video, the method comprising:
obtaining a user image in which a user watching a first video is photographed, during playback of the first video;
generating a user emotion graph of the user watching the first video, by analyzing an emotion of the user in the obtained user image;
obtaining a character emotion graph for a second video, by analyzing an emotion of a character in the second video that is a target of summarization;
obtaining an object emotion graph for an object in the second video, based on the object appearing in the second video;
obtaining an image emotion graph for the second video, based on the character emotion graph and the object emotion graph;
selecting at least one first scene in the second video by comparing the user emotion graph of the user that watched the first video with the image emotion graph for the second video; and
generating the summary video of the second video, based on the at least one first scene.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
selecting at least one second scene in the second video, based on emotion scores in the image emotion graph,
wherein the generating of the summary video comprises generating the summary video of the second video, based on the at least one first scene and the at least one second scene.

### Claim 3 (depends on 2)

The method of claim 2, wherein selecting the at least one first scene comprises:
selecting a first part of the image emotion graph by comparing a slope of emotion curves in the user emotion graph with a slope of emotion curves in the image emotion graph, and
selecting a scene corresponding to the selected first part as the at least one first scene to be included in the summary video.

### Claim 4 (depends on 2)

The method of claim 2, wherein selecting the at least one second scene comprises:
selecting a second part comprising an emotion score greater than a certain threshold value when an emotion score of at least one preset emotion in the image emotion graph is greater than the certain threshold value, and
selecting a scene corresponding to the selected second part as the at least one second scene to be included in the summary video.

### Claim 5 (depends on 2)

The method of claim 2, wherein generating the summary video comprises combining frames corresponding to the at least one first scene with frames corresponding to the at least one second scene.

### Claim 6 (depends on 1)

The method of claim 1, wherein the user emotion graph is generated based on a user emotion score calculated based on at least one of a facial expression or a voice of the user watching the first video.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
the character emotion graph is generated based on a character emotion score calculated for a frame or a scene of the second video, and
obtaining the character emotion graph comprises calculating the character emotion score for a frame or a scene of the second video, by applying different weight values to characters in the second video.

### Claim 8 (depends on 1)

The method of claim 1, wherein obtaining the object emotion graph comprises:
obtaining previously generated object emotion information based on at least one of a facial expression, a voice, or a line of a character at a time of appearance of the object in at least one third video;
calculating an object emotion score for the object appearing in the second video for a frame or a scene of the second video, based on the obtained object emotion information; and
generating an object emotion graph for the second video, based on the calculated object emotion score.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
identifying a sound for a frame or a scene in the second video; and
obtaining a sound emotion graph for a sound output from the second video, based on the identified sound,
wherein obtaining the image emotion graph comprises obtaining the image emotion graph for the second video, based on the character emotion graph, the object emotion graph, and the sound emotion graph.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
identifying a line for a frame or a scene in the second video; and
obtaining a line emotion graph for a line output from the second video, based on the identified line,
wherein obtaining the image emotion graph comprises obtaining the image emotion graph for the second video, based on the character emotion graph, the object emotion graph, the sound emotion graph, and the line emotion graph.

### Claim 11

A device for generating a summary video, the device comprising:
a communication interface;
a display;
a memory storing one or more instructions; and
a processor configured to:
obtain a user image in which a user watching a first video is photographed, during playback of the first video through the display,
generate a user emotion graph of the user watching the first video, by analyzing an emotion of the user in the obtained user image,
obtain a character emotion graph for a second video, by analyzing an emotion of a character in the second video that is a target of summarization,
obtain an object emotion graph for an object in the second video, based on an object appearing in the second video,
obtain an image emotion graph for the second video, based on the character emotion graph and the object emotion graph,
select at least one first scene in the second video by comparing the user emotion graph of the user that watched the first video with the image emotion graph for the second video, and
generate the summary video of the second video, based on the at least one first scene.

### Claim 12 (depends on 11)

The device of claim 11, wherein the processor is further configured to:
select at least one second scene in the second video based on emotion scores in the image emotion graph; and
generate the summary video of the second video based on the at least one first scene and the at least one second scene.

### Claim 13 (depends on 12)

The device of claim 12, wherein the processor is further configured to:
select a first part of the image emotion graph by comparing a slope of emotion curves in the user emotion graph with a slope of emotion curves in the image emotion graph; and
select a scene corresponding to the selected first part as the at least one first scene to be included in the summary video.

### Claim 14 (depends on 12)

The device of claim 12, wherein the processor is further configured to:
select a second part including an emotion score greater than a certain threshold value when an emotion score of at least one preset emotion in the image emotion graph is greater than the certain threshold value; and
select a scene corresponding to the selected second part as the at least one second scene to be included in the summary video.

### Claim 15 (depends on 12)

The device of claim 12, wherein to generate the summary video, the processor is further configured to combine frames corresponding to the at least one first scene with frames corresponding to the at least one second scene.

### Claim 16 (depends on 11)

The device of claim 11, wherein the user emotion graph is generated based on a user emotion score calculated based on at least one of a facial expression or a voice of the user watching the first video.

### Claim 17 (depends on 11)

The device of claim 11, wherein:
the character emotion graph is generated based on a character emotion score calculated for a frame or a scene of the second video, and
the processor is further configured to: calculate the character emotion score for a frame or a scene of the second video, by applying different weight values to characters in the second video.

### Claim 18 (depends on 11)

The device of claim 11, wherein the processor is further configured to:
obtain previously generated object emotion information based on at least one of a facial expression, a voice, or a line of a character at a time of appearance of the object in at least one third video,
calculate an object emotion score for the object appearing in the second video for a frame or a scene of the second video, based on the obtained object emotion information, and
generate an object emotion graph for the second video, based on the calculated object emotion score.

### Claim 19 (depends on 11)

The device of claim 11, wherein the processor is further configured to:
identify a sound for a frame or a scene in the second video; and
obtain a sound emotion graph for a sound output from the second video, based on the identified sound,
wherein the image emotion graph for the second video is obtained, based on the character emotion graph, the object emotion graph, and the sound emotion graph.

### Claim 20

A non-transitory computer-readable recording medium embodying a program, the program comprising computer readable program code that when executed by a processor of an electronic device, causes the processor to:
obtain a user image in which a user watching a first video is photographed, during playback of the first video through a display,
generate a user emotion graph of the user watching the first video, by analyzing an emotion of the user in the obtained user image,
obtain a character emotion graph for a second video, by analyzing an emotion of a character in the second video that is a target of summarization,
obtain an object emotion graph for an object in the second video, based on an object appearing in the second video,
obtain an image emotion graph for the second video, based on the character emotion graph and the object emotion graph,
select at least one first scene in the second video by comparing the user emotion graph of the user that watched the first video with the image emotion graph for the second video, and
generate a summary video of the second video, based on the at least one first scene.
