# US 12,217,470 B2 · System and method for automatic video reconstruction with dynamic point of interest

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,470 B2
- Document ID: US12217470B2
- Status: Granted
- Kind: B2
- Application number: 17/698,309
- Filed: 2022-03-18
- Published: 2022-06-30
- Granted: 2025-02-04
- Earliest claimed: 2020-12-28
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Muchlisin Adi Saputra; Shah Dehan Lazuardi; Billy Gunawan; Irfan Yunus Pramono; Soma Wiraga Saniscara; Junaidillah Fadlil
- Web page: https://patentgrep.com/patent/US12217470B2

## CPC

- G06V 10/235: {based on user input or interaction}
- H04N 21/854: Content authoring
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V 30/19: Recognition using electronic means
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G11B 27/002: {Programmed access in sequence to a plurality of record carriers or indexed parts, e.g. tracks, thereof, e.g. for editing; (transfer of record carriers from magazine G11B15/68, G11B17/10; G11B17/22)}
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- H04N 21/23412: {for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects}
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/234318: {by decomposing into objects, e.g. MPEG-4 objects}
- H04N 21/234345: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment}
- H04N 21/23439: {for generating different versions}
- H04N 21/8405: represented by keywords
- H04N 21/845: Structuring of content, e.g. decomposing content into time segments
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

A system and a method for an automatic video reconstruction to improve scene quality using a dynamic point of interest by finding a point or line of interest are provided. The method includes dividing a first video into a plurality of first frames; determining a first object of interest in the plurality of first frames; converting the plurality of first frames into a plurality of second frames based on the first object of interest; and reconstructing the first video into a second video based on the plurality of second frames.

## Claims

### Claim 1

A method of automatically generating video reconstruction, the method comprising:
dividing a first video into a plurality of first frames;
determining a first object of interest in the plurality of first frames based on a user selection on a user interface or based on an intent recognition module using a natural language processing, wherein the user interface provides a preview of objects present in the plurality of first frames, and wherein the intent recognition module processes a text to determine whether an intent of the text is selecting to focus on a particular object or selecting an unwanted object;
converting the plurality of first frames into a plurality of second frames based on the first object of interest; and
reconstructing the first video into a second video based on the plurality of second frames,
wherein, in a case of determining the first object of interest based on the intent recognition module, the converting the plurality of first frames comprises:
extracting at least one of a point of interest or a line of interest for the first object of interest included in a first frame of the plurality of first frames; and
if the intent of the text is selecting the unwanted object, deleting the first object of interest from the plurality of first frames to obtain the plurality of second frames not including the first object of interest; or
if the intent of the text is selecting to focus on the particular object, reconstructing the first frame based on the at least one of the point of interest or the line of interest to obtain the plurality of second frames including the first object of interest.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the dividing the first video comprises:
dividing the first video into a plurality of scenes based on images included in the first video or a text externally input, and
wherein the determining the first object of interest comprises:
detecting a second object included in the plurality of scenes and tracking the second object; and
classifying a foreground and a background in the plurality of scenes, and determining the second object as the first object of interest based on a result of the classifying.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the dividing the first video into the plurality of scenes comprises:
detecting voices included in the plurality of first frames through automatic speech recognition (ASR), and converting the voices into text;
dividing the images included in the plurality of first frames based on at least one of a color, a shape, or a gradation of each of the images; and
generating a feature vector for each of the converted text and the divided images, and dividing the first video into the plurality of scenes based on the feature vector.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the determining the first object of interest comprises:
determining the first object of interest based on an intent recognition and an entity recognition.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the converting the plurality of first frames comprises:
extracting at least one of a point of interest or a line of interest for a third object included in the first frame of the plurality of first frames; and
cutting the third object included in the first frame or reconstructing the first frame based on the at least one of the point of interest or the line of interest.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the reconstructing the first frame comprises:
fitting a template to the first frame, the template including five points and three straight lines; and
moving the template such that the point of interest or the line of interest is adjacent to or coincides with the five points or the three straight lines.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the converting the plurality of first frames comprises:
removing a partial region of the first frame of the plurality of first frames;
generating a second frame of the plurality of second frames by painting a missing area resulted from removal of the partial region; and
arranging adjacent second frames by applying in-painting and flow estimation to the plurality of second frames.

### Claim 8

A system for automatically generating video reconstruction, the system comprising:
a display configured to output a first video, and output a second video in which the first video is reconstructed; and
a processor configured to process data for the first video and reconstruct the second video,
wherein the processor is further configured to divide the first video into a plurality of first frames, determine a first object of interest from the plurality of first frames, and divide the plurality of first frames into a plurality of second frames based on the first object of interest, and reconstruct the first video into the second video based on the plurality of second frames,
wherein the processor is further configured to determine the first object of interest based on a user selection on a user interface or based on an intent recognition module using a natural language processing, wherein the user interface provides a preview of objects present in the plurality of first frames, and wherein the intent recognition module processes a text to determine whether an intent of the text is selecting to focus on a particular object or selecting an unwanted object,
wherein, in a case of determining the first object of interest based on the intent recognition module, the processor is further configured to convert the plurality of first frames by performing:
extracting at least one of a point of interest or a line of interest for the first object of interest included in a first frame of the plurality of first frames; and
if the intent of the text is selecting the unwanted object, deleting the first object of interest from the plurality of first frames to obtain the plurality of second frames not including the first object of interest; or
if the intent of the text is selecting to focus on the particular object, reconstructing the first frame based on the at least one of the point of interest or the line of interest to obtain the plurality of second frames including the first object of interest.

### Claim 9 (depends on 8)

The system according to claim 8, wherein the processor is further configured to divide the first video into a plurality of scenes based on images included in the first video or a text externally input; detect a second object included in the plurality of scenes and tracking the second object; and classify a foreground and a background in the plurality of scenes, and determining the second object as the first object of interest based on a result of classification.

### Claim 10 (depends on 9)

The system according to claim 9, wherein the processor is further configured to detect voices included in the plurality of first frames through automatic speech recognition (ASR), and converting the voices into text, divide the images included in the plurality of first frames based on at least one of a color, a shape, or a gradation of each of the images; and generate a feature vector for each of the converted text and the divided images, and divide the first video into the plurality of scenes based on the feature vector.

### Claim 11 (depends on 8)

The system according to claim 8, wherein the processor is further configured to determine the first object of interest based on an intent recognition and an entity recognition.

### Claim 12 (depends on 8)

The system according to claim 8, wherein the processor is further configured to extract at least one of a point of interest or a line of interest for a third object included in the first frame of the plurality of first frames; and cut the third object included in the first frame or reconstructing the first frame based on the at least one of the point of interest or the line of interest.

### Claim 13 (depends on 12)

The system according to claim 12, wherein the processor is further configured to fit a template to the first frame, the template including five points and three straight lines; and move the template such that the point of interest or the line of interest is adjacent to or coincides with the five points or the three straight lines.

### Claim 14 (depends on 8)

The system according to claim 8, wherein the processor is further configured to remove a partial region of the first frame of the plurality of first frames, generate a second frame of the plurality of second frames by painting a missing area resulted from removal of the partial region; and arrange adjacent second frames by applying in-painting and flow estimation to the plurality of second frames.

### Claim 15

A computer program product comprising a non-transitory computer-readable medium storing instructions that, when executed by at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
dividing a first video into a plurality of first frames;
determining a first object of interest in the plurality of first frames based on a user selection on a user interface or based on an intent recognition module using a natural language processing, wherein the user interface provides a preview of objects present in the plurality of first frames, and wherein the intent recognition module processes a text to determine whether an intent of the text is selecting to focus on a particular object or selecting an unwanted object;
converting the plurality of first frames into a plurality of second frames based on the first object of interest; and
reconstructing the first video into a second video based on the plurality of second frames,
wherein, in a case of determining the first object of interest based on the intent recognition module, the converting the plurality of first frames comprises:
extracting at least one of a point of interest or a line of interest for the first object of interest included in a first frame of the plurality of first frames; and
if the intent of the text is selecting the unwanted object, deleting the first object of interest from the plurality of first frames to obtain the plurality of second frames not including the first object of interest; or
if the intent of the text is selecting to focus on the particular object, reconstructing the first frame based on the at least one of the point of interest or the line of interest to obtain the plurality of second frames including the first object of interest.