# US 12,002,490 B2 · Method and electronic device for a slow motion video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,490 B2
- Document ID: US12002490B2
- Status: Granted
- Kind: B2
- Application number: 17/853,384
- Filed: 2022-06-29
- Published: 2023-01-05
- Granted: 2024-06-04
- Earliest claimed: 2021-06-30
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Vishwajeet Shukla; Manisha Meena; Mayank Singour; Ishan Pandita
- Web page: https://patentgrep.com/patent/US12002490B2

## CPC

- G11B 27/005: {Reproducing at a different information rate from the information rate of recording (for television signals H04N5/783)}
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for generating a slow motion video. The method includes segmenting, by an electronic device, objects in the video. Further, the method includes determining, by the electronic device, an interaction between the segmented objects. Further, the method includes clustering, by the electronic device, the segmented objects in the video to generate object clusters based on the interaction. Further, the method includes determining, by the electronic device, a degree of slow motion effect to be applied to each of the object clusters in the video based on a significance score of each of the object clusters. Further, the method includes generating, by the electronic device, the slow motion video by applying the degree of slow motion effect to that has been determined to corresponding the object clusters.

## Claims

### Claim 1

A method for generating a slow motion video, the method comprising:
segmenting, by an electronic device, objects in the video;
determining, by the electronic device, an interaction between the segmented objects;
clustering, by the electronic device, the segmented objects in the video to generate object clusters based on the interaction;
determining, by the electronic device, a degree of slow motion effect to be applied to each of the object clusters in the video based on a significance score of each of the object clusters; and
generating, by the electronic device, the slow motion video by applying the degree of slow motion effect to that has been determined to corresponding the object clusters.

### Claim 2 (depends on 1)

The method as claimed in claim 1, wherein the segmenting of the objects in the video comprises:
determining, by the electronic device, a class confidence score of each video frame of the video using a Machine Learning model, the class confidence score being a quantitative measurement of a quality of content in the video frame;
filtering, by the electronic device, the video frames based on the class confidence scores;
identifying, by the electronic device, the objects in the filtered video frames;
estimating, by the electronic device, a depth score of each of the identified objects in the filtered video frames based on depth information in the filtered video frames;
determining, by the electronic device, an optical flow rate of each of the identified objects in the filtered video frames, the optical flow rate being a quantitative measurement of a velocity of movement of pixels corresponding to the identified objects in the filtered video frames; and
generating, by the electronic device, the segmented objects from the filtered video frames by applying the optical flow rate and the depth score of each identified object to a semantic segmentation network.

### Claim 3 (depends on 2)

The method as claimed in claim 2, wherein the determining of the interaction comprises:
determining, by the electronic device, the interaction between the segmented objects for a time duration by applying the optical flow rate and the depth score of each of the segmented objects to a memory based neural network.

### Claim 4 (depends on 3)

The method as claimed in claim 3, wherein the determining of the interaction comprises:
categorizing, by the electronic device, the segmented objects to one of an interacting class in response to determining the interaction between the segmented objects, and a non-interacting class in response to not determining the interaction between the segmented objects.

### Claim 5 (depends on 3)

The method as claimed in claim 3, wherein the memory based neural network outputs a probability score indicating whether the segmented objects are interacting.

### Claim 6 (depends on 1)

The method as claimed in claim 1, wherein the determining of the degree of slow motion effect comprises:
determining, by the electronic device, the significance score of each of the object clusters, the significance score being a measure of a relative importance of each of the object clusters in the video; and
determining the degree of slow motion effect for each of the objects based on the corresponding the significance score for the object clusters.

### Claim 7 (depends on 6)

The method as claimed in claim 6, wherein the determining of the significance score comprises:
determining, by the electronic device, a cluster flow rate of each of the object clusters based on an optical flow rate of each segmented object in the corresponding object clusters;
generating, by the electronic device, a hash table of a proximity change coefficient β based on the change in clustering of the segmented objects at the different instants of time in the video, the proximity change coefficient β being a number of times that the segmented object changes into and out of the object clusters within a time duration; and
determining, by the electronic device, the significance score of each of the object clusters based on based on at least one of the cluster flow rate and the proximity change coefficient β.

### Claim 8 (depends on 6)

The method as claimed in claim 6, wherein the determining of the significance score comprises:
estimating, by the electronic device, a rigidness score of each of the segmented objects based on at least one of a relative frequency of the segmented objects occurring in the video, a perceptual aesthetic quality of the segmented objects and a degree of movement of the segmented objects based on the object classes of the segmented objects, the rigidness score being a measure of static aspects of an affinity for slow motion of the segmented object; and
determining, by the electronic device, the significance score of each of the object clusters based on the rigidness score.

### Claim 9 (depends on 8)

The method as claimed in claim 8, wherein the rigidness score is estimated using a weighted combination of the relative frequency of the segmented objects occurring in the video, the perceptual aesthetic quality of the segmented objects, and the degree of movement of the segmented objects based on the object classes of the segmented objects, weights thereof being determined using regression analysis.

### Claim 10 (depends on 8)

The method as claimed in claim 8, wherein the perceptual aesthetic quality of the segmented objects is determined as a probability using a neural network that is trained on a dataset of images labeled with a score of aesthetic values between 0 and 1.

### Claim 11 (depends on 6)

The method as claimed in claim 6, wherein the generating of the slow motion video comprises:
determining, by the electronic device, an interpolation rate for each of a plurality of sections in the video frames that correspond to the object clusters, based on the significance score of the object cluster corresponding to the section; and
interpolating, by the electronic device, the object clusters in each video frame based on the interpolation rate of the corresponding sections.

### Claim 12

An electronic device for generating a slow motion video, the electronic device comprising:
a memory;
a processor; and
a sectional slow motion controller, coupled to the memory and the processor, the sectional slow motion controller configured to perform operations comprising:
segmenting objects in the video,
determining an interaction between the segmented objects,
clustering the segmented objects in the video to generate object clusters based on the interaction,
determining a degree of slow motion effect to be applied to each of the object clusters in the video based on a significance score of each of the object clusters,
generating the slow motion video by applying the degree of slow motion effect that has been determined to corresponding the object clusters.

### Claim 13 (depends on 12)

The electronic device as claimed in claim 12, wherein the segmenting of the objects comprises:
determining a class confidence score of each video frame of the video using a Machine Learning model, the class confidence score being a quantitative measurement of a quality of content in the video frame;
filtering the video frames based on the class confidence scores;
identifying the objects in the filtered video frames;
estimating a depth score of each of the identified objects in the filtered video frames based on depth information in the filtered video frames;
determining an optical flow rate of each of the identified objects in the filtered video frames, the optical flow rate being a quantitative measurement of a velocity of movement of pixels corresponding to the identified objects in the filtered video frames; and
generating the segmented objects from the filtered video frames by applying the optical flow rate and the depth score of each identified object to a semantic segmentation network.

### Claim 14 (depends on 13)

The electronic device as claimed in claim 13, wherein the determining of the interaction comprises:
determining the interaction between the segmented objects for a time duration by applying the optical flow rate and the depth score of each of the segmented objects to a memory based neural network.

### Claim 15 (depends on 14)

The electronic device as claimed in claim 14, wherein the determining of the interaction comprises:
categorizing the segmented objects to one of an interacting class in response to determining the interaction between the segmented objects, and a non-interacting class in response to not determining the interaction between the segmented objects.

### Claim 16 (depends on 12)

The electronic device as claimed in claim 12, wherein the determining of the degree of slow motion effect comprises:
determining the significance score of each of the object clusters, the significance score being a measure of a relative importance of each of the object clusters in the video; and
determining the degree of slow motion effect for each of the objects based on the corresponding the significance score for the object clusters.

### Claim 17 (depends on 16)

The electronic device as claimed in claim 16, wherein the determining of the significance score comprises:
determining a cluster flow rate of each of the object clusters based on an optical flow rate of each segmented object in the corresponding object clusters,
generating a hash table of a proximity change coefficient based on the change in clustering of the segmented objects at the different instants of time in the video, the proximity change coefficient being a number of times that the segmented object changes into and out of the object clusters within a time duration, and
determining the significance score of each of the object clusters based on based on at least one of the cluster flow rate and the proximity change coefficient.

### Claim 18 (depends on 16)

The electronic device as claimed in claim 16, wherein the determining of the significance score comprises:
estimating, by the electronic device, a rigidness score of each of the segmented objects based on at least one of a relative frequency of the segmented objects occurring in the video, a perceptual aesthetic quality of the segmented objects and a degree of movement of the segmented objects based on the object classes of the segmented objects, the rigidness score being a measure of static aspects of an affinity for slow motion of the segmented object, and
determining the significance score of each of the object clusters based on the rigidness score.

### Claim 19 (depends on 16)

The electronic device as claimed in claim 16, wherein the generating of the slow motion video comprises determining an interpolation rate for each of a plurality of sections in the video frames that correspond to the object clusters, based on the significance score of the object cluster corresponding to the section and interpolating the object clusters in each video frame based on the interpolation rate of the corresponding sections.

### Claim 20 (depends on 1)

A non-transitory computer-readable storage medium, having a computer program stored thereon that performs, when executed by a processor, the method according to claim 1.