# US 2018/0025749 A1 · Automatic generation of semantic-based cinemagraphs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0025749 A1
- Document ID: US20180025749A1
- Status: Published application
- Kind: A1
- Application number: 15/265,461
- Filed: 2016-09-14
- Published: 2018-01-25
- Earliest claimed: 2016-07-22
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Tae-Hyun Oh; Sing Bing Kang; Neel Suresh Joshi; Baoyuan Wang
- Web page: https://patentgrep.com/patent/US20180025749A1

## CPC

- G11B 27/007: {reproducing continuously a part of the information, i.e. repeating}
- G06F 18/23: Clustering techniques
- G06F 18/24133: {Distances to prototypes}
- G06T 7/11: Region-based segmentation
- G06T 13/80: Two-dimensional [2D] animation, e.g. using sprites
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 30/19173: {Classification techniques}
- G11B 27/002: {Programmed access in sequence to a plurality of record carriers or indexed parts, e.g. tracks, thereof, e.g. for editing; (transfer of record carriers from magazine G11B15/68, G11B17/10; G11B17/22)}
- G11B 27/034: on discs (G11B27/036, G11B27/038 take precedence)
- G11B 27/105: {of operating discs}
- H04N 9/8227: {the additional signal being at least another television signal}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Described are various technologies that pertain to automatically generating looped videos or cinemagraphs by selecting objects to animate from an input video. In one implementation a group of semantically labeled objects from an input video is received. Candidate objects from the input video that can appear as a moving object in an output cinemagraph or looped video are identified. Candidate video loops are generated using the selected candidate objects. One or more of these candidate video loops are then selected to create a final cinemagraph. The selection of candidate video loops used to create the final cinemagraph can be made by a user or by a predictive model trained to evaluate the subjective attractiveness of the candidate video loops.

## Claims

### Claim 1

A system for generating a cinemagraph of one or more video loops, comprising:
a cinemagraph generator comprising one or more computing devices, said computing devices being in communication with each other via a computer network whenever there is a plurality of computing devices, and a computer program having a plurality of sub-programs executable by said computing devices, wherein the sub-programs configure said computing devices for,
receiving an input video, wherein the input video comprises a sequence of frames each frame of which comprises pixels,
if the input video is not semantically segmented, semantically segmenting the frames of the input video to identify regions in the frames that correspond to semantic objects,
selecting semantic objects as candidate objects to animate;
generating candidate loops of the selected candidate objects;
selecting one or more of the candidate loops to be used to create the cinemagraph; and
creating a cinemagraph from the selected candidate loops using the input time intervals computed for the pixels of the frames of the input video, wherein the cinemagraph exhibits regions that appear static to a viewer and regions comprising dynamic video loops of the selected candidate objects that appear to the viewer to be changing over time.

### Claim 2 (depends on 1)

The system of claim 1, wherein the sub-program to select one or more of the candidate objects to be used to create the cinemagraph selects the candidate objects by:
receiving for a set of semantic objects identified in the input, video per-pixel semantic labels of all of the pixels in the input video;
constructing histograms by sum-pooling the label maps across all of the frames and pixels to generate a global object label histogram;
discarding object labels that correspond to inherently static objects;
discarding object labels that have low dynamicity in terms of intensity variation across time or motion;
discarding object labels for which connected components are too small in average; and
selecting the top K object labels in terms of high frequency of the global object label histogram as the candidate objects that will appear as moving objects in each candidate video-loop.

### Claim 3 (depends on 2)

The system of claim 2 wherein a candidate object candidate can be a single object type or a combination of possible candidate object types.

### Claim 4 (depends on 1)

The system of claim 1 wherein the candidate loops are generated by:
generating rough explicit masks of the region of each specific candidate object;
constructing a feature set for each mask region associated with a candidate object;
clustering the features in the feature set and selecting the centroids of the clusters as the feature basis that represents the object label for the masked region;
using the feature basis for calculated object candidate looping start times and looping periods, encouraging pixels that have a similar feature basis and a similar representative feature to be dynamic;
creating candidate loops using the dynamic pixels.

### Claim 5 (depends on 1)

The system of claim 1 wherein the candidate loops are selected by using a predictive model that ranks the subjective attractiveness of each candidate loop.

### Claim 6 (depends on 1)

The system of claim 1, further comprising computing input time intervals for the pixels of the frames of the input video, wherein:
an input time interval for a particular pixel comprises a per-pixel loop period and a per-pixel start time of a loop at the particular pixel, and
the input time interval of a pixel is based, in part, on one or more semantic factors which keep pixels associated with the same semantic object in the same video loop.

### Claim 7 (depends on 6)

The system of claim 6, wherein the input time intervals computed for the pixels of the frames of the input video are used to animate the selected candidate loops.

### Claim 8

A computer-implemented process for evaluating the attractiveness of a looped image sequence, comprising:
using a computing device for:
receiving a set of training candidates of looped image sequences for determining a human subjective quality rating of a looped image sequence;
automatically extracting a plurality of features from each training candidate in the set of training candidates;
receiving a human subjective quality rating for each of the training candidates;
generating a predictive model that evaluates the subjective attractiveness of a cinemagraph from a combination of the features extracted from each training candidate and the corresponding human subjective quality ratings of the training candidate.

### Claim 9 (depends on 8)

The computer-implemented process of claim 8, further comprising using the predictive model to rank a set of candidate loops for generating a looped image sequence by:
receiving a set comprising a plurality of candidate loops for generating a looped image sequence;
applying the predictive model to the candidate loops of the candidate set to generate a quality score for each candidate loop, the quality score defining a subjective quality of the corresponding candidate loop; and
ranking the candidate loops based on the quality scores;
automatically generating a looping image sequence using one or more of the highest ranked candidate loops.

### Claim 10 (depends on 9)

The computer-implemented process of claim 9, wherein the quality score of a candidate loop exceeds a predetermined threshold in order to be ranked.

### Claim 11 (depends on 8)

The computer-implemented process of claim 8 wherein the features further comprise features relevant to face, sharpness, motion and loopability.

### Claim 12 (depends on 9)

The computer-implemented process of claim 9, wherein the generated looping image sequence is used to generate one or more portions of a cinemagraph.

### Claim 13

A system for generating a cinemagraph of one or more video loops, comprising:
a cinemagraph generator comprising one or more computing devices, said computing devices being in communication with each other via a computer network whenever there is a plurality of computing devices, and a computer program having a plurality of sub-programs executable by said computing devices, wherein the sub-programs configure said computing devices for,
receiving semantic object labels extracted from an input video as candidate objects to animate;
generating candidate loops of the selected candidate objects;
selecting one or more of the candidate loops to be used to create the cinemagraph; and
creating a cinemagraph from the selected candidate loops.

### Claim 14 (depends on 13)

The system of claim 13 further comprising using input time intervals and looping periods computed for the pixels of the semantic objects to determine the candidate loops.

### Claim 15 (depends on 13)

The system of claim 13 wherein the cinemagraph exhibits regions that appear static to a viewer and regions comprising dynamic video loops of the selected candidate objects that appear to the viewer to be changing over time.

### Claim 16 (depends on 13)

The system of claim 13, wherein one or more of the candidate objects used to create the cinemagraph are selected by:
receiving for a set of semantic objects identified in the input video per-pixel semantic labels of all of the pixels in the input video;
constructing histograms by sum-pooling the label maps across all of the frames and pixels;
discarding object labels that correspond to inherently static objects;
discarding object labels that have low dynamicity in terms of intensity variation across time or motion;
discarding object labels for which connected components are too small in average; and
selecting the top K object labels in terms of high frequency of the histogram as the candidate objects that will appear as moving objects in each candidate video-loop.

### Claim 17 (depends on 13)

The system of claim 13 wherein the candidate loops are selected by using a predictive model trained to assess the subjective attractiveness of looping types of objects in cinemagraph.

### Claim 18 (depends on 13)

The system of claim 13, wherein the candidate loops are generated by using a Markov Random Field operation.

### Claim 19 (depends on 13)

The system of claim 13, wherein one or more of the candidate loops used to create the cinemagraph are selected by a human being.

### Claim 20 (depends on 13)

The system of claim 13, wherein the one or more of the candidate loops used to create the cinemagraph are automatically selected by a predictive model.
