# US 2022/0044459 A1 · System and method for intelligently generating digital composites from user-provided graphics

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044459 A1
- Document ID: US20220044459A1
- Status: Published application
- Kind: A1
- Application number: 16/984,608
- Filed: 2020-08-04
- Published: 2022-02-10
- Earliest claimed: 2020-08-04
- Assignees: Triple Lift, Inc.
- Applicants: Triple Lift, Inc.
- Inventors: Shaun T. Zacharia; Samuel Benjamin Shapiro; Alexander Prokofiev; Luis Manuel Bracamontes Hernandez
- Web page: https://patentgrep.com/patent/US20220044459A1

## CPC

- H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06T 7/11: Region-based segmentation
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/234345: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment}
- H04N 21/23439: {for generating different versions}
- H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)}
- G06Q 30/0241: Advertisements
- G06T 2207/20084: Artificial neural networks [ANN]
- H04N 21/4314: {for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A system and method for inserting a composited image or otherwise generated graphic into a selected video by way of a programmatic process. According to some embodiments, a system may comprise an Automated Placement Opportunity Identification (APOI) engine, a Placement Insertion Interface (PII) engine, a preview system, and an automated compositing service. The system finalizes a graphic composite into a video and provides a user with a preview for final export or further manipulation.

## Claims

### Claim 1

A method of digital image composition, the method comprising:
receiving, by a computer processor, a primary image asset comprising a plurality of areas of interest;
automatically identifying, by the computer processor, first and second ones of the areas of interest to include in a composite image, wherein the first area of interest is identified as at least a portion of a surface of an object depicted in the primary image asset;
training a model to recognize one or more features of interest in an image asset using a plurality of image assets comprising pre-labelled features of interest;
receiving, by the computer processor, a secondary image asset comprising one or more features of interest;
automatically identifying, by the computer processor, a first one of the one or more features of interest in the secondary image asset to include in the composite image by providing as input to the model the secondary image asset and receiving as output from the model the first one of the one or more features of interest; and
generating, by the computer processor, the composite image by combining at least a portion of the primary image asset that includes the first and second ones of the areas of interest with at least a portion of the secondary image asset that includes the first one of the one or more features of interest, wherein the combining comprises compositing the at least a portion of the secondary image asset and the at least a portion of the primary image asset, and wherein the compositing comprises overlaying the at least a portion of the secondary image asset on the at least a portion of the surface of the object.

### Claim 2 (depends on 1)

The method of claim 1, wherein automatically identifying the first area of interest comprises:
extracting, by way of one or more neural networks, characteristics of the primary image asset, the characteristics representative of visually flat surfaces located at a particular location in the primary image asset;
generating, by the computer processor, a confidence value for each of the characteristics of the primary image asset;
determining, by the computer processor, a best characteristic, wherein the best characteristic comprises a highest one of the confidence values; and
labelling, by way of the one or more neural networks, the particular location associated with the best characteristic as the first area of interest.

### Claim 3 (depends on 1)

The method of claim 1, wherein automatically identifying the second area of interest comprises:
extracting, by way of one or more neural networks, characteristics of the primary image asset, the characteristics indicative of a particular location in the primary image asset;
generating, by the computer processor, a confidence value for each of the characteristics of the primary image asset;
determining, by the computer processor, a best characteristic, wherein the best characteristic comprises a highest one of the confidence values;
labelling, by way of the one or more neural networks, the particular location associated with the best characteristic as the first area of interest.

### Claim 4 (depends on 3)

The method of claim 3, wherein the characteristics of the primary image asset include one or more of the following:
common objects visually present within the primary image asset;
text visually present within the primary image asset;
categorical data representative of the scene as presented in the primary image asset; and
audio data comprising recognizable words or speech provided with the primary image asset.

### Claim 5 (depends on 1)

The method of claim 1, wherein automatically identifying the first feature of interest comprises automatic logo identification as provided by one or more neural networks.

### Claim 6 (depends on 1)

The method of claim 1, wherein creating the composite image further comprises manipulating dimensions of the composited image assets to match a predetermined output dimension.

### Claim 7 (depends on 1)

The method of claim 1, wherein the primary image asset is indicative of a digital video asset comprising a series of image assets, wherein the method is programmatically repeated for each image asset of the series.

### Claim 8 (depends on 7)

The method of claim 7, wherein each series of image assets are extracted from a digital video asset by:
receiving, at the computer processor, the digital video asset;
processing, by way of one or more neural networks, pixels of the digital video asset;
identifying, by the one or more neural networks, a first shot boundary of the digital video asset and a second boundary of the digital video asset;
extracting, by the computer processor, one or more video frames located between the first shot boundary and the second shot boundary of the digital video asset; and
generating, by the computer processor, a series of image assets from the one or more video frames as extracted.

### Claim 9

A system for compositing digital images, comprising:
a processor; and
a non-transitory computer-readable medium for storing computer-executable instructions that, when executed by the processor, program the processor to perform operations comprising:
receiving a primary image asset comprising a plurality of areas of interest;
automatically identifying first and second ones of the areas of interest to include in a composite image, wherein the first area of interest is identified as at least a portion of a surface of an object depicted in the primary image asset;
training a model to recognize one or more features of interest in an image asset using a plurality of image assets comprising pre-labelled features of interest;
receiving a secondary image asset comprising one or more features of interest;
automatically identifying a first one of the one or more features of interest in the secondary image asset to include in the composite image by providing as input to the model the secondary image asset and receiving as output from the model the first one of the one or more features of interest; and
generating the composite image by combining at least a portion of the primary image asset that includes the first and second ones of the areas of interest with at least a portion of the secondary image asset that includes the first one of the one or more features of interest, wherein the combining comprises compositing the at least a portion of the secondary image asset and the at least a portion of the primary image asset, and wherein the compositing comprises overlaying the at least a portion of the secondary image asset on the at least a portion of the surface of the object.

### Claim 10 (depends on 9)

The system of claim 9, wherein automatically identifying the first area of interest comprises:
extracting characteristics of the primary image asset using one or more neural networks, the characteristics representative of visually flat surfaces located at a particular location in the primary image asset;
generating a confidence value for each of the characteristics of the primary image asset;
determining a best characteristic, wherein the best characteristic comprises the highest confidence value;
labelling the particular location associated with the best characteristic as the first area of interest using the one or more neural networks.

### Claim 11 (depends on 9)

The system of claim 9, wherein automatically identifying the second area of interest comprises:
extracting characteristics of the primary image asset using the one or more neural networks, the characteristics indicative of a particular location in the primary image asset;
generating a confidence value for each of the characteristics of the primary image asset;
determining a best characteristic, wherein the best characteristic comprises the highest confidence value;
labelling the particular location associated with the best characteristic as the first area of interest using the one or more neural networks.

### Claim 12 (depends on 9)

The system of claim 9, wherein the characteristics of the primary image asset include one or more of the following:
common objects visually present within the primary image asset;
text visually present within the primary image asset;
categorical data representative of the scene as presented in the primary image asset; and
audio data comprising recognizable words or speech provided with the primary image asset.

### Claim 13 (depends on 9)

The system of claim 9, wherein automatically identifying the first feature of interest comprises automatic logo identification as provided by one or more neural networks.

### Claim 14 (depends on 9)

The system of claim 9, wherein creating the composite image further comprises manipulating dimensions of the composited image assets to match a predetermined output dimension.

### Claim 15 (depends on 9)

The system of claim 9, wherein the primary image asset is indicative of a digital video asset comprising a series of image assets, wherein the operations are programmatically repeated for each image asset of the series.

### Claim 16 (depends on 15)

The system of claim 15, wherein the series of image assets are extracted from the digital video asset by operations comprising:
receiving the digital video asset;
processing pixels of the digital video asset using the one or more neural networks;
identifying a first shot boundary of the digital video asset and a second boundary of the digital video asset using the one or more neural networks;
extracting one or more video frames located between the first shot boundary and the second shot boundary of the digital video asset;
generating a series of image assets from the one or more video frames as extracted.
