# US 10,482,607 B1 · Systems and methods for motion correction in synthetic images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,607 B1
- Document ID: US10482607B1
- Status: Granted
- Kind: B1
- Application number: 16/263,839
- Filed: 2019-01-31
- Granted: 2019-11-19
- Earliest claimed: 2018-07-06
- Assignees: Capital One Services, LLC
- Applicants: Capital One Services, LLC
- Inventors: Austin Walters; Jeremy Goodsitt
- Web page: https://patentgrep.com/patent/US10482607B1

## CPC

- G06F 9/541: {via adapters, e.g. between incompatible applications}
- G06F 8/71: Version control (security arrangements therefor G06F21/57); Configuration management
- G06F 9/54: Interprogram communication
- G06F 9/547: {Remote procedure calls [RPC]; Web services}
- G06F 11/3608: {using formal methods, e.g. model checking, abstract interpretation (theorem proving G06N5/013)}
- G06F 11/3628: {of optimised code (optimisation G06F8/443)}
- G06F 11/3636: {by tracing the execution of the program}
- G06F 11/3684: {for test design, e.g. generating new test cases}
- G06F 11/3688: {for test execution, e.g. scheduling of test suites}
- G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
- G06F 16/2237: {Vectors, bitmaps or matrices}
- G06F 16/2264: {Multidimensional index structures}
- G06F 16/2423: {Interactive query statement specification based on a database schema}
- G06F 16/24568: {Data stream processing; Continuous queries}
- G06F 16/248: Presentation of query results
- G06F 16/254: {Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses}
- G06F 16/258: {Data format conversion from or to a database}
- G06F 16/283: {Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP}
- G06F 16/285: {Clustering or classification}
- G06F 16/288: {Entity relationship models}
- G06F 16/335: Filtering based on additional data, e.g. user or group profiles (filtering in web context G06F16/9535, G06F16/9536)
- G06F 16/35: Clustering; Classification
- G06F 16/90332: {Natural language query formulation or dialogue systems}
- G06F 16/90335: {Query processing}
- G06F 16/9038: Presentation of query results
- G06F 16/906: Clustering; Classification
- G06F 16/93: Document management systems
- G06F 17/15: Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)}
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06F 18/2115: by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
- G06F 18/213: Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/2193: {based on specific statistical tests}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/23: Clustering techniques
- G06F 18/24: Classification techniques
- G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines
- G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
- G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system}
- G06F 18/40: Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
- G06F 21/552: {involving long-term monitoring or reporting}
- G06F 21/60: Protecting data
- G06F 21/6245: {Protecting personal data, e.g. for financial or medical purposes}
- G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification}
- G06F 30/20: Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30)
- G06F 40/117: Tagging; Marking up (details of markup languages G06F40/143); Designating a block; Setting of attributes (style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154)
- G06F 40/166: Editing, e.g. inserting or deleting
- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/06: Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 5/00: Computing arrangements using knowledge-based models
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 5/02: Knowledge representation; Symbolic representation
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 5/04: Inference or reasoning models
- G06N 7/00: Computing arrangements based on specific mathematical models
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N 20/20: Ensemble learning
- G06Q 10/04: Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" (market predictions or forecasting for commercial activities G06Q30/0202)
- G06T 7/194: involving foreground-background segmentation
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/248: {involving reference images or patches}
- G06T 7/254: involving subtraction of images
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06V 10/768: {using context analysis, e.g. recognition aided by known co-occurring patterns}
- G06V 10/993: {Evaluation of the quality of the acquired pattern}
- G06V 30/194: References adjustable by an adaptive method, e.g. learning
- G06V 30/1985: {Syntactic analysis, e.g. using a grammatical approach (syntactic image representation G06V30/18171)}
- H04L 63/1416: {Event detection, e.g. attack signature detection}
- H04L 63/1491: {using deception as countermeasure, e.g. honeypots, honeynets, decoys or entrapment}
- H04L 67/306: User profiles
- H04L 67/34: {involving the movement of software or configuration parameters (network booting or remote initial program loading [RIPL] G06F9/4416)}
- H04N 21/23412: {for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects}
- H04N 21/8153: {comprising still images, e.g. texture, background image}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Systems and methods for generating synthetic video are disclosed. For example, the system may include one or more memory units storing instructions and one or more processors configured to execute the instructions to perform operations. The operations may include generating a static background image and determining the location of a reference edge. The operations may include determining a perspective of an observation point. The operations may include generating synthetic difference images that include respective synthetic object movement edges. The operations may include determining a location of the respective synthetic object movement edge and generating adjusted difference images corresponding to the individual synthetic difference images. Adjusted difference images may be based on synthetic difference images, locations of the respective synthetic object movement edges, the perspective of the observation point, and the location of the reference edge. The operations may include generating texturized images based on the adjusted difference images.

## Claims

### Claim 1

A system for generating synthetic video comprising:
one or more memory units storing instructions; and
one or more processors configured to execute the instructions to perform operations comprising:
generating a static background image;
determining a location of a reference edge in the static background image;
determining a perspective of an observation point;
generating, based on the static background image, synthetic difference images comprising a respective synthetic object movement edge;
determining, for individual synthetic difference images, a location of the respective synthetic object movement edge;
generating adjusted difference images corresponding to the individual synthetic difference images, the adjusted difference images being based on:
the corresponding individual synthetic difference images,
the determined locations of the respective synthetic object movement edges of the corresponding individual synthetic difference images,
the perspective of the observation point, and
the location of the reference edge; and
generating texturized images based on the adjusted difference images.

### Claim 2 (depends on 1)

The system of claim 1, wherein generating adjusted difference images comprises using a method of gradient domain processing and using a method of homography.

### Claim 3 (depends on 1)

The system of claim 1, wherein the one or more processors are configured to execute the instructions to perform operations further comprising:
receiving video data comprising a sequence of images;
generating background difference images based on the sequence of images; and
generating merged difference images based on the adjusted difference images and the background difference images,
wherein:
generating the static background image is based on the sequence of images, and
generating the texturized images is based on the merged difference images.

### Claim 4 (depends on 3)

The system of claim 3, wherein generating merged difference images comprises:
determining, for individual adjusted difference images, a location of a respective adjusted object movement edge;
determining, for individual background difference images, a location of a respective background movement object edge; and
merging the individual adjusted difference images with the individual background difference images based on:
the locations of the respective adjusted object movement edges of the individual adjusted difference images, and
the locations of the respective background movement object edges of the individual background difference images.

### Claim 5 (depends on 3)

The system of claim 3, wherein the one or more processors are configured to execute the instructions to perform operations further comprising generating a sequence of normalized images based on the sequence of images and the static background image, wherein generating the background difference images comprises applying a method of image subtraction to the normalized sequence of images.

### Claim 6 (depends on 1)

The system of claim 1, wherein generating individual adjusted difference images comprises scaling the respective synthetic object movement edge of the corresponding synthetic difference image.

### Claim 7 (depends on 1)

The system of claim 1, wherein generating individual adjusted difference images comprises layering a synthetic object movement edge and a background object edge.

### Claim 8 (depends on 1)

The system of claim 1, wherein the one or more processors are configured to execute the instructions to perform operations further comprising:
determining luminance values of the static background image; and
estimating a location and brightness of a light source based on the luminance values of the static background image,
wherein generating adjusted difference images comprises adjusting luminance values of the corresponding individual synthetic difference images based on the location and brightness of the light source.

### Claim 9 (depends on 8)

The system of claim 8, wherein the one or more processors are configured to execute the instructions to perform operations further comprising:
receiving video data comprising a sequence of images, and
generating a converted sequence of images based on the received sequence of images, the converted sequence of images having luminance channel data,
wherein:
generating the static background image is based on the converted sequence of images, and
determining luminance values of the static background image is based on the luminance channel data of the converted sequence of images.

### Claim 10 (depends on 1)

The system of claim 1, wherein determining the location of a reference edge in the static background image comprises:
identifying a plurality of edges in the static background image;
identifying a reference object based on the plurality of edges;
identifying a reference edge of the reference object;
determining a plurality of background edge distances between with the reference edge and the plurality of edges; and
determining a location of the reference edge in the static background image based on the plurality of background edge distances.

### Claim 11 (depends on 10)

The system of claim 10, wherein the observation point is a first observation point, the adjusted difference images are first adjusted difference images corresponding to a first subset of the individual synthetic difference images, and the texturized images are first texturized images, the operations further comprising:
determining a perspective of a second observation point;
generating second adjusted difference images corresponding to a second subset of the individual synthetic difference images, the second adjusted difference images being based on:
a corresponding synthetic difference image,
the location of the respective synthetic object movement edge of the corresponding synthetic difference image,
the perspective of the second observation point, and
the location of the reference edge; and
generating second texturized images based on the second adjusted difference images.

### Claim 12 (depends on 10)

The system of claim 10, the operations further comprising estimating a size of the reference object, wherein determining the plurality of background edge distances is based on the estimated size of the reference object.

### Claim 13 (depends on 10)

The system of claim 10, wherein the one or more processors are configured to execute the instructions to perform operations further comprising generating a three-dimensional model of the static background image based on the plurality of background edge distances, wherein determining the location of the reference edge is based on the three-dimensional model.

### Claim 14 (depends on 1)

The system of claim 1, wherein the one or more processors are configured to execute the instructions to perform operations further comprising generating a three-dimensional spatial model of the individual synthetic difference images based on the perspective of the observation point and the location of the reference edge, wherein the adjusted difference images are based on the three-dimensional spatial model.

### Claim 15 (depends on 1)

The system of claim 1, wherein the synthetic object movement edge is associated with a shadow of a synthetic object, and wherein the generating adjusted difference images comprises determining transparency values associated with the shadow.

### Claim 16 (depends on 1)

The system of claim 1, wherein generating synthetic difference images is performed using a machine learning model comprising at least one of an autoencoder model, a recurrent neural network model, or a convolutional neural network model.

### Claim 17 (depends on 1)

The system of claim 1, wherein generating adjusted difference images is performed using a machine learning model comprising at least one of an autoencoder model, a recurrent neural network model, or a convolutional neural network model.

### Claim 18 (depends on 1)

The system of claim 1, wherein generating merged difference images is performed using a machine learning model comprising at least one of an autoencoder model, a recurrent neural network model, or a convolutional neural network model.

### Claim 19

A method of generating synthetic video comprising:
generating a static background image;
determining a location of a reference edge in the static background image;
determining the perspective of an observation point;
generating, based on the static background image, synthetic difference images comprising a respective synthetic object movement edge;
determining, for individual synthetic difference images, a location of the respective synthetic object movement edge;
generating adjusted difference images corresponding to the individual synthetic difference images, the adjusted difference images being based on:
the corresponding individual synthetic difference images,
the locations of the respective synthetic object movement edges of the corresponding individual synthetic difference images,
the perspective of the observation point, and
the location of the reference edge; and
generating texturized images based on the adjusted difference images.

### Claim 20

A system for generating synthetic video comprising:
one or more memory units storing instructions; and
one or more processors configured to execute the instructions to perform operations comprising:
generating a static background image;
identifying a plurality of edges of the static background image;
generating a three-dimensional model of the static background image using homography, the three-dimensional model being based on the plurality of edges;
determining a perspective of an observation point;
determining luminance values of the static background image;
estimating a location and brightness of a light source based on the luminance values;
generating, based on the static background image, synthetic difference images comprising a respective plurality of synthetic object movement edges;
determining, for individual synthetic difference images, a plurality of locations of the plurality of respective synthetic object movement edges;
generating, using gradient domain processing and homography, adjusted difference images corresponding to the individual synthetic difference images, the adjusted difference images being based on:
a corresponding synthetic difference image,
the locations of the respective plurality of synthetic object movement edges of the corresponding synthetic difference image,
the perspective of the observation point,
the three-dimensional model of the static background image, and
the location and brightness of the light source; and
generating texturized images based on the adjusted difference images.