# US 12,608,621 B2 · Generating artificial video with changed domain > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,608,621 B2 - Document ID: US12608621B2 - Status: Granted - Kind: B2 - Application number: 18/065,672 - Filed: 2022-12-14 - Published: 2024-06-20 - Granted: 2026-04-21 - Earliest claimed: 2022-12-14 - Assignees: Ford Global Technologies, LLC - Applicants: Ford Global Technologies, LLC - Inventors: Akhil Perincherry; Arpita Chand - Web page: https://patentgrep.com/patent/US12608621B2 ## CPC - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods} - G06V 10/776: Validation; Performance evaluation - G06V 10/7788: {the supervisor being a human, e.g. interactive learning with a human teacher} - G06V 10/806: {of extracted features} - G06V 10/82: using neural networks - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06N 3/045: Combinations of networks - G06N 3/047: Probabilistic or stochastic networks - G06N 3/08: Learning methods - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06T 9/002: {using neural networks} - H04N 5/265: Mixing - H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability - H04N 19/172: the region being a picture, frame or field - H04N 19/182: the unit being a pixel - H04N 19/42: characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A computer includes a processor and a memory, and the memory stores instructions executable by the processor to receive an input video of a scene and audio data associated with the input video, the input video being in a first domain; execute an encoder to map the input video and the audio data to a latent vector in a lower-dimensional latent space; and execute a generator to generate an output video of the scene from the latent vector, the output video being in a second domain. The encoder and the generator are trained to maintain temporal consistency between the input video and the output video by using the audio data. ## Claims ### Claim 1 A computer comprising a processor and a memory, the memory storing instructions executable by the processor to: receive an input video of a scene and audio data associated with the input video, the input video being in a first domain; execute an encoder to map the input video and the audio data to a latent vector in a lower-dimensional latent space; and execute a generator to generate an output video of the scene from the latent vector, the output video being in a second domain; wherein the encoder and the generator are trained to maintain temporal consistency between the input video and the output video by using the audio data. ### Claim 2 (depends on 1) The computer of claim 1, wherein the encoder and the generator are supervised by a discriminator during training. ### Claim 3 (depends on 2) The computer of claim 2, wherein the discriminator supervises the training of the encoder and the generator by testing a consistency of the output video with the audio data. ### Claim 4 (depends on 3) The computer of claim 3, wherein, while training the encoder and the generator, the discriminator uses a correlation between the output video and the audio data to test the consistency of the output video with the audio data. ### Claim 5 (depends on 4) The computer of claim 4, wherein, while training the encoder and the generator, the discriminator receives the correlation from a correlation module, the correlation module being pretrained with contrastive learning. ### Claim 6 (depends on 2) The computer of claim 2, wherein the discriminator supervises the training of the encoder and the generator by testing a consistency of the output video with the second domain. ### Claim 7 (depends on 6) The computer of claim 6, wherein the instructions further include instructions to determine an adversarial loss based on an output of the discriminator and to update the encoder and the generator based on the adversarial loss. ### Claim 8 (depends on 1) The computer of claim 1, wherein the first domain and the second domain are mutually exclusive environmental conditions of the scene. ### Claim 9 (depends on 8) The computer of claim 8, wherein the environmental conditions are one of a lighting condition or a weather condition. ### Claim 10 (depends on 1) The computer of claim 1, wherein the first domain and the second domain are mutually exclusive visual rendering characteristics of the input video and output video. ### Claim 11 (depends on 10) The computer of claim 10, wherein the visual rendering characteristics are one of a resolution, a color representation scheme, or simulatedness. ### Claim 12 (depends on 1) The computer of claim 1, wherein the instructions further include instructions to extract visual features from the input video; and executing the encoder is based on the visual features. ### Claim 13 (depends on 12) The computer of claim 12, wherein the instructions further include instructions to extract audio features from the audio data and to fuse the visual features and the audio features; and executing the encoder is based on the fusion of the visual features and the audio features. ### Claim 14 (depends on 1) The computer of claim 1, wherein the instructions further include instructions to extract audio features from the audio data; and executing the encoder is based on the audio features. ### Claim 15 (depends on 1) The computer of claim 1, wherein the encoder is trained to include semantic content of the input video in the latent vector and to exclude domain data of the input video from the latent vector. ### Claim 16 (depends on 1) The computer of claim 1, wherein the encoder is a first encoder; the generator is a first generator; the latent vector is a first latent vector; and training the first encoder and the first generator includes executing a second encoder to map the output video and the audio data to a second latent vector in the lower-dimensional latent space, and executing a second generator to generate a test video of the scene from the second latent vector in the first domain. ### Claim 17 (depends on 16) The computer of claim 16, wherein training the first encoder and the first generator includes updating the first encoder and the first generator based on a difference between the test video and the input video. ### Claim 18 (depends on 1) The computer of claim 1, wherein the instructions further include instructions to train a machine-learning model on training data; and the training data includes the output video. ### Claim 19 (depends on 18) The computer of claim 18, wherein the machine-learning model is an object-recognition model. ### Claim 20 A method comprising: receiving an input video of a scene and audio data associated with the input video, the input video being in a first domain; executing an encoder to map the input video and the audio data to a latent vector in a lower-dimensional latent space; and executing a generator to generate an output video of the scene from the latent vector, the output video being in a second domain; wherein the encoder and the generator are trained to maintain temporal consistency between the input video and the output video by using the audio data.