# US 2025/0166358 A1 · Machine-learning-based detection of fake videos

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166358 A1
- Document ID: US20250166358A1
- Status: Published application
- Kind: A1
- Application number: 18/744,440
- Filed: 2024-06-14
- Published: 2025-05-22
- Earliest claimed: 2023-11-17
- Assignees: Reality Defender, Inc.
- Applicants: Reality Defender, Inc.
- Inventors: Gaurav BHARAJ; Trevine OORLOFF; Surya KOPPISETTI; Nicolò BONETTINI; Ben COLMAN; Ali SHAHRIYARI
- Web page: https://patentgrep.com/patent/US20250166358A1

## CPC

- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 21/64: Protecting data integrity, e.g. using checksums, certificates or signatures
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/094: Adversarial learning
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06V 20/95: {Pattern authentication; Markers therefor; Forgery detection}
- G06V 40/172: {Classification, e.g. identification}
- G06V 40/40: Spoof detection, e.g. liveness detection
- G10L 17/02: Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
- G10L 17/04: Training, enrolment or model building
- G10L 17/26: Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
- G10L 25/57: for processing of video signals
- G06F 21/32: using biometric data, e.g. fingerprints, iris scans or voiceprints
- G10L 17/18: Artificial neural networks; Connectionist approaches
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method for training a model for classifying videos as real or fake can include generating image tiles and audio data segments from an input video, generating a sequence of image embeddings based on the image tiles using a visual encoder and a sequence of audio embeddings based on the audio data segments using an audio encoder, transforming, using a V2A network, a first subset of the sequence of image embeddings into synthetic audio embeddings, transforming, using an A2V network, a first subset of the sequence of audio embeddings into synthetic image embeddings, updating the sequence of image embeddings by using the synthetic image embeddings, updating the sequence of audio embeddings using the synthetic audio embeddings, training the encoders and the networks using the updated sequences of image embeddings and audio embeddings, and training a classifier using the trained encoders and the trained networks.

## Claims

### Claim 1

A method for training a model comprising a visual encoder, an audio encoder, an audio-to-visual (A2V) network, a visual-to-audio (V2A) network, and a classifier for classifying videos as real or fake, the method comprising:
generating a sequence of image tiles from image data from an input video;
generating a plurality of data segments representing audio data from the input video;
generating a sequence of image embeddings based on the sequence of image tiles using the visual encoder;
generating a sequence of audio embeddings based on the sequence of data segments using the audio encoder;
transforming, using the V2A network, a first subset of the sequence of image embeddings into one or more synthetic audio embeddings;
transforming, using the A2V network, a first subset of the sequence of audio embeddings into one or more synthetic image embeddings;
updating the sequence of image embeddings by replacing a second subset of the sequence of image embeddings with the one or more synthetic image embeddings;
updating the sequence of audio embeddings by replacing a second subset of the sequence of audio embeddings with the one or more synthetic audio embeddings;
training the visual encoder, the audio encoder, the V2A network, and the A2V network based on the updated sequence of image embeddings and the updated sequence of audio embeddings;
training the classifier to classify videos as real or fake using the trained visual encoder, the trained audio encoder, the trained V2A network, and the trained A2V network, wherein the classifier is configured to receive image embeddings for the videos from the trained visual encoder, audio embeddings for the videos from the trained audio encoder, synthetic image embeddings for the videos from the trained A2V network, and synthetic audio embeddings for the videos from the trained V2A network.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the first subset of the sequence of image embeddings comprises half of the image embeddings, and
the first subset of the sequence of audio embeddings comprises half of the audio embeddings.

### Claim 3 (depends on 1)

The method of claim 1, wherein the first subset of the sequence of image embeddings and the first subset of the sequence of audio embeddings are randomly selected.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
the first subset of the sequence of image embeddings corresponds to a first set of time points in the input video, and
the first subset of the sequence of audio embeddings corresponds to a second set of time points in the input video different from the first set of time points.

### Claim 5 (depends on 4)

The method of claim 4, wherein:
the second subset of the sequence of image embeddings corresponds to the second set of time points, and
the second subset of the sequence of audio embeddings corresponds to the first set of time points.

### Claim 6 (depends on 1)

The method of claim 1, wherein training the visual encoder, the audio encoder, the V2A network, and the A2V network based on the updated sequence of image embeddings and the updated sequence of audio embeddings comprises:
decoding the updated sequence of image embeddings to produce a reconstruction of the sequence of image tiles; and
decoding the updated sequence of audio embeddings to produce a reconstruction of the plurality of data segments.

### Claim 7 (depends on 6)

The method of claim 6, wherein the updated sequence of image embeddings is decoded using a visual decoder.

### Claim 8 (depends on 6)

The method of claim 6, wherein the updated sequence of audio embeddings is decoded using an audio decoder.

### Claim 9 (depends on 1)

The method of claim 1, wherein training the visual encoder, the audio encoder, the V2A network, and the A2V network based on the updated sequence of image embeddings and the updated sequence of audio embeddings further comprises computing a dual-objective loss, wherein a first objective of the dual-objective loss depends on the sequence of audio embeddings and the sequence of image embeddings and a second objective of the dual-objective loss depends on the sequence of image tiles, the plurality of data segments, the reconstruction of the sequence of image tiles, and the reconstruction of the plurality of data segments.

### Claim 10 (depends on 1)

The method of claim 1, wherein training the classifier to classify videos as real or fake using the trained visual encoder, the trained audio encoder, the trained V2A network, and the trained A2V network comprises:
generating a second sequence of image tiles from image data from a labeled training video comprising a label indicating whether the labeled training video is real or fake;
generating a second plurality of data segments representing audio data from the labeled training video;
generating a second sequence of image embeddings based on the second sequence of image tiles using the trained visual encoder;
generating a second sequence of audio embeddings based on the second sequence of data segments using the trained audio encoder;
transforming, using the trained V2A network, the second sequence of image embeddings into a sequence of synthetic audio embeddings;
transforming, using the trained A2V network, the second sequence of audio embeddings into a sequence synthetic image embeddings;
concatenating the second sequence of image embeddings and the sequence of synthetic image embeddings to produce a combined sequence of image embeddings;
concatenating the second sequence of audio embeddings and the sequence of synthetic audio embeddings to produce a combined sequence of audio embeddings; and
classifying the labeled training video as real or fake based on the combined sequence of audio embeddings and the combined sequence of image embeddings.

### Claim 11 (depends on 10)

The method of claim 10, wherein training the classifier to classify videos as real or fake using the trained visual encoder, the trained audio encoder, the trained V2A network, and the trained A2V network further comprises computing a cross-entropy loss objective using label indicating whether the labeled training video is real or fake.

### Claim 12 (depends on 1)

The method of claim 1, wherein the classifier comprises an audio mode patch reduction network, a visual mode patch reduction network, and a classifier head.

### Claim 13 (depends on 1)

The method of claim 1, wherein a number of image tiles in the sequence of image tiles and a number of data segments in the sequence of data segments are determined based on a sampling frequency of the image data, a sampling frequency of the audio data, and a time duration of the input video.

### Claim 14 (depends on 1)

The method of claim 1, further comprising:
providing the trained model with a second input video; and
classifying the second input video as real or fake using the trained model.

### Claim 15 (depends on 14)

The method of claim 14, wherein the second input video comprises real audio data and AI-generated image data.

### Claim 16 (depends on 14)

The method of claim 14, wherein the second input video comprises real image data and AI-generated audio data.

### Claim 17 (depends on 14)

The method of claim 14, wherein the second input video comprises AI-generated image data and AI-generated audio data.

### Claim 18 (depends on 1)

The method of claim 1, wherein the input video shows a human face.

### Claim 19

A system for training a model comprising a visual encoder, an audio encoder, an audio-to-visual (A2V) network, a visual-to-audio (V2A) network, and a classifier for classifying videos as real or fake, the system comprising one or more processors configured to:
generate a sequence of image tiles from image data from an input video;
generate a plurality of data segments representing audio data from the input video;
generate a sequence of image embeddings based on the sequence of image tiles using the visual encoder;
generate a sequence of audio embeddings based on the sequence of data segments using the audio encoder;
transform, using the V2A network, a first subset of the sequence of image embeddings into one or more synthetic audio embeddings;
transform, using the A2V network, a first subset of the sequence of audio embeddings into one or more synthetic image embeddings;
update the sequence of image embeddings by replacing a second subset of the sequence of image embeddings with the one or more synthetic image embeddings;
update the sequence of audio embeddings by replacing a second subset of the sequence of audio embeddings with the one or more synthetic audio embeddings;
train the visual encoder, the audio encoder, the V2A network, and the A2V network based on the updated sequence of image embeddings and the updated sequence of audio embeddings;
train the classifier to classify videos as real or fake using the trained visual encoder, the trained audio encoder, the trained V2A network, and the trained A2V network, wherein the classifier is configured to receive image embeddings for the videos from the trained visual encoder, audio embeddings for the videos from the trained audio encoder, synthetic image embeddings for the videos from the trained A2V network, and synthetic audio embeddings for the videos from the trained V2A network.

### Claim 20

A non-transitory computer readable storage medium storing instructions for training a model comprising a visual encoder, an audio encoder, an audio-to-visual (A2V) network, a visual-to-audio (V2A) network, and a classifier for classifying videos as real or fake that, when executed by one or more processors of a computer system, cause the computer system to:
generate a sequence of image tiles from image data from an input video;
generate a plurality of data segments representing audio data from the input video;
generate a sequence of image embeddings based on the sequence of image tiles using the visual encoder;
generate a sequence of audio embeddings based on the sequence of data segments using the audio encoder;
transform, using the V2A network, a first subset of the sequence of image embeddings into one or more synthetic audio embeddings;
transform, using the A2V network, a first subset of the sequence of audio embeddings into one or more synthetic image embeddings;
update the sequence of image embeddings by replacing a second subset of the sequence of image embeddings with the one or more synthetic image embeddings;
update the sequence of audio embeddings by replacing a second subset of the sequence of audio embeddings with the one or more synthetic audio embeddings;
train the visual encoder, the audio encoder, the V2A network, and the A2V network based on the updated sequence of image embeddings and the updated sequence of audio embeddings;
train the classifier to classify videos as real or fake using the trained visual encoder, the trained audio encoder, the trained V2A network, and the trained A2V network, wherein the classifier is configured to receive image embeddings for the videos from the trained visual encoder, audio embeddings for the videos from the trained audio encoder, synthetic image embeddings for the videos from the trained A2V network, and synthetic audio embeddings for the videos from the trained V2A network.
