# US 2023/0410505 A1 · Video manipulation detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410505 A1
- Document ID: US20230410505A1
- Status: Published application
- Kind: A1
- Application number: 17/845,353
- Filed: 2022-06-21
- Published: 2023-12-21
- Earliest claimed: 2022-06-21
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Ritwik Sinha; Viswanathan Swaminathan; Trisha Mittal; John Philip Collomosse
- Web page: https://patentgrep.com/patent/US20230410505A1

## CPC

- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06V 10/00: Arrangements for image or video recognition or understanding (character recognition in images or video G06V30/10)
- G06V 20/44: {Event detection}
- G06V 40/18: Eye characteristics, e.g. of the iris
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Techniques for video manipulation detection are described to detect one or more manipulations present in digital content such as a digital video. A detection system, for instance, receives a frame of a digital video that depicts at least one entity. Coordinates of the frame that correspond to a gaze location of the entity are determined, and the detection system determines whether the coordinates correspond to a portion of an object depicted in the frame to calculate a gaze confidence score. A manipulation score is generated that indicates whether the digital video has been manipulated based on the gaze confidence score. In some examples, the manipulation score is based on at least one additional confidence score.

## Claims

### Claim 1

A method comprising:
receiving, by a processing device, a frame of a digital video that depicts at least one entity;
identifying, by the processing device, coordinates of the frame that correspond to a gaze location of the at least one entity using gaze tracking;
determining, by the processing device, whether the coordinates of the frame correspond to a portion of an object depicted in the frame;
calculating, by the processing device, a gaze confidence score based on the determining; and
generating, by the processing device, a manipulation score indicating whether the digital video has been manipulated based on the gaze confidence score and at least one additional confidence score.

### Claim 2 (depends on 1)

The method as described in claim 1, further comprising:
determining that the digital video has been manipulated based on the manipulation score; and
generating an indication of a spatial location of a manipulation in the digital video for display in a user interface.

### Claim 3 (depends on 1)

The method as described in claim 1, wherein the at least one additional confidence score includes a visual artifact confidence score calculated by leveraging a convolutional neural network to detect resolution inconsistencies associated with affine warpings.

### Claim 4 (depends on 1)

The method as described in claim 1, wherein the at least one additional confidence score includes a temporal confidence score calculated by determining an optical flow between adjacent frames of a plurality of frames of the digital video.

### Claim 5 (depends on 4)

The method as described in claim 4, wherein the temporal confidence score is calculated by comparing audial features of the digital video to visual features of the digital video.

### Claim 6 (depends on 1)

The method as described in claim 1, wherein the at least one additional confidence score includes an affective state confidence score calculated by leveraging a machine learning model to detect discrepancies in affective states for the at least one entity.

### Claim 7 (depends on 6)

The method as described in claim 6, wherein the affective states are partially based on facial expressions of the at least one entity, body postures of the at least one entity, or contextual data associated with the digital video.

### Claim 8 (depends on 1)

The method as described in claim 1, wherein the manipulation score is a maximum of the gaze confidence score and the at least one additional confidence score.

### Claim 9 (depends on 1)

The method as described in claim 1, wherein generating the manipulation score includes applying a weighting to at least one of the gaze confidence score, a visual artifact confidence score, a temporal confidence score, or an affective state confidence score.

### Claim 10

A system comprising:
a detection module implemented by one or more processing devices to:
receive a frame of a digital video that depicts at least one entity;
identify coordinates of the frame that correspond to a gaze location of the at least one entity using gaze tracking;
determine whether the coordinates of the frame correspond to a portion of an object depicted in the frame;
calculate a gaze confidence score based on the determining; and
generate a manipulation score indicating whether the digital video has been manipulated based on the gaze confidence score and one or more of a visual artifact confidence score, a temporal confidence score, or an affective state confidence score.

### Claim 11 (depends on 10)

The system as described in claim 10, the detection module further configured to:
determine that the digital video is manipulated based on the manipulation score being greater than a threshold; and
generate an indication of a type of manipulation present in the digital video for display in a user interface.

### Claim 12 (depends on 10)

The system as described in claim 10, wherein the visual artifact confidence score is calculated by leveraging a convolutional neural network to detect resolution inconsistencies associated with affine warpings.

### Claim 13 (depends on 10)

The system as described in claim 10, wherein the temporal confidence score is calculated by determining an optical flow between adjacent frames of a plurality of frames of the digital video.

### Claim 14 (depends on 10)

The system as described in claim 10, wherein the temporal confidence score is calculated by comparing audial features of the digital video to visual features of the digital video.

### Claim 15 (depends on 10)

The system as described in claim 10, wherein the digital video includes a plurality of entities and wherein the affective state confidence score is calculated by leveraging a machine learning model to generate labels for each entity of the plurality of entities, and comparing the labels, one to another.

### Claim 16 (depends on 10)

The system as described in claim 10, wherein the manipulation score is a maximum of the gaze confidence score and the one or more of the visual artifact confidence score, the temporal confidence score, or the affective state confidence score.

### Claim 17

A non-transitory computer-readable storage medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
receiving a frame of a digital video that depicts at least one entity;
identifying coordinates of the frame that correspond to a gaze location of the at least one entity using gaze tracking;
determining whether the coordinates of the frame correspond to a portion of an object depicted in the frame; and
calculating a gaze confidence score based on the determining, the gaze confidence score indicating a probability that the digital video includes a spatial manipulation.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium as described in claim 17, wherein the operations further comprise generating a manipulation score based on the gaze confidence score and at least one of a visual artifact confidence score, a temporal confidence score, or an affective state confidence score.

### Claim 19 (depends on 17)

The non-transitory computer-readable storage medium as described in claim 17, wherein the spatial manipulation includes an added visual feature inserted into the digital video or a removed visual feature removed from the digital video.

### Claim 20 (depends on 17)

The non-transitory computer-readable storage medium as described in claim 17, wherein the digital video includes a plurality of entities, and wherein calculating the gaze confidence score includes comparing a plurality of gaze locations for each entity of the plurality of entities, one to another.
