# US 12,223,721 B2 · Method and apparatus for video frame processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,721 B2
- Document ID: US12223721B2
- Status: Granted
- Kind: B2
- Application number: 17/575,140
- Filed: 2022-01-13
- Published: 2022-05-05
- Granted: 2025-02-11
- Earliest claimed: 2019-07-18
- Assignees: Tencent Music Entertainment Technology (Shenzhen) Co., Ltd.
- Applicants: Tencent Music Entertainment Technology (Shenzhen) Co., Ltd.
- Inventors: Mengye Lv; Zhi Dong; Shenyuan Li
- Web page: https://patentgrep.com/patent/US12223721B2

## CPC

- G06F 16/75: Clustering; Classification
- G06V 20/48: {Matching video sequences}
- G06F 16/783: using metadata automatically derived from the content
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and an apparatus for video frame processing are provided. The method includes: obtaining a convolutional neural network (CNN) feature and a local feature of a target video frame; performing dimension reduction on the CNN feature to obtain a CNN feature with a reduced dimension of the target video frame; obtaining a first video frame from multiple sample video frames, a distance between a CNN feature with a reduced dimension of the first video frame and the CNN feature with the reduced dimension of the target video frame meets a first preset condition; obtaining a local feature of the first video frame; calculating a matching degree between the local feature of the first video frame and the local feature of the target video frame; determining the first video frame as a duplicate video frame of the target video frame if the matching degree meets a second preset condition.

## Claims

### Claim 1

A method for video frame processing, comprising:
obtaining a convolutional neural network (CNN) feature of a target video frame and a local feature of the target video frame, the local feature of the target video frame comprising a first key point of the target video frame and a feature descriptor corresponding to the first key point;
performing dimension reduction on the CNN feature of the target video frame to obtain a CNN feature with a reduced dimension of the target video frame;
obtaining a first video frame from a plurality of sample video frames, a distance between a CNN feature with a reduced dimension of the first video frame and the CNN feature with the reduced dimension of the target video frame meeting a first preset condition;
obtaining a local feature of the first video frame, the local feature of the first video frame comprising a second key point in the first video frame and a feature descriptor corresponding to the second key point;
calculating a matching degree between the local feature of the first video frame and the local feature of the target video frame; and
determining the first video frame as a duplicate video frame of the target video frame if the matching degree meets a second preset condition.

### Claim 2 (depends on 1)

The method of claim 1, wherein obtaining the CNN feature of the target video frame comprises:
obtaining a video to-be-queried;
taking equal interval video screenshots of the video to-be-queried to obtain the target video frame, the target video frame being any of a plurality of video frames obtained by taking equal interval video screenshots of the video to-be-queried; and
obtaining the CNN feature of the target video frame by inputting the target video frame into a CNN for processing.

### Claim 3 (depends on 1)

The method of claim 1, wherein obtaining the first video frame from the plurality of sample video frames comprises:
obtaining a CNN feature index of a sample video, the sample video comprising the plurality of sample video frames, the CNN feature index being indicative of a plurality of clusters formed by clustering of a CNN feature with a reduced dimension of the plurality of sample video frames, each cluster comprising a cluster center and a CNN feature with a reduced dimension of at least one sample video frame in the cluster;
calculating a distance between the CNN feature with the reduced dimension of the target video frame and the cluster center of each of the plurality of clusters and determining a cluster with a cluster center having the shortest distance as a target cluster; and
calculating a distance between the CNN feature with the reduced dimension of the target video frame and a CNN feature with a reduced dimension of each of the at least one sample video frame comprised in the target cluster, and determining a sample video frame, which corresponds to a CNN feature with a reduced dimension at the shortest distance, as the first video frame.

### Claim 4 (depends on 3)

The method of claim 3, wherein the local feature of the target video frame comprises m first key points and m first feature descriptors corresponding to the m first key points, one first key point corresponds to one first feature descriptor, the local feature of the first video frame comprises n second key points and n second feature descriptors corresponding to the n second key points, one second key point corresponds to one second feature descriptor, m being a natural number greater than or equal to 2, and n being a natural number greater than or equal to 2, and wherein
calculating the matching degree between the local feature of the first video frame and the local feature of the target video frame comprises:
for each first feature descriptor, obtaining a distance between each of the n second feature descriptors and the first feature descriptor to obtain n distances;
sorting the n distances to form a sorting queue in order from large to small;
obtaining last k distances in the sorting queue, k being a natural number greater than or equal to 2;
determining that the first feature descriptor is an effective descriptor matching the first video frame, according to the k distances; and
determining the matching degree between the local feature of the first video frame and the local feature of the target video frame according to the number of effective descriptors in the m first feature descriptors.

### Claim 5 (depends on 3)

The method of claim 3, further comprising:
before obtaining the CNN feature of the target video frame and the local feature of the target video frame,
obtaining a CNN feature of the plurality of sample video frames;
performing dimension reduction on the CNN feature of the plurality of sample video frames with a principal component analysis (PCA) matrix to obtain the CNN feature with the reduced dimension of the plurality of sample video frames;
performing k-means clustering on the CNN feature with the reduced dimension of the plurality of sample video frames to form a plurality of clusters, each cluster comprising a CNN feature with a reduced dimension of at least one sample video frame;
performing quantization compression on the CNN feature with the reduced dimension of the at least one sample video frame comprised in each cluster to obtain a compressed CNN feature corresponding to the cluster; and
generating the CNN feature index of the sample video according to the plurality of clusters, the compressed CNN feature corresponding to each cluster, and the cluster center of each cluster.

### Claim 6 (depends on 1)

The method of claim 1, wherein the target video frame belongs to a video to-be-queried, a playing time point of the target video frame in the video to-be-queried is a first playing time point, and the first video frame belongs to a target video in a sample video, wherein determining the first video frame as the duplicate video frame of the target video frame comprises:
obtaining a frame identifier (ID) of the first video frame; and
searching for a second playing time point in the target video corresponding to the frame ID of the first video frame, and determining a video frame at the second playing time point in the target video as a duplicate video frame of a video frame at the first playing time point in the video to-be-queried.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
if the number of all duplicate video frames of the video to-be-queried and the target video is greater than a second threshold and duplicate video frames meeting a continuous distribution condition among all the duplicate video frames are distributed in a first time period of the video to-be-queried and a second time period of the target video, determining a video in the first time period of the video to-be-queried and a video in the second time period in the target video as duplicate video segments, wherein the first time period comprises the first playing time point, the second time period comprises the second playing time point, and the continuous distribution condition comprises that a time difference between adjacent duplicate video frames is less than the third threshold.

### Claim 8

An apparatus for video frame processing, comprising:
a processor; and
a memory storing program instructions which, when executed by the processor, are operable with the processor to:
obtain a convolutional neural network (CNN) feature of a target video frame and a local feature of the target video frame, the local feature of the target video frame comprising a first key point of the target video frame and a feature descriptor corresponding to the first key point;
perform dimension reduction on the CNN feature of the target video frame to obtain a CNN feature with a reduced dimension of the target video frame;
obtain a first video frame from a plurality of sample video frames, a distance between a CNN feature with a reduced dimension of the first video frame and the CNN feature with the reduced dimension of the target video frame meeting a first preset condition;
obtain a local feature of the first video frame, the local feature of the first video frame comprising a second key point in the first video frame and a feature descriptor corresponding to the second key point;
calculate a matching degree between the local feature of the first video frame and the local feature of the target video frame; and
determine the first video frame as a duplicate video frame of the target video frame if the matching degree meets a second preset condition.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the processor configured to obtain the CNN feature of the target video frame and the local feature of the target video frame is configured to:
obtain a video to-be-queried;
take equal interval video screenshots of the video to-be-queried to obtain the target video frame, the target video frame being any of a plurality of video frames obtained by taking equal interval video screenshots of the video to-be-queried; and
obtain the CNN feature of the target video frame by inputting the target video frame into a CNN for processing.

### Claim 10 (depends on 8)

The apparatus of claim 8, wherein the processor configured to obtain the first video frame from the plurality of sample video frames is configured to:
obtain a CNN feature index of a sample video, the sample video comprising the plurality of sample video frames, the CNN feature index being indicative of a plurality of clusters formed by clustering of a CNN feature with a reduced dimension of the plurality of sample video frames, each cluster comprising a cluster center and a CNN feature with a reduced dimension of at least one sample video frame in the cluster;
calculate a distance between the CNN feature with the reduced dimension of the target video frame and the cluster center of each of the plurality of clusters and determining a cluster with a cluster center at the shortest distance as a target cluster; and
calculate a distance between the CNN feature with the reduced dimension of the target video frame and a CNN feature with a reduced dimension of each of the at least one sample video frame comprised in the target cluster, and determining a sample video frame, which corresponds to a CNN feature with a reduced dimension at the shortest distance, as the first video frame.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein the local feature of the target video frame comprises m first key points and m first feature descriptors corresponding to the m first key points, one first key point corresponds to one first feature descriptor, the local feature of the first video frame comprises n second key points and n second feature descriptors corresponding to the n second key points, one second key point corresponds to one second feature descriptor, m being a natural number greater than or equal to 2, and n being a natural number greater than or equal to 2, and wherein
the processor configured to calculate the matching degree between the local feature of the first video frame and the local feature of the target video frame is configured to:
obtain, for each first feature descriptor, a distance between each of the n second feature descriptors and the first feature descriptor to obtain n distances;
sort the n distances to form a sorting queue in order from large to small;
obtain last k distances in the sorting queue, k being a natural number greater than or equal to 2;
determine, according to the k distances, that the first feature descriptor is an effective descriptor matching the first video frame; and
determine the matching degree between the local feature of the first video frame and the local feature of the target video frame according to the number of effective descriptors in the m first feature descriptors.

### Claim 12 (depends on 10)

The apparatus of claim 10, wherein the processor is further configured to:
obtain a CNN feature of the plurality of sample video frames;
perform dimension reduction on the CNN feature of the plurality of sample video frames with a principal component analysis (PCA) matrix to obtain the CNN feature with the reduced dimension of the plurality of sample video frames;
perform k-means clustering on the CNN feature with the reduced dimension of the plurality of sample video frames to form a plurality of clusters, each cluster comprising a CNN feature with a reduced dimension of at least one sample video frame;
perform quantization compression on the CNN feature with the reduced dimension of the at least one sample video frame comprised in each cluster to obtain a compressed CNN feature corresponding to the cluster; and
generate the CNN feature index of the sample video according to the plurality of clusters, the compressed CNN feature corresponding to each cluster, and the cluster center of each cluster.

### Claim 13 (depends on 8)

The apparatus of claim 8, wherein the target video frame belongs to a video to-be-queried, a playing time point of the target video frame in the video to-be-queried is a first playing time point, and the first video frame belongs to a target video in a sample video, wherein the processor configured to determine the first video frame as a duplicate video frame of the target video frame if the matching degree meets the second preset condition is configured to:
obtain a frame identifier (ID) of the first video frame; and
search for a second playing time point in the target video corresponding to the frame ID of the first video frame, and determine a video frame at the second playing time point in the target video as a duplicate video frame of a video frame at the first playing time point in the video to-be-queried.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the processor is further configured to:
determine a video in a first time period of the video to-be-queried and a video in a second time period in the target video as duplicate video segments if the number of all duplicate video frames of the video to-be-queried and the target video is greater than a second threshold and duplicate video frames meeting a continuous distribution condition among all the duplicate video frames are distributed in the first time period of the video to-be-queried and the second time period of the target video, wherein the first time period comprises the first playing time point, the second time period comprises the second playing time point, and the continuous distribution condition comprises that a time difference between adjacent duplicate video frames is less than the third threshold.

### Claim 15

A non-transitory computer readable storage medium storing program instructions which, when executed by a computer, are operable with the computer to:
obtain a convolutional neural network (CNN) feature of a target video frame and a local feature of the target video frame, the local feature of the target video frame comprising a first key point of the target video frame and a feature descriptor corresponding to the first key point;
perform dimension reduction on the CNN feature of the target video frame to obtain a CNN feature with a reduced dimension of the target video frame;
obtain a first video frame from a plurality of sample video frames, a distance between a CNN feature with a reduced dimension of the first video frame and the CNN feature with the reduced dimension of the target video frame meeting a first preset condition;
obtain a local feature of the first video frame, the local feature of the first video frame comprising a second key point in the first video frame and a feature descriptor corresponding to the second key point;
calculate a matching degree between the local feature of the first video frame and the local feature of the target video frame; and
determine the first video frame as a duplicate video frame of the target video frame if the matching degree meets a second preset condition.

### Claim 16 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein the program instructions operable with the computer to obtain the CNN feature of the target video frame are operable with the computer to:
obtain a video to-be-queried;
take equal interval video screenshots of the video to-be-queried to obtain the target video frame, the target video frame being any of a plurality of video frames obtained by taking equal interval video screenshots of the video to-be-queried; and
obtain the CNN feature of the target video frame by inputting the target video frame into a CNN for processing.

### Claim 17 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein the program instructions operable with the computer to obtain the first video frame from the plurality of sample video frames are operable with the computer to:
obtain a CNN feature index of a sample video, the sample video comprising the plurality of sample video frames, the CNN feature index being indicative of a plurality of clusters formed by clustering of a CNN feature with a reduced dimension of the plurality of sample video frames, each cluster comprising a cluster center and a CNN feature with a reduced dimension of at least one sample video frame in the cluster;
calculate a distance between the CNN feature with the reduced dimension of the target video frame and the cluster center of each of the plurality of clusters and determining a cluster with a cluster center having the shortest distance as a target cluster; and
calculate a distance between the CNN feature with the reduced dimension of the target video frame and a CNN feature with a reduced dimension of each of the at least one sample video frame comprised in the target cluster, and determining a sample video frame, which corresponds to a CNN feature with a reduced dimension at the shortest distance, as the first video frame.

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the local feature of the target video frame comprises m first key points and m first feature descriptors corresponding to the m first key points, one first key point corresponds to one first feature descriptor, the local feature of the first video frame comprises n second key points and n second feature descriptors corresponding to the n second key points, one second key point corresponds to one second feature descriptor, m being a natural number greater than or equal to 2, and n being a natural number greater than or equal to 2, and wherein
the program instructions operable with the computer to calculate the matching degree between the local feature of the first video frame and the local feature of the target video frame are operable with the computer to:
for each first feature descriptor, obtain a distance between each of the n second feature descriptors and the first feature descriptor to obtain n distances;
sort the n distances to form a sorting queue in order from large to small;
obtain last k distances in the sorting queue, k being a natural number greater than or equal to 2;
determine that the first feature descriptor is an effective descriptor matching the first video frame, according to the k distances; and
determine the matching degree between the local feature of the first video frame and the local feature of the target video frame according to the number of effective descriptors in the m first feature descriptors.

### Claim 19 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the program instructions are further operable with the computer to:
obtain a CNN feature of the plurality of sample video frames;
perform dimension reduction on the CNN feature of the plurality of sample video frames with a principal component analysis (PCA) matrix to obtain the CNN feature with the reduced dimension of the plurality of sample video frames;
perform k-means clustering on the CNN feature with the reduced dimension of the plurality of sample video frames to form a plurality of clusters, each cluster comprising a CNN feature with a reduced dimension of at least one sample video frame;
perform quantization compression on the CNN feature with the reduced dimension of the at least one sample video frame comprised in each cluster to obtain a compressed CNN feature corresponding to the cluster; and
generate the CNN feature index of the sample video according to the plurality of clusters, the compressed CNN feature corresponding to each cluster, and the cluster center of each cluster.

### Claim 20 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein the target video frame belongs to a video to-be-queried, a playing time point of the target video frame in the video to-be-queried is a first playing time point, and the first video frame belongs to a target video in a sample video, wherein the program instructions operable with the computer to determine the first video frame as the duplicate video frame of the target video frame are operable with the computer to:
obtain a frame identifier (ID) of the first video frame; and
search for a second playing time point in the target video corresponding to the frame ID of the first video frame, and determine a video frame at the second playing time point in the target video as a duplicate video frame of a video frame at the first playing time point in the video to-be-queried.