# US 12,222,985 B2 · Video retrieval method and apparatus using vectorized segmented videos based on key frame detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,222,985 B2
- Document ID: US12222985B2
- Status: Granted
- Kind: B2
- Application number: 17/883,494
- Filed: 2022-08-08
- Published: 2023-09-28
- Granted: 2025-02-11
- Earliest claimed: 2022-03-07
- Assignees: Twelve Labs, Inc.
- Applicants: Twelve Labs, Inc.
- Inventors: Seung Joon Lee; Sung Jun Kim; Raehyuk Jung; Haram Jo
- Web page: https://patentgrep.com/patent/US12222985B2

## CPC

- G06F 16/7837: {using objects detected or recognised in the video content}
- G06F 16/783: using metadata automatically derived from the content
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/48: {Matching video sequences}
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In order to implement the foregoing object, an exemplary embodiment of the present disclosure discloses a video retrieval method performed by a computing device. The video retrieval method may include: segmenting one or more video data into two or more unit video data; encoding, by one or more encoders comprised in a machine learning enabled key frame detection module, two or more unit video data; generating one or more key frame detection vectors for each of the two or more unit video data based on the result of the encoding; and generating feature vector of one or more retrieval video data based on a combination of one or more vectors among the key frame detection vectors.

## Claims

### Claim 1

A video retrieval method performed by a computing device, the video retrieval method comprising:
segmenting one or more video data into two or more unit video data;
encoding, by one or more encoders comprised in a machine learning enabled key frame detecting module, two or more unit video data;
generating one or more key frame detection vectors for each of the two or more unit video data based on the result of the encoding; and
generating feature vector of one or more target retrieval video data based on a combination of one or more vectors among the key frame detection vectors.

### Claim 2 (depends on 1)

The video retrieval method of claim 1, further comprising:
identifying key frame information among the two or more unit video data based on the one or more key frame detection vectors.

### Claim 3 (depends on 2)

The video retrieval method of claim 2,
wherein the encoding the two or more unit video data includes:
generating, by one or more unit video data encoding modules, one or more unit video data sub tokens for each of the two or more unit video data; and
generating, by the key frame detection module, the one or more key frame detection vectors for the unit video data based on the one or more unit video data sub tokens.

### Claim 4 (depends on 2)

The video retrieval method of claim 2, further comprising:
generating the one or more target retrieval video data by grouping the two or more unit video data based on the identified key frame information.

### Claim 5 (depends on 1)

The video retrieval method of claim 1, wherein each of the one or more target retrieval video data comprises unit video data grouped based on the values of the unit video data sub tokens.

### Claim 6 (depends on 5)

The video retrieval method of claim 5, wherein the each of the one or more target retrieval video data comprises two or more temporally continuous unit video data.

### Claim 7 (depends on 6)

The video retrieval method of claim 6, wherein the each of the one or more target retrieval video data comprises at least one unit video data identified as key frame, and the unit video data identified as key frame is temporally the most preceding or temporally the most trailing among the two or more temporally continuous unit video data.

### Claim 8

A non-transitory computer readable storage medium storing a computer program, in which when the computer program is executed in one or more processors, the computer program causes the one or more processors to perform operations for performing a video retrieval method, the video retrieval method comprising:
segmenting one or more video data into two or more unit video data;
encoding, by one or more encoders comprised in a machine learning enabled key frame detection module, two or more unit video data;
generating one or more key frame detection vectors for each of the two or more unit video data based on the encoding result; and
generating feature vector of one or more retrieval video data based on a combination of one or more vectors among the key frame detection vectors.

### Claim 9

A computing device performing a video retrieval method, the computing device comprising:
a processor including at least one core; and
a memory including program codes executable in the processor,
wherein the processor:
segments one or more video data into two or more unit video data;
encodes, by one or more encoders comprised in a machine learning enabled key frame detection module, two or more unit video data;
generates one or more key frame detection vectors for each of the two or more unit video data based on the encoding result; and
generates feature vector of one or more target retrieval video data based on a combination of one or more vectors among the key frame detection vectors.