# US 12,008,037 B2 · Method of video search in an electronic device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,037 B2
- Document ID: US12008037B2
- Status: Granted
- Kind: B2
- Application number: 17/554,658
- Filed: 2021-12-17
- Published: 2022-05-12
- Granted: 2024-06-11
- Earliest claimed: 2020-11-03
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Sathish Chalasani; Ujjawal Soni; Kiran Nanjunda Iyer; Shreyasi Das; Samarjeet Sahoo
- Web page: https://patentgrep.com/patent/US12008037B2

## CPC

- G06F 16/739: {in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames}
- G06F 16/732: Query formulation
- G06F 16/738: Presentation of query results
- G06F 16/743: {a collection of video files or sequences}
- G06F 16/75: Clustering; Classification
- G06F 16/783: using metadata automatically derived from the content
- G06F 16/7867: {using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/82: using neural networks
- G06V 20/30: in albums, collections or shared content, e.g. social network photos or video
- G06V 20/35: {Categorising the entire scene, e.g. birthday party or wedding scene}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method of video search in an electronic device is provided. The method includes extracting a plurality of visual features vectors comprising visual features from a plurality of videos stored in a video database, wherein each visual feature corresponds to a pre-defined moment, generating a temporal map of the plurality of visual feature vectors, generating at least one video tag and at least one time stamp associated with the visual feature vector, storing the at least one video tag, the visual feature vectors and the at least one time stamp for each visual feature in a vector database, receiving a search query from a user for a specific moment in a plurality of videos in the electronic device, running the search query on the vector database, and displaying visual features associated with the specific moments from the plurality of videos.

## Claims

### Claim 1

A method of retrieving a video moment in a video database of an electronic device, the method comprising:
receiving, by the electronic device, a search query from a user to find the video moment in at least one video from a plurality of videos in the video database;
extracting, by the electronic device, at least one semantic vector from the search query;
mapping, by the electronic device, the extracted at least one semantic vector with corresponding visual features moment vector in a pre-stored feature vector database;
determining, by the electronic device, at least one moment tag and at least one time stamp associated with the mapped visual moment vector; and
retrieving, by the electronic device, a video moment clip using the at least one moment tag and at least one time stamp from the video database, wherein the video moment clip corresponds to the video moment in the search query,
wherein the feature vector database is generated by the electronic device from the video database in an offline mode.

### Claim 2 (depends on 1)

The method as claimed in claim 1, wherein the video database is a gallery comprising the plurality of videos in the electronic device.

### Claim 3 (depends on 1)

The method as claimed in claim 1, wherein the generating of the feature vector database comprises:
extracting, by the electronic device, a plurality of visual feature vectors comprising visual features from the plurality of videos stored in the video database of the electronic device, wherein each visual feature corresponds to a pre-defined moment;
generating, by the electronic device, a temporal map of the plurality of visual feature vectors;
generating, by the electronic device, at least one video tag and at least one time stamp associated with the visual feature vector for each visual feature in the plurality of visual feature vectors; and
storing, by the electronic device, the at least one video tag, the visual feature vectors, and the at least one time stamp for each visual feature in the feature vector database.

### Claim 4

A method of video search based on a user query in an electronic device, the method comprising:
receiving, by the electronic device, a search query from a user for a video moment in a plurality of videos in the electronic device;
extracting, by the electronic device, semantic vector using natural language processing (NLP) from the search query;
mapping, by the electronic device, the extracted semantic vector from the search query with at least one visual moment vector stored in a feature vector database of the electronic device;
determining, by the electronic device, at least one moment tag and at least one time stamp associated with the mapped at least one visual moment vector; and
extracting, by the electronic device, a video moment corresponding to the query based on the least one moment tag and at least one time stamp from a video database of the electronic device,
wherein the feature vector database is generated from the video database in an offline mode.

### Claim 5 (depends on 4)

The method as claimed in claim 4,
wherein the video moment corresponding to the query is extracted in real time upon receiving the query.

### Claim 6 (depends on 4)

The method as claimed in claim 4, wherein the video database corresponds to a gallery of the electronic device comprising the plurality of videos.

### Claim 7 (depends on 5)

The method as claimed in claim 5, wherein the generating of the feature vector database comprises:
extracting, by the electronic device, a plurality of visual feature vectors comprising visual features from the plurality of videos stored in the video database of the electronic device, wherein each visual feature corresponds to a pre-defined moment;
generating, by the electronic device, a temporal map of the plurality of visual feature vectors;
generating, by the electronic device, at least one video tag and at least one time stamp associated with the visual feature vector for each visual feature in the plurality of visual feature vectors; and
storing, by the electronic device, the at least one video tag, the visual feature vectors, and the at least one time stamp for each visual feature in the feature vector database.

### Claim 8

A method of generating a feature vector database for a plurality of videos in an offline mode in an electronic device, the method comprising:
extracting, by the electronic device, a plurality of visual feature vectors comprising visual features from a plurality of videos stored in a video database of the electronic device, wherein each visual feature corresponds to a pre-defined moment;
generating, by the electronic device, a temporal map of the plurality of visual feature vectors;
generating, by the electronic device, at least one video tag and at least one time stamp associated with the visual feature vector for each visual feature in the plurality of visual feature vectors; and
storing, by the electronic device, the at least one video tag, the visual feature vectors, and the at least one time stamp for each visual feature in the feature vector database.

### Claim 9 (depends on 8)

The method as claimed in claim 8, wherein the extracting, by the electronic device, of the plurality of visual features from the plurality of videos stored in the video database comprises:
inputting, by the electronic device, each video from the plurality of videos in the videos database to a neural network specific to feature extraction;
performing, by the neural network specific to feature extraction, a bi-directional shifting of a feature map of the plurality of videos in the video database along a temporal dimension;
performing, by the neural network specific to feature extraction, an average-maximum pooling in a temporal domain to obtain a pooled clip feature vector;
determining, by the neural network specific to feature extraction, that a score of the features matches a threshold value; and
extracting, by the neural network specific to feature extraction, the plurality of visual features from the plurality of videos in response to the threshold value being met.

### Claim 10 (depends on 9)

The method as claimed in claim 9, wherein the generating of the temporal map comprises:
passing, by the electronic device, the pooled clip feature vector through bi-directional gated recurrent units (GRU) to get a moment proposal vector; and
forming, by the electronic device, a convolutional feature map using the moment proposal vector.

### Claim 11 (depends on 8)

The method as claimed in claim 8, wherein the extracting, by the electronic device, of the plurality of feature vectors, the at least one video tag and the at least one time stamp for each visual feature from the plurality of visual features comprises:
tagging, by the electronic device, each visual feature vector extracted in each video with a specific tag;
determining, by the electronic device, a time and a duration of each visual feature in each video;
providing, by the electronic device, the at least one time stamp to each visual feature in each video based on the time and a duration; and
storing, by the electronic device the plurality of feature vectors, the tags and the at least one time stamp.

### Claim 12 (depends on 8)

The method as claimed in claim 8, further comprising:
receiving, by the electronic device, a search query from a user for a specific moment in a plurality of videos in the electronic device;
extracting, by the electronic device, linguistic visual features using natural language processing (NLP) from the search query;
mapping, by the electronic device, the extracted linguistic features from the search query with at least one visual feature vector stored in the feature vector database in the electronic device;
determining, by the electronic device, at least one moment tag and at least one time stamp associated with the visual features; and
extracting, by the electronic device, a video moment based on the least one moment tag and at least one time stamp from a video database of the electronic device.

### Claim 13 (depends on 8)

The method as claimed in claim 8, wherein the pre-defined moment is an activity in the video.

### Claim 14

An electronic device for performing video search, the electronic device comprising:
a memory; and
a processor configured to:
extract a plurality of visual features vectors comprising visual features from a plurality of videos stored in a video database of the electronic device, wherein each visual feature corresponds to a pre-defined moment in an offline mode,
generate a temporal map of the plurality of visual feature vectors in an offline mode,
generate at least one video tag and at least one time stamp associated with the visual feature vector for each visual feature in the plurality of visual feature vectors in an offline mode,
store the at least one videos tag, the visual feature vectors, and the at least one time stamp for each visual feature in a feature vector database in an offline mode,
receive a search query from a user for a specific moment in a plurality of videos in the electronic device,
run the search query on the feature vector database, and
control to display the visual features associated with the specific moments from the plurality of videos based on the at least one video tag, the plurality of feature vectors, and the at least one time stamp in the feature vector database.

### Claim 15 (depends on 14)

The electronic device as claimed in claim 14, wherein the processor, to extract the plurality of visual features from the plurality of videos stored in the video database, is configured to:
input each video, from the plurality of videos in the videos database, to a neural network specific to feature extraction;
perform, by the neural network specific to feature extraction, a bi-directional shifting of a feature map of the plurality of videos in the video database along a temporal dimension;
perform, by the neural network specific to feature extraction, an average-maximum pooling in a temporal domain to obtain a pooled clip feature vector;
determine, by the neural network specific to feature extraction, that a score of the features matches a threshold value; and
extract, by the neural network specific to feature extraction, the plurality of visual features from the plurality of videos in response to the threshold value being met.

### Claim 16 (depends on 15)

The electronic device as claimed in in claim 15, wherein the processor, to generate the temporal map, is configured to:
pass the pooled clip feature vector through bi-directional gated recurrent units (GRU) to get a moment proposal vector; and
form a convolutional feature map using the moment proposal vector.

### Claim 17 (depends on 14)

The electronic device as claimed in claim 14, wherein the processor, to generate the plurality of feature vectors, the at least one video tag, and the at least one time stamps for each visual feature from the plurality of visual features, is configured to:
tag each visual feature vector extracted in each video with a specific tag;
determine a time and a duration of each visual feature in each video;
provide the at least one time stamp to each visual feature in each video based on the time and a duration; and
store the plurality of feature vectors, the at least one tag, and the at least one time stamp.

### Claim 18 (depends on 14)

The electronic device as claimed in claim 14, wherein the processor, to control to display the visual features corresponding to specific event, is configured to:
extract linguistic visual features using Natural Language Processing (NLP) from the search query;
compare the extracted linguistic visual features from the search query with the visual features from the plurality of feature vectors using the at least one video tag in the feature vector database;
identify the visual features vector based on the at least one video tag which has a best match with the linguistic visual features;
extract the video for which the visual feature is selected from the video database; and
control to display the extracted video to the user, wherein the extracted video comprises the specific moment asked for in the search query.

### Claim 19 (depends on 14)

The electronic device as claimed in claim 14, wherein the pre-defined moment is an activity in the video.