# US 10,482,328 B2 · Feature-based video annotation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,328 B2 - Document ID: US10482328B2 - Status: Granted - Kind: B2 - Application number: 15/722,756 - Filed: 2017-10-02 - Published: 2018-01-25 - Granted: 2019-11-19 - Earliest claimed: 2015-08-11 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Balakrishnan Varadarajan; George Dan Toderici; Apostol Natsev; Nitin Khandelwal; Sudheendra Vijayanarasimhan; Weilong Yang; Sanketh Shetty - Web page: https://patentgrep.com/patent/US10482328B2 ## CPC - G06F 16/783: using metadata automatically derived from the content - G06F 16/7867: {using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/2413: based on distances to training or reference patterns - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - H04N 5/265: Mixing - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system and methodology provide for annotating videos with entities and associated probabilities of existence of the entities within video frames. A computer-implemented method identifies an entity from a plurality of entities identifying characteristics of video items. The computer-implemented method selects a set of features correlated with the entity based on a value of a feature of a plurality of features, determines a classifier for the entity using the set of features, and determines an aggregation calibration function for the entity based on the set of features. The computer-implemented method selects a video frame from a video item, where the video frame having associated features, and determines a probability of existence of the entity based on the associated features using the classifier and the aggregation calibration function. ## Claims ### Claim 1 A computer-implemented method for annotating videos, the method comprising: identifying, using a hardware processor, a plurality of entities identifying characteristics of video items; selecting, using the hardware processor, a set of features correlated with the plurality of entities based on a value of a feature of a plurality of features; determining, using the hardware processor, a classifier for the plurality of entities using the set of features; determining, using the hardware processor, an aggregation calibration function for the plurality of entities based on the set of features; selecting, using the hardware processor, a video frame from a video item, the video frame having associated features; and determining, using the hardware processor, a probability of existence for each entity of the plurality of entities based on the associated features of the video frame using the classifier and the aggregation calibration function, wherein the probability of existence for each entity of the plurality of entities are normalized. ### Claim 2 (depends on 1) The method of claim 1, further comprising labelling the video frame with the plurality of entities and the probability of existence for each entity in the plurality of entities. ### Claim 3 (depends on 1) The method of claim 1, further comprising: receiving a search query; and providing the video item including the video frame in response to determining the search query matches at least a portion of the plurality of entities. ### Claim 4 (depends on 1) The method of claim 1, wherein the classifier is a linear fusion model for the plurality of entities based on the set of features and wherein determining the classifier for the plurality of entities further comprises determining a weight vector for each feature of the set of features. ### Claim 5 (depends on 4) The method of claim 4, wherein determining the classifier for the plurality of entities further comprises maximizing a precision for linear fusion model and maintaining a recall to be at least a given threshold value. ### Claim 6 (depends on 1) The method of claim 1, further comprising determining the probability of existence for each entity of the plurality of entities based on the associated set of features for each video frame of the video item. ### Claim 7 (depends on 1) The method of claim 1, further comprising determining a calibration function that calibrates a value of a feature of the plurality of features to an individual probability of existence of one of the plurality of entities. ### Claim 8 A system for annotating videos, the method comprising: a memory; and a hardware processor that, when executing computer executable instructions stored in the memory, is configured to: identify a plurality of entities identifying characteristics of video items; select a set of features correlated with the plurality of entities based on a value of a feature of a plurality of features; determine a classifier for the plurality of entities using the set of features; determine an aggregation calibration function for the plurality of entities based on the set of features; select a video frame from a video item, the video frame having associated features; and determine a probability of existence for each entity of the plurality of entities based on the associated features of the video frame using the classifier and the aggregation calibration function, wherein the probability of existence for each entity of the plurality of entities are normalized. ### Claim 9 (depends on 8) The system of claim 8, wherein the hardware processor is further configured to label the video frame with the plurality of entities and the probability of existence for each entity in the plurality of entities. ### Claim 10 (depends on 8) The system of claim 8, wherein the hardware processor is further configured to: receive a search query; and provide the video item including the video frame in response to determining the search query matches at least a portion of the plurality of entities. ### Claim 11 (depends on 8) The system of claim 8, wherein the classifier is a linear fusion model for the plurality of entities based on the set of features and wherein determining the classifier for the plurality of entities further comprises determining a weight vector for each feature of the set of features. ### Claim 12 (depends on 11) The system of claim 11, wherein determining the classifier for the plurality of entities further comprises maximizing a precision for linear fusion model and maintaining a recall to be at least a given threshold value. ### Claim 13 (depends on 8) The system of claim 8, further comprising determining the probability of existence for each entity of the plurality of entities based on the associated set of features for each video frame of the video item. ### Claim 14 (depends on 8) The system of claim 8, wherein the hardware processor is further configured to determine a calibration function that calibrates a value of a feature of the plurality of features to an individual probability of existence of one of the plurality of entities. ### Claim 15 A non-transitory computer-readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for annotating videos, the method comprising: identifying a plurality of entities identifying characteristics of video items; selecting a set of features correlated with the plurality of entities based on a value of a feature of a plurality of features; determining a classifier for the plurality of entities using the set of features; determining an aggregation calibration function for the plurality of entities based on the set of features; selecting a video frame from a video item, the video frame having associated features; and determining a probability of existence for each entity of the plurality of entities based on the associated features of the video frame using the classifier and the aggregation calibration function, wherein the probability of existence for each entity of the plurality of entities are normalized. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the method further comprises labelling the video frame with the plurality of entities and the probability of existence for each entity in the plurality of entities. ### Claim 17 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the method further comprises: receiving a search query; and providing the video item including the video frame in response to determining the search query matches at least a portion of the plurality of entities. ### Claim 18 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the classifier is a linear fusion model for the plurality of entities based on the set of features and wherein determining the classifier for the plurality of entities further comprises determining a weight vector for each feature of the set of features. ### Claim 19 (depends on 18) The non-transitory computer-readable medium of claim 18, wherein determining the classifier for the plurality of entities further comprises maximizing a precision for linear fusion model and maintaining a recall to be at least a given threshold value. ### Claim 20 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the method further comprises determining the probability of existence for each entity of the plurality of entities based on the associated set of features for each video frame of the video item. ### Claim 21 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the method further comprises determining a calibration function that calibrates a value of a feature of the plurality of features to an individual probability of existence of one of the plurality of entities.