# US 12,217,493 B2 · Methods, systems, and media for generating video classifications using multimodal video analysis > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,493 B2 - Document ID: US12217493B2 - Status: Granted - Kind: B2 - Application number: 17/890,112 - Filed: 2022-08-17 - Published: 2023-02-23 - Granted: 2025-02-04 - Earliest claimed: 2021-08-17 - Assignees: Integral Ad Science, Inc. - Applicants: Integral Ad Science, Inc. - Inventors: Mattia Fumagalli; Stefano Vegnaduzzo; Charlie Schumacher; Varun Medappa; Tyler Hughes; Aleks Navratil; Josh Ryan; Igor Benyaminov - Web page: https://patentgrep.com/patent/US12217493B2 ## CPC - G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition} - G06Q 30/0276: {Advertisement creation} - G06V 10/82: using neural networks - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/48: {Matching video sequences} - G06V 20/62: Text, e.g. of license plates, overlay texts or captions on TV images - G06V 30/191: {Design or setup of recognition systems or techniques; Extraction of features in feature space; Clustering techniques; Blind source separation} - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods, systems, and media for generating video classifications using multimodal video analysis are provided. In some embodiments, a method for classifying videos comprising: receiving, from a computing device, a video identifier; parsing a video associated with the video identifier into an audio portion and a plurality of image frames; analyzing the plurality of images frames associated with the video using (i) an optical character recognition technique to obtain first textual information corresponding to text appearing in at least one of the plurality of image frames and (ii) an image classifier to obtain, for each of a plurality of objects appearing in at least one of the plurality of frames of the video, a probability that an object appearing in at least one of the plurality of images falls within an image class; concurrently with analyzing the plurality of image frames associated with the video, analyzing the audio portion of the video using an automated speech recognition technique to obtain second textual information corresponding to words spoken in the video; combining the first textual information, the probability of each of the plurality of objects appearing in the at least one of the plurality of frames of the video, and the second textual information to obtain a combined analysis output for the video; determining, using a neural network, a safety score for each of a plurality of categories that the video contains content belonging to a category of the plurality of categories, wherein the combined analysis output is input into the neural network; and, in response to receiving the video identifier, transmitting a plurality of safety scores corresponding to the plurality of categories to the computing device for the video associated with the video identifier. ## Claims ### Claim 1 A method for classifying videos, the method comprising: receiving, from a computing device, a video identifier; parsing a video associated with the video identifier into an audio portion and a plurality of image frames; analyzing the plurality of images frames associated with the video using (i) an optical character recognition technique to obtain first textual information corresponding to text appearing in at least one of the plurality of image frames and (ii) an image classifier to obtain, for each of a plurality of objects appearing in at least one of the plurality of frames of the video, a probability that an object appearing in at least one of the plurality of images falls within an image class; concurrently with analyzing the plurality of image frames associated with the video, analyzing the audio portion of the video using an automated speech recognition technique to obtain second textual information corresponding to words spoken in the video; combining the first textual information, the probability of each of the plurality of objects appearing in the at least one of the plurality of frames of the video, and the second textual information to obtain a combined analysis output for the video; determining, using a neural network, a safety score for each of a plurality of categories that the video contains content belonging to a category of the plurality of categories, wherein the combined analysis output is input into the neural network; and in response to receiving the video identifier, transmitting a plurality of safety scores corresponding to the plurality of categories to the computing device for the video associated with the video identifier. ### Claim 2 (depends on 1) The method of claim 1, further comprising: determining a threshold for each category in the plurality of categories; and comparing, for each of the plurality of categories, the safety score that the video contains content belonging to the category of the plurality of categories against the threshold for that category. ### Claim 3 (depends on 2) The method of claim 2, further comprising: associating categories with the video based on the comparison of the safety score and the threshold for each of the plurality of safety categories; and transmitting the associated categories to the computing device for the video associated with the video identifier. ### Claim 4 (depends on 2) The method of claim 2, wherein the threshold for each category in the plurality of categories is determined based on a set labeled data that is applied to the neural network. ### Claim 5 (depends on 1) The method of claim 1, further comprising determining whether one or more advertisements associated with the computing device should be placed in connection with the video associated with the video identifier based on the safety score for each of the plurality of categories. ### Claim 6 (depends on 1) The method of claim 1, further comprising determining a number of advertisements served in connection with a plurality of videos that are deemed to be unsafe based on the safety score for each of the plurality of categories corresponding to that video. ### Claim 7 (depends on 1) The method of claim 1, wherein the plurality of categories includes categories that are considered unsafe in association with one or more advertisements by an advertiser. ### Claim 8 (depends on 1) The method of claim 1, wherein the video is generated by a user on a social media platform. ### Claim 9 (depends on 1) The method of claim 1, wherein the audio portion of the video is further analyzed using an audio tagging classifier to detect sounds occurring within the audio portion of the video and wherein the detected sounds are incorporated into the combined analysis output for the video. ### Claim 10 (depends on 1) The method of claim 1, wherein the plurality of images frames are further analyzed using an object detector to detect objects appearing in at least one of the plurality of images and wherein the detected objects are incorporated into the combined analysis output for the video. ### Claim 11 A system for classifying videos, the system comprising: a server that includes a hardware processor, wherein the hardware processor is configured to: receive, from a computing device, a video identifier; parse a video associated with the video identifier into an audio portion and a plurality of image frames; analyze the plurality of images frames associated with the video using (i) an optical character recognition technique to obtain first textual information corresponding to text appearing in at least one of the plurality of image frames and (ii) an image classifier to obtain, for each of a plurality of objects appearing in at least one of the plurality of frames of the video, a probability that an object appearing in at least one of the plurality of images falls within an image class; concurrently with analyzing the plurality of image frames associated with the video, analyze the audio portion of the video using an automated speech recognition technique to obtain second textual information corresponding to words spoken in the video; combine the first textual information, the probability of each of the plurality of objects appearing in the at least one of the plurality of frames of the video, and the second textual information to obtain a combined analysis output for the video; determine, using a neural network, a safety score for each of a plurality of categories that the video contains content belonging to a category of the plurality of categories, wherein the combined analysis output is input into the neural network; and in response to receiving the video identifier, transmit a plurality of safety scores corresponding to the plurality of categories to the computing device for the video associated with the video identifier. ### Claim 12 (depends on 11) The system of claim 11, wherein the hardware processor is further configured to: determine a threshold for each category in the plurality of categories; compare, for each of the plurality of categories, the safety score that the video contains content belonging to the category of the plurality of categories against the threshold for that category. ### Claim 13 (depends on 12) The system of claim 12, wherein the hardware processor is further configured to: associate categories with the video based on the comparison of the safety score and the threshold for each of the plurality of safety categories; and transmit the associated categories to the computing device for the video associated with the video identifier. ### Claim 14 (depends on 12) The system of claim 12, wherein the threshold for each category in the plurality of categories is determined based on a set labeled data that is applied to the neural network. ### Claim 15 (depends on 11) The system of claim 11, wherein the hardware processor is further configured to determine whether one or more advertisements associated with the computing device should be placed in connection with the video associated with the video identifier based on the safety score for each of the plurality of categories. ### Claim 16 (depends on 11) The system of claim 11, wherein the hardware processor is further configured to determine a number of advertisements served in connection with a plurality of videos that are deemed to be unsafe based on the safety score for each of the plurality of categories corresponding to that video. ### Claim 17 (depends on 11) The system of claim 11, wherein the plurality of categories includes categories that are considered unsafe in association with one or more advertisements by an advertiser. ### Claim 18 (depends on 11) The system of claim 11, wherein the video is generated by a user on a social media platform. ### Claim 19 (depends on 11) The system of claim 11, wherein the audio portion of the video is further analyzed using an audio tagging classifier to detect sounds occurring within the audio portion of the video and wherein the detected sounds are incorporated into the combined analysis output for the video. ### Claim 20 (depends on 11) The system of claim 11, wherein the plurality of images frames are further analyzed using an object detector to detect objects appearing in at least one of the plurality of images and wherein the detected objects are incorporated into the combined analysis output for the video. ### Claim 21 A non-transitory computer-readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for classifying videos, the method comprising: receiving, from a computing device, a video identifier; parsing a video associated with the video identifier into an audio portion and a plurality of image frames; analyzing the plurality of images frames associated with the video using (i) an optical character recognition technique to obtain first textual information corresponding to text appearing in at least one of the plurality of image frames and (ii) an image classifier to obtain, for each of a plurality of objects appearing in at least one of the plurality of frames of the video, a probability that an object appearing in at least one of the plurality of images falls within an image class; concurrently with analyzing the plurality of image frames associated with the video, analyzing the audio portion of the video using an automated speech recognition technique to obtain second textual information corresponding to words spoken in the video; combining the first textual information, the probability of each of the plurality of objects appearing in the at least one of the plurality of frames of the video, and the second textual information to obtain a combined analysis output for the video; determining, using a neural network, a safety score for each of a plurality of categories that the video contains content belonging to a category of the plurality of categories, wherein the combined analysis output is input into the neural network; and in response to receiving the video identifier, transmitting a plurality of safety scores corresponding to the plurality of categories to the computing device for the video associated with the video identifier.