# US 2019/0171886 A1 · Object recognition in video > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0171886 A1 - Document ID: US20190171886A1 - Status: Published application - Kind: A1 - Application number: 15/832,881 - Filed: 2017-12-06 - Published: 2019-06-06 - Earliest claimed: 2017-12-06 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Gal Ashour; Yevgeny Burshtein; Tal Hakim; Dror Porat; Daniel Nechemia Rotman - Web page: https://patentgrep.com/patent/US20190171886A1 ## CPC - G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes} - G06F 18/2163: {Partitioning the feature space} - G06F 18/22: Matching criteria, e.g. proximity measures - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06V 20/62: Text, e.g. of license plates, overlay texts or captions on TV images - G06V 2201/10: Recognition assisted with metadata - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A system comprising a non-transient computer-readable storage medium having stored thereon instructions and at least one hardware processor configured to execute the instructions, to receive a video sequence; divide the video sequence into one or more scenes based on scene boundaries, wherein each scene comprises a plurality of temporally-contiguous image frames, and wherein said scene boundaries are being determined based on a similarity metric between two temporally-contiguous image frames meeting a dissimilarity threshold; and, for each scene of the one or more scenes, (i) generate a plurality of preliminary classifications of an object appearing in at least some of said image frames in the scene, wherein each of said plurality of preliminary classifications has a confidence score, and (ii) calculate a combined classification of the object based on said plurality of preliminary classifications, wherein each of said preliminary classifications is weighted in accordance with its confidence score. ## Claims ### Claim 1 A system comprising: (a) a non-transient computer-readable storage medium having stored thereon instructions to: receive a video sequence, divide the video sequence into one or more scenes based on scene boundaries, wherein each scene comprises a plurality of temporally-contiguous image frames, and wherein said scene boundaries are being determined based on a similarity metric between two temporally-contiguous image frames meeting a dissimilarity threshold, and for each scene of the one or more scenes: (i) generate a plurality of preliminary classifications of an object appearing in at least some of said image frames in the scene, wherein each of said plurality of preliminary classifications has a confidence score, and (ii) calculate a combined classification of the object based on said plurality of preliminary classifications, wherein each of said preliminary classifications is weighted in accordance with its confidence score; and (b) at least one hardware processor configured to execute the instructions. ### Claim 2 (depends on 1) The system of claim 1, wherein said instructions further comprise iteratively repeating the step of dividing of the video sequence into one or more scenes, wherein in a next iteration, the determining of scene boundaries is based, at least in part, on said combined classification. ### Claim 3 (depends on 1) The system of claim 1, wherein said scene boundaries are being determined, at least in part, based on one or more of: video metadata associated with the video sequence; a time range definition; textual information from the video sequence; audio content associated with the video sequence; and closed-captioning information associated with the video sequence. ### Claim 4 (depends on 1) The system of claim 1, wherein said instructions are further configured to execute facial recognition. ### Claim 5 (depends on 1) The system of claim 1, wherein said instructions are further configured to execute Optical Character Recognition (OCR). ### Claim 6 (depends on 1) The system of claim 1, wherein said classifications are executed by a neural network. ### Claim 7 (depends on 6) The system of claim 6, wherein said neural network is trained on a set of labelled image data to learn characteristics for each type of object. ### Claim 8 A computer program product, the computer program product comprising a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by at least one hardware processor to: receive a video sequence; divide the video sequence into one or more scenes based on scene boundaries, wherein each scene comprises a plurality of temporally-contiguous image frames, and wherein said scene boundaries are being determined based on a similarity metric between two temporally-contiguous image frames meeting a dissimilarity threshold; and for each scene of the one or more scenes: (i) generate a plurality of preliminary classifications of an object appearing in at least some of said image frames in the scene, wherein each of said plurality of preliminary classifications has a confidence score, and (ii) calculate a combined classification of the object based on said plurality of preliminary classifications, wherein each of said preliminary classifications is weighted in accordance with its confidence score. ### Claim 9 (depends on 8) The computer program product of claim 8, further comprising iteratively repeating the step of dividing of the video sequence into one or more scenes, wherein in a next iteration, the determining of scene boundaries is based, at least in part, on said combined classification. ### Claim 10 (depends on 8) The computer program product of claim 8, wherein said scene boundaries are being determined at least in part based on one or more of: video metadata associated with the video sequence; a time range definition; textual information from the video sequence; audio content associated with the video sequence; and closed-captioning information associated with the video sequence. ### Claim 11 (depends on 8) The computer program product of claim 8, further comprising program code for executing facial recognition. ### Claim 12 (depends on 8) The computer program product of claim 8, further comprising program code for executing Optical Character Recognition (OCR). ### Claim 13 (depends on 8) The computer program product of claim 8, wherein said classifications are executed by a neural network. ### Claim 14 (depends on 13) The computer program product of claim 13, wherein said neural network is trained on a set of labelled image data to learn characteristics for each type of object. ### Claim 15 A system comprising: (a) a non-transient computer-readable storage medium having stored thereon instructions to: receive a video sequence, (i) divide the video sequence into scenes based on scene boundaries, wherein each scene comprises a plurality of temporally-contiguous image frames, and wherein said scene boundaries are being determined based on a distance metric between two temporally-contiguous image frames, and with respect to each scene of the scenes: (ii) generate a plurality of preliminary classifications of an object appearing in at least some of said image frames in the scene, wherein each of said plurality of preliminary classifications has a confidence score, and (iii) calculate a combined classification of the object based on said plurality of preliminary classifications, wherein each of said preliminary classifications is weighted in accordance with its confidence score, and iteratively repeat steps (i) through (iii), wherein for a next iteration, said dividing is based, at least in part, on said calculated combined classification; and (b) at least one hardware processor configured to execute the instructions. ### Claim 16 (depends on 15) The system of claim 15, wherein said scene boundaries are being determined at least in part based on one or more of: video metadata associated with the video sequence; a time range definition; textual information from the video sequence; audio content associated with the video sequence; and closed-captioning information associated with the video sequence. ### Claim 17 (depends on 15) The system of claim 15, wherein said instructions are further configured to execute facial recognition. ### Claim 18 (depends on 15) The system of claim 15, wherein said instructions are further configured to execute Optical Character Recognition (OCR). ### Claim 19 (depends on 15) The system of claim 15, wherein said classifications are executed by a neural network. ### Claim 20 (depends on 19) The system of claim 19, wherein said neural network is trained on a set of labelled image data to learn characteristics for each type of object.