# US 11,113,822 B2 · Moving object identification from a video stream > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,822 B2 - Document ID: US11113822B2 - Status: Granted - Kind: B2 - Application number: 16/540,268 - Filed: 2019-08-14 - Published: 2021-02-18 - Granted: 2021-09-07 - Earliest claimed: 2019-08-14 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Egan Francis Ford; Jonathan Dye; Dmitry Rekesh; Galina Grunin - Web page: https://patentgrep.com/patent/US11113822B2 ## CPC - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06F 18/211: Selection of the most significant subset of features - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06T 2207/10016: Video; Image sequence - G06T 2207/20084: Artificial neural networks [ANN] - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A video stream moving object identifier takes a series of video frames as input, reduces the scale of the video frames, then performs pixel analysis on the sequential video frames to identify moving objects. Once moving objects are identified, the moving objects are resized according to input rules for a neural network object classifier to make the resized objects the correct size to be input to the neural network object classifier. The moving objects are then sent to a neural network object classifier, which processes the objects and returns an identification of the moving objects. The neural network object classifier can operate using one or more whitelists and one or more blacklists. ## Claims ### Claim 1 An apparatus comprising: at least one processor; a memory coupled to the at least one processor; a video stream residing in the memory, wherein the video stream comprises a plurality of frames; and a video stream moving object identifier that processes the plurality of frames in the video stream by performing pixel analysis of sequential frames to identify a moving object, resizes the moving object according to input rules for a neural network object classifier, sends the resized moving object to the neural network object classifier, and receives from the neural network object classifier an identification of the resized moving object, wherein the resizing of the moving object comprises: resizing a large dimension of the moving object to be less than a largest maximum dimension for an input object for the neural network classifier, and filling a remainder of the moving object with random noise to make the moving object a desired size for the neural network object classifier. ### Claim 2 (depends on 1) The apparatus of claim 1 wherein the input rules for the neural network object classifier specify pixel size for objects input to the neural network object classifier. ### Claim 3 (depends on 1) The apparatus of claim 1 wherein the neural network object classifier processes a plurality of input objects in parallel. ### Claim 4 (depends on 1) The apparatus of claim 1 wherein, prior to performing the pixel analysis, the video stream moving object identifier reduces scale of the plurality of sequential frames. ### Claim 5 (depends on 4) The apparatus of claim 4 wherein the video stream moving object identifier reduces scale of the plurality of sequential frames by at least half. ### Claim 6 (depends on 1) The apparatus of claim 1 wherein the pixel analysis of sequential frames comprises a rolling average pixel difference for each pixel in the sequential frames. ### Claim 7 (depends on 6) The apparatus of claim 6 wherein the rolling average pixel difference for each pixel in the sequential frames is performed in a first pass for a smaller number of frames and in a second pass for a larger number of frames that is at least an order of magnitude larger than the smaller number of frames. ### Claim 8 (depends on 7) The apparatus of claim 7 wherein the pixel analysis places a bounding box around each moving object. ### Claim 9 A method for identifying moving objects in a video stream comprising a plurality of frames, the method comprising: processing the plurality of frames in the video stream by performing pixel analysis of sequential frames to identify a moving object; resizing the moving object according to input rules for a neural network object classifier; sending the resized moving object to the neural network object classifier; and receiving from the neural network object classifier an identification of the resized moving object, wherein the resizing the moving object comprises: resizing a large dimension of the moving object to be less than a largest maximum dimension for an input object for the neural network classifier; and filling a remainder of the moving object with random noise to make the moving object a desired size for the neural network object classifier. ### Claim 10 (depends on 9) The method of claim 9 wherein the input rules for the neural network object classifier specify pixel size for objects input to the neural network object classifier. ### Claim 11 (depends on 9) The method of claim 9 further comprising the neural network object classifier processing a plurality of input objects in parallel. ### Claim 12 (depends on 9) The method of claim 9 wherein, prior to performing the pixel analysis, reducing scale of the plurality of sequential frames. ### Claim 13 (depends on 12) The method of claim 12 wherein the reducing scale comprises reducing scale of the plurality of sequential frames by at least half. ### Claim 14 (depends on 9) The method of claim 9 wherein the pixel analysis of sequential frames comprises a rolling average pixel difference for each pixel in the sequential frames. ### Claim 15 (depends on 14) The method of claim 14 wherein the rolling average pixel difference for each pixel in the sequential frames is performed in a first pass for a smaller number of frames and in a second pass for a larger number of frames that is at least an order of magnitude larger than the smaller number of frames. ### Claim 16 (depends on 9) The method of claim 9 wherein the pixel analysis places a bounding box around each moving object. ### Claim 17 A method for identifying moving objects in a video stream comprising a plurality of frames, the method comprising: reducing scale of a plurality of sequential frames by at least half; processing the plurality of frames in the video stream by performing pixel analysis of sequential frames to identify a moving object using rolling average pixel difference for each pixel in the sequential frames, wherein the rolling average pixel difference for each pixel in the sequential frames is performed in a first pass for a smaller number of frames and in a second pass for a larger number of frames that is at least an order of magnitude larger than the smaller number of frames; resizing the moving object according to input rules for a neural network object classifier, wherein the input rules for the neural network object classifier specify pixel size for objects input to the neural network object classifier, wherein the resizing the moving object comprises: resizing a large dimension of the moving object to be less than a largest maximum dimension for an input object for the neural network classifier; and filling a remainder of the moving object with random noise to make the moving object a desired size for the neural network object classifier; sending the resized moving object to the neural network object classifier; the neural network object classifier processing the resized moving object and a plurality of other input objects in parallel; and receiving from the neural network object classifier an identification of the resized moving object.