# US 2019/0378283 A1 · System and method for transforming video data into directional object count > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0378283 A1 - Document ID: US20190378283A1 - Status: Published application - Kind: A1 - Application number: 16/435,008 - Filed: 2019-06-07 - Published: 2019-12-12 - Earliest claimed: 2018-06-09 - Assignees: Lot Spot Inc. - Applicants: Lot Spot Inc. - Inventors: Terrance Edward Boult - Web page: https://patentgrep.com/patent/US20190378283A1 ## CPC - G06T 7/254: involving subtraction of images - G06N 20/00: Machine learning - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/215: Motion-based segmentation - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06T 2207/20081: Training; Learning - G06T 2207/30236: Traffic on road, railway or crossing - G06T 2207/30242: Counting objects in image - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present invention is a computer-implemented system and method for transforming video data into directional object counts. The method of transforming video data is uniquely efficient in that it uses only a single column or row of pixels in a video camera to define the background from a moving object, count the number of objects and determine their direction. By taking an image of a single column or row every frame and concatenating them together, the result is an image of the object that has passed, referred to herein as a sweep image. In order to determine the direction, two different methods can be used. Method one involves constructing another image using the same method. The two images are then compared, and the direction is determined by the location of the object in the second image compared to the location of the object in the first image. Due to this recording method, elongation or compression of the objects can occur because of acceleration or deceleration of the objects and can be uniquely utilized to determine the speed or movement path of the objects. The second method of determining direction involves comparing the object in the image to an established marker. The transformations can also be used to produce labeled data for training machine learning models: bounding-boxes provided in sweep image can be transformed to bound boxes in video, and boxes in video can be transformed into boxes in the sweep image. ## Claims ### Claim 1 A method of transforming video data into moving object counts comprising the steps of: extracting a 1-dimensional region from each frame of video; constructing a sweep image by appending a plurality of the 1-dimensional regions to form a 2-dimensional image; processing the 2-dimensional sweep image to determine each distinct object within the 2-dimensional image; and returning the count of detected objects. ### Claim 2 (depends on 1) The method according to claim 1, where the step of processing the 2-dimensional sweep image to determine each distinct object comprises the steps of: processing the 2-dimensional sweep image to estimate a 1-dimensional background region corresponding to no objects present; comparing the 1-dimensional regions of the 2-dimensional sweep image to the 1-dimensional background region to determine regions of significant changes; and analyzing regions of significant changes to determine separate objects. ### Claim 3 (depends on 1) The method according to claim 1, where the step of processing the 2-dimensional sweep image to determine each distinct object comprises the steps of: computing edge response perpendicular to the 1-dimensional region used to create the 2-dimensional sweep image; and connecting pairs of adjacent regions with significant edge response to determine separate objects. ### Claim 4 (depends on 1) The method according to claim 1, wherein a direction of object motion is estimated based on relative object position within the 1-dimensional slice. ### Claim 5 (depends on 1) The method according to claim 1, wherein: at least two 2-dimensional sweep images are constructed by extracting approximately parallel 1-dimensional regions from different regions from each frame of video; a location analysis step determines a location of each object in each 2-dimensional sweep image; a matching step is performed associating objects between the at least two 2-dimensional sweep images; and an analysis is performed to compute a direction of motion by comparing a relative location of matching objects between the at least two 2-dimensional sweep images. ### Claim 6 (depends on 5) The method according to claim 5, wherein background is computed as a 1-dimensional representation where each pixel is a pixel vise median of associated pixels in each 1-dimensional region making up the 2-dimensional sweep image. ### Claim 7 (depends on 5) The method according to claim 5, wherein the step of processing the 2-dimensional sweep image to determine each distinct object is accomplished using the quasi-connected components algorithm providing for multi-resolution processing with a plurality of thresholds. ### Claim 8 (depends on 5) The method according to claim 5 wherein the analysis to compute the direction of motion is augmented with a secondary process that matches visual features between the at least two 2-dimensional sweep images and fuses the resulting estimate with results based on matched object location. ### Claim 9 (depends on 8) The method according to claim 8, wherein the fusion of direction further includes an estimate of the direction of object motion based on relative object position within the 1-dimensional region. ### Claim 10 (depends on 1) The method according to claim 1, wherein the step of constructing the 2-dimensional sweep image is used to produce ground truth data for machine learning by using a secondary process analyzing the 2-dimensional sweep image to determine a ground truth bounding box of each object within the 2-dimensional sweep image, with a set of such ground truth bounding boxes being used in machine learning models to optimize parameters of object detection the 2-dimensional sweep image. ### Claim 12 (depends on 10) The method according to claim 10, wherein the step of constructing the 2-dimensional sweep image stores an identifier of each frame and Wherein detected object location is then used to determine first and last frames when the object was passing over a back-projection of the 1-dimensional region, with frames of video then being used for training a in chine learning model to optimize detection parameters from an original video. ### Claim 13 (depends on 11) The method according to claim 11, where data from the original frame in the neighborhood of the back-projected 1-dimensional region is used to determine a front and a rear of the object, which is then used to extract bounding boxes for each object to be used for machine learning to optimize a secondary systems detection parameters. ### Claim 14 An system for transforming a sequence of image data into moving object counts comprising: an input means for sequentially entering input images containing a moving object region to be counted; a sweep image generation means for extracting 1-dimensional regions for each image and combining them into a 2-dimensional sweep image; storage means for storing the 2-dimensional sweep image; object detection means for processing the 2-dimensional sweep image to detect object locations; counting means to process the detected object locations and determine object counts; and an output means that communicates the object count to an external system. ### Claim 15 (depends on 14) The system according to claim 14, wherein the object detection means estimates the expected 1-dimensional signal when no object is present and compares that with the 1-dimensional signals in the 2-dimensional sweep image to detect and localize each object. ### Claim 16 (depends on 14) The system according to claim 14, wherein the object detection means estimates edge features perpendicular to the 1-dimensional signal of the 2-dimensional sweep image and uses the estimated edge features to detect and localize each object. ### Claim 17 (depends on 14) The system according to claim 14, further including a direction computation means that combines the object locations from the object detection means and the direction of motion is estimated based on the relative object position within the 1-dimensional slice. ### Claim 18 (depends on 14) The system according to claim 14, wherein the 2-dimensional sweep image generation means produces at least two 2-dimensional sweep images from non-overlapping 1-dimensional regions, the object detection means produces object locations for each of the at least two 2-dimensional sweep images, and the system includes a direction computation means that combines the object locations from the object detection means and determines the direction of motion. ### Claim 19 (depends on 18) The system according to claim 18, wherein multiple estimates of direction of notion are fused. ### Claim 20 (depends on 14) The system according to claim 14, further including a ground-truth estimation means that estimated each object location within the 2-dimensional sweep image and a machine-learning means that uses the result of the ground estimation means to tune system parameters to improve performance.