# US 10,372,977 B2 · Video processing for human occupancy detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,372,977 B2 - Document ID: US10372977B2 - Status: Granted - Kind: B2 - Application number: 14/794,991 - Filed: 2015-07-09 - Published: 2017-01-12 - Granted: 2019-08-06 - Earliest claimed: 2015-07-09 - Assignees: Analog Devices Gloval Unlimited Company - Applicants: Analog Devices Global Unlimited Company - Inventors: Raka Singh - Web page: https://patentgrep.com/patent/US10372977B2 ## CPC - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions - G06T 2207/10016: Video; Image sequence - G06T 2207/30196: Human being; Person - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Many conventional video processing algorithms attempting to detect human presence in a video stream often generate false positives on non-human movements such as plants moving in the wind, rotating fan, etc. To reduce false positives, a technique exploiting temporal correlation of non-human movements can accurately detect human occupancy while reject non-human movements. Specifically, the technique involves performing temporal analysis on a time-series signal generated based on an accumulation of foreground maps and an accumulation of motion map and analyzing the running mean and the running variance of the time-series signal. By determining whether the time-series signal is correlated in time, the technique is able to distinguish human movements and non-human movements. Besides having superior accuracy, the technique lends itself to an efficient algorithm which can be implemented on low cost, low power digital signal processor or other suitable hardware. ## Claims ### Claim 1 A method for determining occupancy in a video stream, comprising: processing a plurality of frames of the video stream, wherein for a given frame, processing comprises localizing one or more foreground areas in a foreground map and generating a motion map, wherein localizing the one or more foreground areas comprises assigning a given pixel to a foreground area if an integrated value of foreground pixel values for the given pixel over a moving window of foreground maps is greater than a threshold; generating a time-series signal based on the motion map and the localized one or more foreground areas; and determining whether there is occupancy in the video stream based on temporal analysis of the time-series signal, wherein the temporal determines whether peaks of the time-series signal are uncorrelated that indicates occupancy. ### Claim 2 (depends on 1) The method of claim 1, wherein processing the plurality of frames further comprises generating the foreground map having foreground pixel values identifying pixels of the given frame that are different from a background reference frame. ### Claim 3 (depends on 2) The method of claim 2, wherein generating the foreground map comprises: if a difference between a pixel value of a given pixel of the given frame and a corresponding pixel value of the background reference frame is greater than a second threshold, assigning a first foreground pixel value to the given pixel; and otherwise, assigning a second foreground pixel value to the given pixel. ### Claim 4 (depends on 1) The method of claim 1, wherein localizing the one or more foreground areas comprises rejecting pixels having episodic or periodic change in foreground pixel values over the moving window of foreground maps as not being part of the foreground area. ### Claim 5 (depends on 1) The method of claim 1, wherein: localizing the one or more foreground areas in the foreground map comprises integrating on a per pixel basis, foreground pixel values in the foreground map over a moving window of foreground maps; and the method further comprises integrating the foreground pixel values computing a weighted integration of the foreground pixel values for the given pixel over the moving window of foreground maps. ### Claim 6 (depends on 1) The method of claim 1, wherein processing the plurality of frames further comprises generating a difference map having difference pixel values identifying pixels of the given frame with pixel values different from corresponding pixel values of the pixels of a frame previous to the given frame. ### Claim 7 (depends on 6) The method of claim 6, wherein generating the difference map comprises: if a difference between a pixel value of a given pixel of the given frame and a corresponding pixel value of the frame previous to the given frame is greater than a second threshold, assigning a first difference pixel value to the given pixel; and otherwise, assigning a second difference pixel value to the given pixel. ### Claim 8 (depends on 1) The method of claim 1, wherein the temporal analysis comprises performing a running mean or a running variance of the time-series signal. ### Claim 9 (depends on 1) The method of claim 1, wherein performing temporal analysis comprises: computing a running mean of the time-series signal; and computing a finite difference of the running mean. ### Claim 10 (depends on 9) The method of claim 9, wherein performing temporal analysis comprises: computing a plurality of average values, wherein each average value is computed based on a window of values of the finite difference. ### Claim 11 (depends on 10) The method of claim 10, wherein performing temporal analysis comprises: if one or more ones of the plurality of average values in series is greater than a second threshold, determining occupancy; and otherwise, determining no occupancy. ### Claim 12 (depends on 1) The method of claim 1, wherein performing temporal analysis comprises: computing a running variance of the time-series signal; and computing a finite difference of the running variance. ### Claim 13 (depends on 12) The method of claim 12, wherein performing temporal analysis comprises: computing a plurality of average values, wherein each average value is computed based on a window of values of the finite difference. ### Claim 14 (depends on 13) The method of claim 13, wherein performing temporal analysis comprises: if one or more ones of the plurality of average values in series is greater than a second threshold, determining occupancy; and otherwise, determining no occupancy. ### Claim 15 (depends on 1) The method of claim 1, further comprising: in response to determining occupancy, triggering one or more classifiers for determining human presence in the one or more foreground areas. ### Claim 16 A computer-readable non-transitory medium comprising one or more instructions for determining occupancy in a video stream, that when executed on a processor, configure the processor to: process a plurality of frames of the video stream, wherein for a given frame, processing comprises localizing one or more foreground areas in a foreground map and generating a motion map; wherein: localizing the one or more foreground areas in the foreground map comprises integrating, on a per pixel basis, foreground pixel values in the foreground map over a moving window of foreground maps, and integrating the foreground pixel values comprises computing, using a first order infinite impulse response filter, an integrated value for a given pixel based on a weighted sum of a previous integrated value and a present foreground pixel value for the given pixel; generate a time-series signal based on the localized one or more foreground areas and the motion map; and determine whether there is occupancy in the video stream based on temporal analysis of the time-series signal, wherein the temporal analysis determines whether peaks of the time-series signal are uncorrelated that indicates occupancy. ### Claim 17 (depends on 16) The computer-readable non-transitory medium of claim 16, wherein the temporal analysis comprises performing a running mean or a running variance of the time-series signal. ### Claim 18 A method for determining occupancy in a video stream, comprising: processing a plurality of frames of the video stream, wherein for a given frame, processing comprises localizing one or more foreground areas in a foreground map and generating a motion map, wherein generating the motion map comprises accumulating, on a per pixel basis, difference pixel values over a moving window of difference maps, and summing motion pixel values of the motion map in the one or more foreground areas to generate a sample of a time-series signal; generating the time-series signal based on a plurality of samples of the time-series signal generated by the processing of the plurality of the frames of the video stream; and determining whether there is occupancy in the video stream based on temporal analysis of the time-series signal, wherein the temporal analysis determines whether peaks of the time-series signal are uncorrelated that indicates occupancy. ### Claim 19 (depends on 18) The method of claim 18, wherein generating the motion map further comprises: if an accumulated value of difference pixel values for a given pixel over the moving window of difference maps is greater than a threshold, assigning a first motion pixel value to the given pixel; and otherwise, assigning a second motion pixel value to the given pixel. ### Claim 20 (depends on 18) The method of claim 18, wherein accumulating difference pixel values further comprises computing a sum of difference pixel values for a given pixel over the moving window of difference maps. ### Claim 21 (depends on 18) The method of claim 18, wherein localizing the one or more foreground areas comprises assigning a given pixel to a foreground area if an integrated value of foreground pixel values for the given pixel over a moving window of foreground maps is greater than a threshold. ### Claim 22 (depends on 18) The method of claim 18, wherein: localizing the one or more foreground areas in the foreground map comprises integrating, on a per pixel basis, foreground pixel values in the foreground map over a moving window of foreground maps; and integrating the foreground pixel values comprises computing a weighted integration of the foreground pixel values for a given pixel over the moving window of foreground maps. ### Claim 23 A computer-readable non-transitory medium comprising one or more instructions for determining occupancy in a video stream, that when executed on a processor, configure the processor to: process a plurality of frames of the video stream, wherein for a given frame, processing comprises localizing one or more foreground areas in a foreground map and generating a motion map; generate a time-series signal based on the localized one or more foreground areas and the motion map; and determine whether there is occupancy in the video stream based on temporal analysis of the time-series signal, wherein the temporal analysis determines whether peaks of the time-series signal are uncorrelated that indicates occupancy. ### Claim 24 (depends on 23) The computer-readable non-transitory medium of claim 23, wherein generating the motion map comprises accumulating, on a per pixel basis, difference pixel values over a moving window of difference maps. ### Claim 25 (depends on 23) The computer-readable non-transitory medium of claim 23, wherein the temporal analysis comprises performing a running mean or a running variance of the time-series signal.