# US 10,311,579 B2 · Apparatus and method for detecting foreground in image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,579 B2 - Document ID: US10311579B2 - Status: Granted - Kind: B2 - Application number: 15/405,735 - Filed: 2017-01-13 - Published: 2017-07-27 - Granted: 2019-06-04 - Earliest claimed: 2016-01-22 - Assignees: Samsung Electronics Co., Ltd.; Seoul National University R&Db Foundation - Applicants: Samsung Electronics Co., Ltd.; Seoul National University R&DB Foundation - Inventors: Ki Min Yun; Jin Young Choi; Jong In Lim - Web page: https://patentgrep.com/patent/US10311579B2 ## CPC - G06T 7/194: involving foreground-background segmentation - G06T 7/11: Region-based segmentation - G06T 7/174: involving the use of two or more images - G06T 7/215: Motion-based segmentation - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/28: Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An apparatus and method for detecting a foreground in an image is provided, and the foreground detecting apparatus includes a context information estimator configured to estimate context information on a scene from an image frame of the image, a background model constructor configured to construct a background model of the image frame using the estimated context information, and a foreground detector configured to detect a foreground from the image frame based on the constructed background model. ## Claims ### Claim 1 An apparatus for detecting a foreground in an image, the apparatus comprising: a memory configured to store instructions; and a processor configured to execute the instructions to: estimate context information, including a foreground speed, on a scene from an image frame of the image; construct a background model of the image frame using the estimated context information; and detect a foreground from the image frame based on the constructed background model, wherein the foreground speed is estimated by calculating a speed using an optical flow of the image frame and a previous image frame, and correcting the speed based on a projective transform matrix that represents background motion. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the context information further includes an illumination change. ### Claim 3 (depends on 2) The apparatus of claim 2, wherein the processor is further configured to execute the instructions to: generate the projective transform matrix using the speed; and estimate the illumination change based on a difference between a mean of brightness intensities of the image frame and a mean of brightness intensities of the background model. ### Claim 4 (depends on 3) The apparatus of claim 3, wherein the previous image frame comprises an immediately previous image frame. ### Claim 5 (depends on 2) The apparatus of claim 2, wherein for the construction of the background model, the processor is further configured to execute the instructions to generate the background model of the image frame using a background model of the previous image frame and the background motion. ### Claim 6 (depends on 5) The apparatus of claim 5, wherein for the construction of the background model, the processor is further configured to execute the instructions to determine locations on the previous image frame based on a location of the previous image frame which corresponds to a first location of the image frame, and to generate a background model of the image frame for the first location by performing weighted summation on background models of the previous image frame for the determined locations. ### Claim 7 (depends on 6) The apparatus of claim 6, wherein for the construction of the background model, the processor is further configured to execute the instructions to correct a variance of the generated background model, in response to a value of a context variable based on the foreground speed being greater than or equal to a block size. ### Claim 8 (depends on 7) The apparatus of claim 7, wherein for the construction of the background model, the processor is further configured to execute the instructions to correct a variance of the background model of the first location by reflecting differences between a mean of the background model of the first location and means of the background models at the determined locations. ### Claim 9 (depends on 5) The apparatus of claim 5, wherein for the construction of the background model, the processor is further configured to execute the instructions to update the generated background model based on the illumination change. ### Claim 10 (depends on 9) The apparatus of claim 9, wherein for the updating of the generated background model, the processor is further configured to execute the instructions to update a mean of the generated background model by reflecting the illumination change, in response to value of a context variable being smaller than a block size, and to update the mean and a variance of the generated background model by reflecting the illumination change and additional information, in response to the value of the context variable being greater than or equal to the block size. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein the additional information comprises either one or both of a mean of block-specific brightness intensities and information on a difference between a mean of the generated background model and the image frame. ### Claim 12 (depends on 10) The apparatus of claim 10, wherein for the updating of the generated background model, the processor is further configured to execute the instructions to calculate a value of a time-variant variable in units of blocks starting from an initial image frame input from the camera to the image frame, and to adjust an update intensity of the generated background model using the value of the time-variant variable. ### Claim 13 (depends on 1) The apparatus of claim 1, wherein for the updating of the generated background model, the processor is further configured to execute the instructions to generate a foreground probability map by calculating a foreground probability for each pixel of the image frame based on the constructed background model, and extract the foreground from the image frame using the generated foreground probability map. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein for the extracting of the foreground, the processor is further configured to execute the instructions to classify each pixel of the image frame into any one or any combination of any two or more of the foreground, a candidate, and a background based on a comparison of the foreground probability of each pixel of the image frame with a threshold value, and to categorize pixels classified into the candidate into the foreground or the background based on applying watershed segmentation to the pixels classified into the candidate. ### Claim 15 A method of detecting a foreground in an image, the method comprising: estimating context information, including a foreground speed, on a scene from an image frame of the image; constructing a background model of the image frame using the estimated context information; correcting a variance of the constructed background model, dependent on whether a value of a context variable based on the foreground speed is greater than or equal to a block size; and detecting a foreground from the image frame based on the variance-corrected constructed background model. ### Claim 16 (depends on 15) The method of claim 15, wherein the context information further comprises an illumination change. ### Claim 17 (depends on 16) The method of claim 16, wherein the constructing of the background model comprises generating the background model of the image frame using a background model of a previous image frame based on information on a background motion. ### Claim 18 (depends on 17) The method of claim 17, wherein the constructing of the background model further comprises updating the constructed background model based on the illumination change. ### Claim 19 (depends on 18) The method of claim 18, wherein the updating of the background model further comprises, updating a mean of the constructed background model by reflecting the illumination change, in response to a value of the context variable being smaller than a block size, and updating a mean and the variance of the constructed background model by reflecting the illumination change and additional information, in response to the value of the context variable being greater than or equal to the block size. ### Claim 20 (depends on 15) The method of claim 15, wherein the detecting of the foreground comprises generating a foreground probability map by calculating a foreground probability for each pixel of the image frame based on the constructed background model, and extracting the foreground from the image frame using the generated foreground probability map. ### Claim 21 (depends on 15) A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 15. ### Claim 22 A digital device, comprising: an antenna; a cellular radio configured to transmit and receive data via the antenna according to a cellular communications standard; a touch-sensitive display; a camera configured to capture an image; a memory configured to store instructions; and a processor configured to execute the instructions to receive the image from the camera, estimate context information, including an illumination change, on a scene from an image frame of the image, construct a background model of the image frame using the estimated context information, update a mean of the constructed background model by reflecting the illumination change, dependent on whether a value of a context variable is smaller than a block size, and detect a foreground from the image frame based on the constructed background model.