# US 10,373,316 B2 · Images background subtraction for dynamic lighting scenarios > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,373,316 B2 - Document ID: US10373316B2 - Status: Granted - Kind: B2 - Application number: 15/492,961 - Filed: 2017-04-20 - Published: 2018-10-25 - Granted: 2019-08-06 - Earliest claimed: 2017-04-20 - Assignees: Ford Global Technologies, LLC - Applicants: Ford Global Technologies, LLC - Inventors: Bruno Sielly Jales Costa; Madeline J Goh - Web page: https://patentgrep.com/patent/US10373316B2 ## CPC - G06T 7/11: Region-based segmentation - G06T 7/194: involving foreground-background segmentation - G06T 7/254: involving subtraction of images - G06T 11/60: Creating or editing images; Combining images with text - G08B 21/18: Status alarms (G08B21/02 takes precedence) - H04N 7/183: {for receiving images from a single remote source} - G06T 2207/10024: Color image - G06T 2207/10152: Varying illumination - G06T 2207/30268: Vehicle interior - G08B 13/19604: {involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change} - G08B 13/19691: {Signalling events for better perception by user, e.g. indicating alarms by making display brighter, adding text, creating a sound} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method and an apparatus for background subtraction highly applicable in autonomous driving scenarios are described. The method involves a reduction of illumination effects by constructing a normality background image from a normality model based on a plurality of baseline images taken under different illuminating conditions. A subtracted image is obtained by subtracting the normality background image from a scene image pixel-wise (i.e., pixel-by-pixel). The scene image may contain one or more foreground objects. The foreground objects are identified by highlighting the pixels in the subtracted image whose intensity is more than a predetermined standard deviation in the normality model. An illumination-invariant color space transformation algorithm may optionally be utilized to further reduce the variant illumination effects. ## Claims ### Claim 1 A method of sending a signal to a passenger of a transportation vehicle, comprising: receiving a scene image of an interior of the transportation vehicle, the scene image containing a background and one or more foreground objects of interest; projecting a normality background image of the interior from a normality model, wherein the normality background image comprises a plurality of background pixels that represents the background without the one or more foreground objects; generating a subtracted image by subtracting the normality background image pixel-wise from the scene image; highlighting a plurality of pixels in the subtracted image based on a standard deviation threshold to identify at least one of the one or more foreground objects as a foreign object to the transportation vehicle; displaying the subtracted image including the plurality of highlighted pixels on a display device; and generating a text message, a beeping sound or a flashing light as the signal to remind the passenger of the foreign object, wherein the normality model comprises corresponding mean values and standard deviations of pixel intensity data associated with each of the plurality of background pixels of the normality background image, wherein each of the plurality of pixels highlighted in the subtracted image has a corresponding pixel intensity that is more than the standard deviation threshold of the respective pixel, and wherein the standard deviation threshold of the respective pixel is a multiple of the corresponding standard deviation associated with the respective pixel in the normality model. ### Claim 2 (depends on 1) The method of claim 1, further comprising: constructing the normality model based on a plurality of baseline images taken under a plurality of illuminating conditions respectively, wherein each of the baseline images contains the background excluding the one or more foreground objects. ### Claim 3 (depends on 2) The method of claim 2, wherein the pixel intensity data associated with each of the plurality of background pixels of the normality background image is obtained from the plurality of baseline images at a same locale of the respective pixel, and wherein the constructing of the normality model comprises: calculating the corresponding mean value of the pixel intensity data for each of the plurality of background pixels; and calculating the corresponding standard deviation of the pixel intensity data for each of the plurality of background pixels. ### Claim 4 (depends on 2) The method of claim 2, wherein the mean values and the standard deviations comprise a Gaussian distribution or a student-t distribution that substantially fits the plurality of baseline images statistically. ### Claim 5 (depends on 1) The method of claim 1, wherein the receiving of the scene image comprises assessing an illuminating condition of the scene image, wherein the projecting of the normality background image is based on the illuminating condition, and wherein the standard deviation threshold is determined based on the illuminating condition. ### Claim 6 A method, comprising: receiving a scene image containing a background and one or more foreground objects of interest; projecting a normality background image from a normality model, wherein the normality background image contains the background without the one or more foreground objects; converting the scene image to a first illumination-invariant image using an illumination-invariant color space transformation algorithm; converting the normality background image to a second illumination-invariant image using the illumination-invariant color space transformation algorithm; generating a first subtracted image by subtracting the normality background image pixel-wise from the scene image; generating a second subtracted image by subtracting the second illumination-invariant image pixel-wise from the first illumination-invariant image; highlighting a plurality of pixels in the first subtracted image based on a standard deviation threshold to obtain a residual image as an identification of at least one of the one or more foreground objects; applying a multiplying matrix operation between the residual image and the second subtracted image to generate an output image; displaying the output image on a display device; and generating a text message, a beeping sound or a flashing light in response to the identification. ### Claim 7 (depends on 6) The method of claim 6, wherein the normality background image comprises a plurality of background pixels, and wherein the normality model comprises corresponding mean values and standard deviations of pixel intensity data associated with each of the background pixels of the normality background image. ### Claim 8 (depends on 7) The method of claim 7, further comprising: constructing the normality model based on a plurality of baseline images taken under a plurality of illuminating conditions respectively, wherein each of the baseline images contains the background excluding the one or more foreground objects. ### Claim 9 (depends on 8) The method of claim 8, wherein the pixel intensity data associated with each of the plurality of background pixels of the normality background image is obtained from the plurality of baseline images at a same locale of the respective pixel, and wherein the constructing of the normality model comprises: calculating the corresponding mean value of the pixel intensity data for each of the plurality of background pixels; and calculating the corresponding standard deviation of the pixel intensity data for each of the plurality of background pixels. ### Claim 10 (depends on 6) The method of claim 6, wherein the highlighting of the pixels in the first subtracted image based on the standard deviation threshold comprises highlighting the pixels in the first subtracted image having corresponding pixel intensity data more than the standard deviation threshold. ### Claim 11 (depends on 6) The method of claim 6, wherein the receiving of the scene image comprises assessing an illuminating condition of the scene image, wherein the projecting of the normality background image is based on the illuminating condition, and wherein the standard deviation threshold is determined based on the illuminating condition. ### Claim 12 (depends on 6) The method of claim 6, wherein the multiplying matrix operation takes the residual image and the second subtracted image of a same dimension as input to produce the output image with the same dimension, and wherein the multiplying matrix operation multiplies pixel intensity data of a same pixel location in the residual image and the second subtracted image respectively. ### Claim 13 An apparatus implementable to an autonomous vehicle, comprising: a camera configured to monitor a scene of interest containing a background; and a processor communicatively coupled to the camera, the processor configured to perform acts comprising: controlling the camera to take a plurality of baseline images of the scene under a plurality of illuminating conditions, wherein the baseline images are absent from one or more foreground objects; controlling the camera to record a plurality of scene images; creating a normality background image based on the baseline images; generating a first subtracted image by subtracting the normality background image pixel-wise from one scene image of the plurality of scene images to determine whether the one or more foreground objects exist in the one scene image; generating, via an autonomous driving system of the autonomous vehicle, a signal as a reminder to a passenger of the autonomous vehicle in response to an existence of the one or more foreground objects in the one scene image; marking the one or more foreground objects in the first subtracted image as a residual image in response to the existence of the one or more foreground objects in the one scene image; converting the one scene image to a first illumination-invariant image using an illumination-invariant color space transformation algorithm; converting the normality background image to a second illumination-invariant image using the illumination-invariant color space transformation algorithm; generating a second subtracted image by subtracting the second illumination-invariant image pixel-wise from the first illumination-invariant image; applying a multiplying matrix operation between the residual image and the second subtracted image to generate an output image; and displaying the output image on a display device of the autonomous vehicle to the passenger. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the normality background image comprises a plurality of background pixels, and wherein, in creating of the normality background image, the processor is configured to construct a normality model from the plurality of the baseline images taken under the plurality of illuminating conditions, the normality model comprising corresponding mean values and standard deviations of pixel intensity data associated with each of the background pixels of the normality background image. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the pixel intensity data associated with each of the plurality of background pixels of the normality background image is obtained from the plurality of baseline images at a same locale of the respective pixel, and wherein, in constructing the normality model, the processor is configured to perform acts comprising: calculating the corresponding mean value of the pixel intensity data for each of the plurality of background pixels; and calculating the corresponding standard deviation of the pixel intensity data for each of the plurality of background pixels. ### Claim 16 (depends on 14) The apparatus of claim 14, wherein the mean values and the standard deviations comprise a Gaussian distribution or a student-t distribution that substantially fits the plurality of baseline images statistically. ### Claim 17 (depends on 13) The apparatus of claim 13, wherein the processor is further configured to perform a marking of a plurality of pixels in the first subtracted image based on a standard deviation threshold, and wherein the marking of the pixels in the first subtracted image comprises highlighting the pixels having corresponding pixel intensity data more than the standard deviation threshold. ### Claim 18 (depends on 13) The apparatus of claim 13, wherein the multiplying matrix operation takes the residual image and the second subtracted image of a same dimension as input to produce the output image with the same dimension, and wherein the multiplying matrix operation multiplies pixel intensity data of a same pixel location in the residual image and the second subtracted image respectively. ### Claim 19 (depends on 13) The apparatus of claim 13, wherein the signal comprises a text message, a beeping sound, a flashing light, or a combination thereof.