# US 10,504,240 B1 · Daytime heatmap for night vision detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,504,240 B1 - Document ID: US10504240B1 - Status: Granted - Kind: B1 - Application number: 15/787,527 - Filed: 2017-10-18 - Granted: 2019-12-10 - Earliest claimed: 2017-10-18 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Mashhour Solh; Manik Gupta - Web page: https://patentgrep.com/patent/US10504240B1 ## CPC - G06T 7/254: involving subtraction of images - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06T 5/40: using histogram techniques - G06T 7/11: Region-based segmentation - G06T 7/174: involving the use of two or more images - G06T 7/194: involving foreground-background segmentation - H04N 23/11: for generating image signals from visible and infrared light wavelengths - H04N 23/23: from thermal infrared radiation - H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes - H04N 23/73: by influencing the exposure time - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/10048: Infrared image - G06T 2207/10152: Varying illumination - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30196: Human being; Person - G06T 2207/30232: Surveillance - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A computer-implemented method includes receiving an image frame of image data associated with a camera. The method also includes receiving a human presence heatmap for the image frame, receiving a motion heatmap for the image frame, and receiving an image metric heatmap for the image frame. The method further includes receiving a combined heatmap as a combination of the human presence heatmap, the motion heatmap, and the image metric heatmap and setting auto-exposure parameters for the camera based on the combined heatmap. ## Claims ### Claim 1 A computer-implemented method of setting infrared auto-exposure parameters for a video camera, comprising: receiving a first image frame of RGB image data; receiving a second image frame of RGB image data, the second image frame of RGB image data including a plurality of pixels; determining that data in a first region of the second image frame indicates human presence in the first region, wherein the first region is defined by a first group of pixels of the plurality of pixels; generating a human presence heatmap associated with the first region, wherein each pixel of the first group has a normalized pixel weight between 0 and 1; determining, using the first image frame and the second image frame, motion in a second region of the second image frame, wherein the second region is defined by a second group of pixels of the plurality of pixels; generating a motion heatmap associated with the second region, wherein each pixel of the second group of pixels has a normalized pixel weight between 0 and 1; determining a group of saturated pixels in the second image frame, wherein the group of saturated pixels is characterized by a maximum pixel intensity value; generating an image quality heatmap associated with the group of saturated pixels, wherein each pixel of the group of saturated pixels has a normalized pixel weight between 0 and 1; generating a combined heatmap by summing: a first coefficient between zero and one multiplied by the normalized pixel weights of each pixel of the human presence heatmap; a second coefficient between zero and one multiplied by the normalized pixel weight of each pixel of the motion heatmap; and a third coefficient between zero and one multiplied by the normalized pixel weight of each pixel of the image quality heatmap; and deriving values for the infrared auto-exposure parameters for the video camera using the combined heatmap. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising performing a temporal update to the combined heatmap comprising: generating a second combined heatmap by summing: the first coefficient multiplied by pixel weights of each pixel of a second human presence heatmap for a third image frame; the second coefficient multiplied by pixel weights of each pixel of a second motion heatmap for the third image frame; and the third coefficient multiplied by pixel weights of each pixel of a second image quality heatmap; generating an updated combined heatmap by summing: an update factor multiplied by pixel weights of each pixel of the combined heatmap; and the second combined heatmap minus the update factor multiplied by the second combined heatmap. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein an auto-exposure metering table comprises a plurality of auto-exposure blocks, each having an exposure setting, wherein deriving values for the infrared auto-exposure parameters comprises averaging, for pixels in each of the plurality of auto-exposure blocks, pixel weights of pixels in the combined heatmap to determine a corresponding block average, and multiplying the exposure setting for the plurality of auto-exposure blocks times the corresponding block average. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, further comprising: receiving a first infrared image frame of infrared image data; receiving a second infrared image frame of infrared image data; generating an infrared human presence heatmap using the second infrared image frame; generating an infrared motion heatmap using the first infrared image frame and the second infrared image frame; updating the combined heatmap using the infrared human presence heatmap and the infrared motion heatmap to provide an updated combined heatmap; and updating the values for the infrared auto-exposure parameters for the video camera using the updated combined heatmap. ### Claim 5 A computer-implemented method, comprising: receiving an image frame of image data associated with a camera; receiving a human presence heatmap for the image frame; receiving a motion heatmap for the image frame; receiving an image metric heatmap for the image frame; receiving a combined heatmap as a combination of the human presence heatmap, the motion heatmap, and the image metric heatmap; and setting auto-exposure parameters for the camera based on the combined heatmap. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein the camera comprises a video camera, the method further comprising receiving additional image frames of image data, generating additional combined heatmaps using the additional image frames of image data, and defining updated auto-exposure parameters for the camera based on the combined heatmap. ### Claim 7 (depends on 5) The computer-implemented method of claim 5, wherein the image data comprises RGB image data and the auto-exposure parameters comprise infrared auto-exposure parameters. ### Claim 8 (depends on 5) The computer-implemented method of claim 5, wherein the human presence heatmap comprises one or more regions of the image frame including pixels having a normalized weight between 0 and 1. ### Claim 9 (depends on 5) The computer-implemented method of claim 5, wherein the motion heatmap comprises one or more additional regions of the image frame associated with motion and including pixels having a normalized weight between 0 and 1. ### Claim 10 (depends on 5) The computer-implemented method of claim 5, wherein the image frame includes a set of saturated pixels characterized by a maximum pixel intensity value and wherein the image metric heatmap comprises a set of pixels having a normalized pixel weight between 0 and 1, each of the pixels in the set of pixels being associated with one of the saturated pixels in the set of saturated pixels. ### Claim 11 (depends on 5) The computer-implemented method of claim 5, wherein the image metric heatmap comprises a set of pixels having a normalized pixel weight between 0 and 1, wherein the normalized pixel weight of each pixel in the set of pixels is a function of a saturation level of a corresponding pixel in the image frame. ### Claim 12 (depends on 5) The computer-implemented method of claim 5, further comprising receiving an infrared image frame associated with a scene illuminated with infrared illumination light from one or more infrared illumination sources coupled to the camera. ### Claim 13 (depends on 5) The computer-implemented method of claim 5, wherein generating the combined heatmap comprises adding: a first coefficient multiplied by the human presence heatmap; a second coefficient multiplied by the motion heatmap; and a third coefficient multiplied by the image metric heatmap. ### Claim 14 A video camera system comprising: an optical imaging lens; an image sensor; a plurality of infrared illumination sources; at least one processor; and a data storage device in communication with the at least one processor, wherein the data storage device comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to: receive a plurality of image frames of image data captured by the image sensor; generate a human presence heatmap for one of the plurality of image frames; generate a motion heatmap for the one of the plurality of image frames using a second one of the plurality of image frames preceding the one of the plurality of image frames; generate an image metric heatmap for the one of the plurality of image frames; generate a combined heatmap as a combination of the human presence heatmap, the motion heatmap, and the image metric heatmap; and define infrared auto-exposure parameters for the video camera system based on the combined heatmap. ### Claim 15 (depends on 14) The video camera system of claim 14, wherein generating the human presence heatmap comprises instructions stored on the data storage device that, when executed by the at least one processor, cause the at least one processor to: weight pixel values in one or more regions of the one of the plurality of image frames associated with human presence with a normalized pixel weight between 0 and 1. ### Claim 16 (depends on 14) The video camera system of claim 14, wherein generating the motion heatmap comprises instructions stored on the data storage device that, when executed by the at least one processor, cause the at least one processor to: weight pixel values in one or more additional regions of the one of the plurality of image frames associated with motion with a normalized pixel weight between 0 and 1. ### Claim 17 (depends on 14) The video camera system of claim 14, wherein generating the image metric heatmap comprises instructions stored on the data storage device that, when executed by the at least one processor, cause the at least one processor to: determine a saturation level for pixels in the one of the plurality of image frames; and weight pixel values for the pixels in the image metric heatmap with a normalized weight between 0 and 1, wherein the pixel values are inversely related to the saturation level of the pixels in the one of the plurality of image frames. ### Claim 18 (depends on 14) The video camera system of claim 14, wherein generating the combined heatmap comprises instructions stored on the data storage device that, when executed by the at least one processor, cause the at least one processor to add: a first coefficient multiplied by the human presence heatmap; a second coefficient multiplied by the motion heatmap; and a third coefficient multiplied by the image metric heatmap. ### Claim 19 (depends on 18) The video camera system of claim 18, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to: generate a second combined heatmap by summing: the first coefficient multiplied by pixel weights of each pixel of a second human presence heatmap for a third image frame; the second coefficient multiplied by pixel weights of each pixel of a second motion heatmap for the third image frame; and the third coefficient multiplied by pixel weights of each pixel of a second image metric heatmap for the third image frame; generate an updated combined heatmap by summing: an update factor multiplied by pixel weights of each pixel of the combined heatmap; and the second combined heatmap minus the update factor multiplied by the second combined heatmap. ### Claim 20 (depends on 14) The video camera system of claim 14, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to: receive a first infrared image frame of infrared image data; receive a second infrared image frame of infrared image data; generate an infrared human presence heatmap using the second infrared image frame; generate an infrared motion heatmap using the first infrared image frame and the second infrared image frame; update the combined heatmap using the infrared human presence heatmap and the infrared motion heatmap to provide an updated combined heatmap; and update the infrared auto-exposure parameters for the video camera system using the updated combined heatmap.