# US 10,121,256 B2 · Temporal saliency map > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,121,256 B2 - Document ID: US10121256B2 - Status: Granted - Kind: B2 - Application number: 15/326,614 - Filed: 2014-08-28 - Published: 2017-07-13 - Granted: 2018-11-06 - Earliest claimed: 2014-08-28 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated; Xin Zhong; Yu Sun; Dashan Gao - Inventors: Xin Zhong; Yu Sun; Dashan Gao - Web page: https://patentgrep.com/patent/US10121256B2 ## CPC - G06T 7/248: {involving reference images or patches} - G06T 7/11: Region-based segmentation - G06T 7/174: involving the use of two or more images - G06T 7/187: involving region growing; involving region merging; involving connected component labelling - G06T 7/194: involving foreground-background segmentation - G06T 7/215: Motion-based segmentation - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/20076: Probabilistic image processing - G06T 2207/20132: Image cropping - G06T 2207/20156: Automatic seed setting - G06T 2207/20164: Salient point detection; Corner detection - G06T 2207/30201: Face - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method of generating a temporal saliency map is disclosed. In a particular embodiment, the method includes receiving an object bounding box from an object tracker. The method includes cropping a video frame based at least in part on the object bounding box to generate a cropped image. The method further includes performing spatial dual segmentation on the cropped image to generate an initial mask and performing temporal mask refinement on the initial mask to generate a refined mask. The method also includes generating a temporal saliency map based at least in part on the refined mask. ## Claims ### Claim 1 A method comprising: receiving information associated with an object bounding box from an object tracker; cropping a video frame based at least in part on the object bounding box to generate a cropped image; performing, at a processor, spatial dual-layer segmentation on the cropped image to generate an initial mask; performing, at the processor, temporal mask refinement on the initial mask to generate a refined mask; and generating a temporal saliency map based at least in part on the refined mask. ### Claim 2 (depends on 1) The method of claim 1, further comprising providing the temporal saliency map to a special effect application. ### Claim 3 (depends on 2) The method of claim 2, wherein the special effect application is a Bokeh effect application. ### Claim 4 (depends on 2) The method of claim 2, wherein the special effect application is a black and white effect application. ### Claim 5 (depends on 2) The method of claim 2, wherein the special effect application is a lens effect application. ### Claim 6 (depends on 1) The method of claim 1, further comprising generating a binary mask based on the temporal saliency map. ### Claim 7 (depends on 6) The method of claim 6, further comprising applying the binary mask to the video frame for a special effect application. ### Claim 8 (depends on 1) The method of claim 1, wherein performing the spatial dual-layer segmentation includes: performing region grow segmentation on a first layer of an image based on a first set of seed points; and performing region grow segmentation on a second layer of the image based on a second set of seed points. ### Claim 9 (depends on 8) The method of claim 8, wherein the first set of seed points includes a first plurality of points associated with a first boundary region that is expanded in size with respect to the object bounding box. ### Claim 10 (depends on 9) The method of claim 9, wherein the second set of seed points includes a second plurality of points associated with a second boundary region that is reduced in size with respect to the object bounding box. ### Claim 11 (depends on 10) The method of claim 10, further comprising fusing results of the region grow segmentation on the first layer of the image with results of the region grow segmentation on the second layer of the image to generate the initial mask. ### Claim 12 An apparatus comprising: a processor; an object tracker component executable by the processor to generate information associated with an object bounding box; a frame cropping component executable by the processor to crop a video frame based at least in part on the information associated with the object bounding box to generate a cropped image; a segmentation component executable by the processor to perform spatial multiple-layer segmentation on the cropped image to generate an initial mask; a mask refinement component executable by the processor to perform temporal mask refinement on the initial mask to generate a refined mask; and a temporal saliency map generation component executable by the processor to generate a temporal saliency map based at least in part on the refined mask. ### Claim 13 (depends on 12) The apparatus of claim 12, wherein the spatial multiple-layer segmentation includes spatial dual-layer segmentation. ### Claim 14 (depends on 12) The apparatus of claim 12, wherein the mask refinement component is executable by the processor to align the initial mask with one or more masks associated with one or more second video frames that precede the video frame in a sequence of video frames. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the mask refinement component is executable by the processor to align the initial mask with the one or more masks associated with the one or more second video frames by removing one or more pixels that represent outliers for a particular video frame. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein the one or more pixels appear in the particular video frame but do not appear in at least one of the one or more second video frames. ### Claim 17 (depends on 12) The apparatus of claim 12, further comprising a camera to capture the video frame. ### Claim 18 A method comprising: receiving information associated with an object bounding box from an object tracker; performing, at a processor, spatial dual-layer segmentation on a portion of a video frame to generate an initial mask, wherein a boundary of the portion of the video frame is determined based at least in part on the information associated with the object bounding box; performing, at the processor, temporal mask refinement on the initial mask to generate a refined mask; generating a temporal saliency map based at least in part on the refined mask; and providing the temporal saliency map as feedback to the object tracker. ### Claim 19 (depends on 18) The method of claim 18, wherein the temporal saliency map is provided to a normalized cross-correlation (NCC) verification component that is associated with a detection component of the object tracker. ### Claim 20 (depends on 19) The method of claim 19, wherein the NCC verification component is configured to apply the temporal saliency map to an image in order to separate a foreground portion of the image that includes an object of interest from a background portion of the image. ### Claim 21 (depends on 20) The method of claim 20, wherein the NCC verification component is configured to determine a subset of pixels within the object bounding box based on the temporal saliency map, wherein the subset of pixels are to be used by the NCC verification component for one or more comparison operations. ### Claim 22 (depends on 21) The method of claim 21, wherein the subset of pixels represents the foreground portion of the image that includes the object of interest. ### Claim 23 (depends on 22) The method of claim 22, wherein a second subset of pixels representing the background portion of the image is not used by the NCC verification component for the one or more comparison operations. ### Claim 24 (depends on 19) The method of claim 19, wherein the NCC verification component utilizes an object appearance model for object verification, and wherein the object appearance model is updated based at least in part on an output of the NCC verification component associated with the temporal saliency map. ### Claim 25 An apparatus comprising: a processor; an object tracker component executable by the processor to generate information associated with an object bounding box; a segmentation component executable by the processor to perform spatial dual-layer segmentation on a portion of a video frame to generate an initial mask, wherein a boundary of the portion of the video frame is determined based at least in part on the information associated with the object bounding box; a mask refinement component executable by the processor to perform temporal mask refinement on the initial mask to generate a refined mask; and a temporal saliency map generation component executable by the processor to: generate a temporal saliency map based at least in part on the refined mask; and provide the temporal saliency map as feedback to the object tracker component. ### Claim 26 (depends on 25) The apparatus of claim 25, wherein the temporal saliency map is further generated based on a saliency map associated with a second video frame that precedes the video frame in a sequence of video frames. ### Claim 27 (depends on 26) The apparatus of claim 26, wherein a first weighting factor is applied to the saliency map associated with the second video frame. ### Claim 28 (depends on 27) The apparatus of claim 27, wherein a second weighting factor is applied to the refined mask. ### Claim 29 (depends on 28) The apparatus of claim 28, wherein the second weighting factor is determined based on the first weighting factor. ### Claim 30 (depends on 28) The apparatus of claim 28, wherein the first weighting factor is larger than the second weighting factor to apply a larger weight to the saliency map associated with the second video frame than the refined mask.