# US 10,503,999 B2 · System for detecting salient objects in images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,999 B2 - Document ID: US10503999B2 - Status: Granted - Kind: B2 - Application number: 15/946,693 - Filed: 2018-04-05 - Published: 2018-10-25 - Granted: 2019-12-10 - Earliest claimed: 2015-03-24 - Assignees: HRL Laboratories, LLC - Applicants: HRL Laboratories, LLC - Inventors: Shankar R. Rao; Andras Horvath - Web page: https://patentgrep.com/patent/US10503999B2 ## CPC - G06V 10/82: using neural networks - G06F 18/2413: based on distances to training or reference patterns - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/431: {Frequency domain transformation; Autocorrelation} - G06V 10/764: using classification, e.g. of video objects - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/584: {of vehicle lights or traffic lights} - G08G 1/13: the indicator being in the form of a map - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G08G 1/164: {Centralised systems, e.g. external to vehicles} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) ## Abstract Described is a system for detecting salient objects in images. During operation, the system maps an input image into a frequency domain having a spectral magnitude. The spectral magnitude is replaced with weights from a weight matrix W. The frequency domain is then transformed with the weights to a saliency map in the image domain, the saliency map having pixels with pixel values. A squaring operation is then performed on the saliency map by squaring the pixel values to generate a pixel-value altered saliency map. A final saliency map is generated by filtering the pixel-value altered saliency map. A number of devices may then be operated based on the saliency map. ## Claims ### Claim 1 A system for detecting salient objects in images, the system comprising: one or more processors and a memory, the memory being a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform operations of: mapping an input image into a frequency domain having a spectral magnitude; replacing the spectral magnitude with weights from a weight matrix W; transforming the frequency domain with the weights to a saliency map in the image domain, the saliency map having pixels with pixel values; performing a squaring operation on the saliency map by squaring the pixel values to generate a pixel-value altered saliency map such that all values in the pixel-value altered saliency map are nonnegative; generating a final saliency map by filtering the pixel-value altered saliency map; and generating the weight matrix W, such that the weight matrix W is a foreground weight matrix F divided by the background weight matrix B, wherein the foreground weight matrix F is an average spectral magnitude of foreground regions from a training set, and wherein the background weight matrix B is an average spectral magnitude of background images. ### Claim 2 (depends on 1) The system as set forth in claim 1, further comprising an operation of controlling a device based on the saliency map. ### Claim 3 (depends on 1) The system as set forth in claim 1, further comprising operations of: designating a region within the final saliency map as an object; and causing an autonomous vehicle to perform a maneuver to avoid collision with the object. ### Claim 4 (depends on 1) The system as set forth in claim 1, wherein in filtering the pixel-value altered saliency map, salient edges within the pixel-value altered saliency map are blurred into salient regions. ### Claim 5 (depends on 1) The system as set forth in claim 1, further comprising operations of: classifying an object within the final saliency map; and displaying the classification on a display device. ### Claim 6 (depends on 1) The system as set forth in claim 1, further comprising an operation of sending the final saliency map to a cell phone or central monitoring facility. ### Claim 7 A system for detecting salient objects in images, the system comprising: one or more processors and a memory, the memory being a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform operations of: mapping an input image into a frequency domain having a spectral magnitude; replacing the spectral magnitude with weights from a weight matrix W; transforming the frequency domain with the weights to a saliency map in the image domain, the saliency map having pixels with pixel values: performing a squaring operation on the saliency map by squaring the pixel values to generate a pixel-value altered saliency map such that all values in the pixel-value altered saliency map are nonnegative; generating a final saliency map by filtering the pixel-value altered saliency map; generating the weight matrix W, such that the weight matrix W is a foreground weight matrix F divided by the background weight matrix B, wherein the foreground weight matrix F is an average spectral magnitude of foreground regions from a training set, and wherein the background weight matrix B is an average spectral magnitude of background images; designating a region within the final saliency map as an object; classifying an object within the final saliency map; displaying the classification on a display device; causing an autonomous vehicle to perform a maneuver to avoid collision with the object; and wherein in filtering the pixel-value altered saliency map, salient edges within the pixel-value altered saliency map are blurred into salient regions. ### Claim 8 A computer program product for detecting salient objects in images, the computer program product comprising: a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of: mapping an input image into a frequency domain having a spectral magnitude; replacing the spectral magnitude with weights from a weight matrix W; transforming the frequency domain with the weights to a saliency map in the image domain, the saliency map having pixels with pixel values; performing a squaring operation on the saliency map by squaring the pixel values to generate a pixel-value altered saliency map such that all values in the pixel-value altered saliency map are nonnegative; generating a final saliency map by filtering the pixel-value altered saliency map; and generating the weight matrix W, such that the weight matrix W is a foreground weight matrix F divided by the background weight matrix B, wherein the foreground weight matrix F is an average spectral magnitude of foreground regions from a training set, and wherein the background weight matrix B is an average spectral magnitude of background images. ### Claim 9 (depends on 8) The computer program product as set forth in claim 8, further comprising an operation of controlling a device based on the saliency map. ### Claim 10 (depends on 8) The computer program product as set forth in claim 8, further comprising operations of: designating a region within the final saliency map as an object; and causing an autonomous vehicle to perform a maneuver to avoid collision with the object. ### Claim 11 (depends on 8) The computer program product as set forth in claim 8, wherein in filtering the pixel-value altered saliency map, salient edges within the pixel-value altered saliency map are blurred into salient regions. ### Claim 12 (depends on 8) The computer program product as set forth in claim 8, further comprising operations of: classifying an object within the final saliency map; and displaying the classification on a display device. ### Claim 13 (depends on 8) The computer program product as set forth in claim 8, further comprising an operation of sending the final saliency map to a cell phone or central monitoring facility. ### Claim 14 A computer program product for detecting salient objects in images, the computer program product comprising: a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of: mapping an input image into a frequency domain having a spectral magnitude; replacing the spectral magnitude with weights from a weight matrix W; transforming the frequency domain with the weights to a saliency map in the image domain, the saliency map having pixels with pixel values; performing a squaring operation on the saliency map by squaring the pixel values to generate a pixel-value altered saliency map such that all values in the pixel-value altered saliency map are nonnegative; generating a final saliency map by filtering the pixel-value altered saliency map; generating the weight matrix W, such that the weight matrix W is a foreground weight matrix F divided by the background weight matrix B, wherein the foreground weight matrix F is an average spectral magnitude of foreground regions from a training set, and wherein the background weight matrix B is an average spectral magnitude of background images; designating a region within the final saliency map as an object; classifying an object within the final saliency map; displaying the classification on a display device; causing an autonomous vehicle to perform a maneuver to avoid collision with the object; and wherein in filtering the pixel-value altered saliency map, salient edges within the pixel-value altered saliency map are blurred into salient regions. ### Claim 15 A method for detecting salient objects in images, the method comprising acts of: mapping an input image into a frequency domain having a spectral magnitude; replacing the spectral magnitude with weights from a weight matrix W; transforming the frequency domain with the weights to a saliency map in the image domain, the saliency map having pixels with pixel values; performing a squaring operation on the saliency map by squaring the pixel values to generate a pixel-value altered saliency map such that all values in the pixel-value altered saliency map are nonnegative; generating a final saliency map by filtering the pixel-value altered saliency map; and generating the weight matrix W, such that the weight matrix W is a foreground weight matrix F divided by the background weight matrix B, wherein the foreground weight matrix F is an average spectral magnitude of foreground regions from a training set, and wherein the background weight matrix B is an average spectral magnitude of background images. ### Claim 16 (depends on 15) The method as set forth in claim 15, further comprising an operation of controlling a device based on the saliency map. ### Claim 17 (depends on 15) The method as set forth in claim 15, further comprising operations of: designating a region within the final saliency map as an object; and causing an autonomous vehicle to perform a maneuver to avoid collision with the object. ### Claim 18 (depends on 15) The method as set forth in claim 15, wherein in filtering the pixel-value altered saliency map, salient edges within the pixel-value altered saliency map are blurred into salient regions. ### Claim 19 (depends on 15) The method as set forth in claim 15, further comprising operations of: classifying an object within the final saliency map; and displaying the classification on a display device. ### Claim 20 (depends on 15) The method as set forth in claim 15, further comprising an operation of sending the final saliency map to a cell phone or central monitoring facility. ### Claim 21 A method for detecting salient objects in images, the method comprising acts of: mapping an input image into a frequency domain having a spectral magnitude; replacing the spectral magnitude with weights from a weight matrix W; transforming the frequency domain with the weights to a saliency map in the image domain, the saliency map having pixels with pixel values; performing a squaring operation on the saliency map by squaring the pixel values to generate a pixel-value altered saliency map such that all values in the pixel-value altered saliency map are nonnegative; generating a final saliency map by filtering the pixel-value altered saliency map; generating the weight matrix W, such that the weight matrix W is a foreground weight matrix F divided by the background weight matrix B, wherein the foreground weight matrix F is an average spectral magnitude of foreground regions from a training set, and wherein the background weight matrix B is an average spectral magnitude of background images; designating a region within the final saliency map as an object; classifying an object within the final saliency map; displaying the classification on a display device; causing an autonomous vehicle to perform a maneuver to avoid collision with the object; and wherein in filtering the pixel-value altered saliency map, salient edges within the pixel-value altered saliency map are blurred into salient regions.