# US 10,121,067 B2 · Image processing apparatus that determines processing target area of an image based on degree of saliency, image processing method, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,121,067 B2 - Document ID: US10121067B2 - Status: Granted - Kind: B2 - Application number: 14/850,599 - Filed: 2015-09-10 - Published: 2016-03-31 - Granted: 2018-11-06 - Earliest claimed: 2014-09-30 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Satoru Kobayashi - Web page: https://patentgrep.com/patent/US10121067B2 ## CPC - G02B 7/36: using image sharpness techniques {, e.g. image processing techniques for generating autofocus signals (image data processing per se G06T; in cameras having a solid state image sensor H04N23/67)} - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - H04N 23/61: based on recognised objects - H04N 23/672: {based on the phase difference signals} - H04N 23/71: Circuitry for evaluating the brightness variation - G06V 10/464: {using a plurality of salient features, e.g. bag-of-words [BoW] representations} - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An image processing apparatus is provided with a spatial information calculation unit for calculating spatial information of a subject, which is the information of an area in which the subject in an image is predicted to be present, a first area setting unit for setting a first area in the image based on the spatial information, a second area setting unit for setting a second area outside the first area, a first feature amount calculation unit for calculating a first feature amount of the first area, a second feature amount calculation unit for calculating a second feature amount of the second area, the second feature amount being a feature amount of the same type as the first feature amount, and an saliency calculation unit for calculating a degree of visual saliency of the subject. ## Claims ### Claim 1 An image processing apparatus comprising: at least one processor; a memory that includes instructions for the at least one processor to, by being executed by the at least one processor; set a search area based on spatial information of a subject, the spatial information being information of an area in which the subject in an image is predicted to be present; set a first area inside the search area but not outside the search area; set a second area outside the first area; calculate a first feature amount of the first area; calculate a second feature amount of the second area, the second feature amount being a feature amount of the same type as the first feature amount; and calculate a degree of visual saliency of the subject in the first area based on a difference between the first feature amount and the second feature amount, wherein in a case that a plurality of search areas are set, the first area and the second area are set and the degree of visual saliency is calculated in each of the search areas, and a processing target area is determined from the first areas of the plurality of search areas based on the calculated degree of visual saliency. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the spatial information includes at least one of a position and a size of the subject in the image. ### Claim 3 (depends on 2) The image processing apparatus according to claim 2, wherein a size of the first area is set based on the size of the subject contained in the spatial information. ### Claim 4 (depends on 2) The image processing apparatus according to claim 2, wherein a position of the first area is set based on the position of the subject contained in the spatial information. ### Claim 5 (depends on 1) The image processing apparatus according to claim 1, wherein a plurality of first areas are set in the search area. ### Claim 6 (depends on 1) The image processing apparatus according to claim 1, wherein the memory includes an instruction for the at least one processor to, by being executed by the at least one processor: calculate information of a focused area, wherein the spatial information of the subject is calculated based on the information of the focused area. ### Claim 7 (depends on 6) The image processing apparatus according to claim 6, wherein the information of the focused area is calculated based on information indicating whether or not the area is in focus during a focusing operation. ### Claim 8 (depends on 1) The image processing apparatus according to claim 1, wherein the memory includes an instruction for the at least one processor to, by being executed by the at least one processor: calculate information of an area in which a moving body is present, wherein the spatial information of the subject is calculated based on the information of the area in which the moving body is present. ### Claim 9 (depends on 8) The image processing apparatus according to claim 8, wherein the information of the area in which the moving body is present is calculated based on an absolute difference between a plurality of images. ### Claim 10 (depends on 1) The image processing apparatus according to claim 1, wherein the memory includes an instruction for the at least one processor to, by being executed by the at least one processor: calculate information of an area in which a subject is present, wherein spatial information of the subject is calculated based on the information of the area in which the subject is present. ### Claim 11 (depends on 10) The image processing apparatus according to claim 10, wherein the information of the area in which the subject is present is calculated in accordance with at least one of face detection information and human body detection information. ### Claim 12 (depends on 1) The image processing apparatus according to claim 1, wherein degree of visual saliency of only an area whose area measurement is larger than a predetermined value is calculated. ### Claim 13 (depends on 1) The image processing apparatus according to claim 1, wherein the first and second feature amounts include at least one of brightness, color, edge intensity, a brightness histogram, a color histogram, and an edge intensity histogram. ### Claim 14 (depends on 1) The image processing apparatus according to claim 1, further comprising a lens group including a focus lens. ### Claim 15 (depends on 14) The image processing apparatus according to claim 14, wherein the memory includes an instruction for the at least one processor to, by being executed by the at least one processor: perform an autofocus control using the focus lens so that the processing target area become in-focus state. ### Claim 16 (depends on 1) The image processing apparatus according to claim 1, wherein the second area is an area whose center is the same as that of the first area and is a remaining area of an area that is larger than the first area excluding the first area. ### Claim 17 An image processing method comprising: setting a search area based on spatial information of a subject, the spatial information being information of an area in which a subject in an image is predicted to be present; setting a first area inside the search area but not outside the search area; setting a second area outside the first area; calculating a first feature amount of the first area; calculating a second feature amount of the second area, the second feature amount being a feature amount of the same type as the first feature amount; and calculating a degree of visual saliency of the subject in the first area based on a difference between the first feature amount and the second feature amount, wherein in a case that a plurality of search areas are set, the first area and the second area are set and the degree of visual saliency is calculated in each of the search areas, and a processing target area is determined from the first areas of the plurality of search areas based on the calculated degree of visual saliency. ### Claim 18 A non-transitory computer readable storage medium storing a computer-executable program for executing a method for controlling an image processing apparatus using the computer, the method comprising: setting a search area based on spatial information of a subject, the spatial information being information of an area in which a subject in an image is predicted to be present; setting a first area inside the search area but not outside the search area; setting a second area outside the first area; calculating a first feature amount of the first area; calculating a second feature amount of the second area, the second feature amount being a feature amount of the same type as the first feature amount; and calculating a degree of visual saliency of the subject in the first area based on a difference between the first feature amount and the second feature amount, wherein in a case that a plurality of search areas are set, the first area and the second area are set and the degree of visual saliency is calculated in each of the search areas, and a processing target area is determined from the first areas of the plurality of search areas based on the calculated degree of visual saliency.