# US 12,604,087 B2 · Information processing apparatus, shooting system, method, and non-transitory computer-readable storage medium that determine whether image qualities are favorable > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,604,087 B2 - Document ID: US12604087B2 - Status: Granted - Kind: B2 - Application number: 18/067,919 - Filed: 2022-12-19 - Published: 2023-06-22 - Granted: 2026-04-14 - Earliest claimed: 2021-12-22 - Assignees: Canon Kabushiki Kaisha; Central Nippon Highway Engineering Tokyo Company Limited - Applicants: Canon Kabushiki Kaisha; Central Nippon Highway Engineering Tokyo Company Limited - Inventors: Mahoro Anabuki; Hirokuni Fujioka; Naoya Nishida; Masatsugu Sahara - Web page: https://patentgrep.com/patent/US12604087B2 ## CPC - H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image} - G06T 7/0002: {Inspection of images, e.g. flaw detection} - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 2207/20212: Image combination - G06T 2207/30168: Image quality inspection - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An information processing apparatus including a storage unit configured to store pieces of image quality information, respectively, for a plurality of images that show a subject, a determination unit configured to determine whether image qualities of the images are favorable based on the pieces of image quality information of the images, a composition unit configured to generate a composite image using images that have been determined to have favorable image qualities by the determination unit, and a notification unit configured, in a case when there is a missing pixel area in the composite image, to provide a notification about a re-shooting method for the subject corresponding to the missing pixel area, wherein the determination unit determines whether the image qualities of the images are favorable based on a shooting resolution, focus, and blurring. ## Claims ### Claim 1 An information processing apparatus comprising: at least one processor coupled to at least one memory, storing instructions that, when executed by the at least one processor, causes the image processing apparatus to function as: a storage unit configured to store pieces of image quality information, respectively, for a plurality of images that show a subject; a determination unit configured to determine whether image qualities of the images are favorable based on the pieces of image quality information of the images; a composition unit configured to generate a composite image using images that have been determined to have favorable image qualities by the determination unit; and a notification unit configured, in a case when there is a missing pixel area in the composite image, to provide a notification about a re-shooting method for the subject corresponding to the missing pixel area, wherein the determination unit determines whether the image qualities of the images are favorable based on a shooting resolution, focus, and blurring, wherein (1), in a case when the determination unit determines that shooting resolution is not favorable based on the pieces of the image quality information of the images, the notification unit provides a notification so that setting of focal length of a shooting unit is changed to a telephoto side, wherein (2), in a case when the determination unit determines that the focus is not unit provides a notification so that an f-number of the shooting unit is increased, wherein (3), in a case when the determination unit determines that the blurring is not favorable based on the pieces of the image quality information of the images, the notification unit provides a notification so that shutter speed of the shooting unit is increased, wherein (4), in a case when the determination unit determines that a reason for the missing pixel area is a shortage of overlap, the notification unit provides a notification so that speed at which the shooting range is changed when the shooting unit shoots the subject is re-set, and wherein (5), in a case when the determination unit determines that a reason for the missing pixel area is a failure in shooting, the notification unit provides a notification so that an interval at which a shutter of the shooting unit is released is reduced. ### Claim 2 (depends on 1) The information processing apparatus according to claim 1, wherein the image processing apparatus is further caused to function as an estimation unit configured, in a case when there is the missing pixel area in the composite image, to estimate a reason for missing pixels. ### Claim 3 (depends on 2) The information processing apparatus according to claim 2, wherein the estimation unit estimates a position of a used image in the composite image and estimates the reason for missing pixels based on whether there is the missing pixel area in an area surrounding the used image, the used image having been used to generate the composite image and having a shooting time that is different, by a predetermined range or less, from a shooting time of an unused image that has not been used to generate the composite image. ### Claim 4 (depends on 2) The information processing apparatus according to claim 2, wherein the estimation unit estimates the reason for missing pixels based on whether there is a missing image between an image located in an area surrounding the missing pixel area in the composite image and an image that is included among used images that have been used to generate the composite image and that is adjacent to the image in terms of a shooting order. ### Claim 5 (depends on 2) The information processing apparatus according to claim 2, wherein the estimation unit estimates the reason for missing pixels based on whether a size of the missing pixel area in the composite image exceeds a threshold. ### Claim 6 (depends on 2) The information processing apparatus according to claim 2, wherein the estimation unit estimates, based on at least one of a time of occurrence of a shooting failure event in the shooting unit for shooting the subject and a time at which the subject is shot when a moving speed of a mobile object on which the shooting unit is mounted has exceeded a threshold, a position of a used image with a shooting time that is different from the time by a predetermined range or less in the composite image, and estimates the reason for missing pixels, based on whether there is the missing pixel area in an area surrounding the used image. ### Claim 7 (depends on 2) The information processing apparatus according to claim 2, wherein the notification unit provides the notification about the re-shooting method for the subject corresponding to the missing pixel area based on the reason for missing pixels estimated by the estimation unit. ### Claim 8 (depends on 2) The information processing apparatus according to claim 2, wherein the reason for missing pixels includes at least one of the shortage of overlap, the failure in shooting, and poor image quality. ### Claim 9 (depends on 1) The information processing apparatus according to claim 1, wherein the notification unit provides the notification about the re-shooting method for the subject corresponding to the missing pixel area based on pieces of image quality information of unused images that have not been used to generate the composite image. ### Claim 10 (depends on 1) The information processing apparatus according to claim 1, wherein the notification unit notifies a terminal of a user of information in which a position of the missing pixel area in the composite image and the re-shooting method for the subject corresponding to the missing pixel area are associated with each other, and the notification unit provides, for each missing pixel area in the composite image, a notification about a re-shooting method of any one of (1) to (5). ### Claim 11 (depends on 1) The information processing apparatus according to claim 1, wherein the pieces of image quality information of the images include a plurality of elements with which the image qualities of the images are evaluated, and the determination unit determines whether the image qualities of the images are favorable based on at least one of the plurality of elements. ### Claim 12 (depends on 1) The information processing apparatus according to claim 1, wherein the storage unit stores a determination result of the determination made by the determination unit, and transmits, to the composition unit, images for which the determination result indicates that the image qualities are favorable. ### Claim 13 (depends on 1) The information processing apparatus according to claim 1, wherein the image processing apparatus is further caused to function as the shooting unit configured to shoot the subject. ### Claim 14 (depends on 1) The information processing apparatus according to claim 1, wherein the subject is a structure, and the information processing apparatus is used in an inspection of the structure. ### Claim 15 A shooting system comprising: a mobile object including a shooting unit configured to shoot a subject; and an information processing apparatus that processes images shot by the shooting unit, wherein the information processing apparatus includes: at least one processor coupled to at least one memory, storing instructions that, when executed by the at least one processor, causes the image processing apparatus to function as: a storage unit configured to store pieces of image quality information, respectively, for a plurality of images that show the subject; a determination unit configured to determine whether image qualities of the images are favorable based on the pieces of image quality information of the images; a composition unit configured to generate a composite image using images that have been determined to have favorable image qualities by the determination unit; and a notification unit configured, in a case when there is a missing pixel area in the composite image, to provide a notification about a re-shooting method for the subject corresponding to the missing pixel area, wherein the determination unit determines whether the image qualities of the images are favorable based on a shooting resolution, focus, and blurring, wherein (1), in a case when the determination unit determines that shooting resolution is not favorable based on the pieces of the image quality information of the images, the notification unit provides a notification so that setting of focal length of a shooting unit is changed to a telephoto side, wherein (2), in a case when the determination unit determines that the focus is not unit provides a notification so that an f-number of the shooting unit is increased, wherein (3), in a case when the determination unit determines that the blurring is not favorable based on the pieces of the image quality information of the images, the notification unit provides a notification so that shutter speed of the shooting unit is increased, wherein (4), in a case when the determination unit determines that a reason for the missing pixel area is a shortage of overlap, the notification unit provides a notification so that speed at which the shooting range is changed when the shooting unit shoots the subject is re-set, and wherein (5), in a case when the determination unit determines that a reason for the missing pixel area is a failure in shooting, the notification unit provides a notification so that an interval at which a shutter of the shooting unit is released is reduced. ### Claim 16 A shooting system comprising: a tripod head; a shooting unit configured to shoot a subject, the shooting unit being mounted on the tripod head; and an information processing apparatus that processes images shot by the shooting unit, wherein the information processing apparatus includes: at least one processor coupled to at least one memory, storing instructions that, when executed by the at least one processor, causes the shooting system to function as: a storage unit configured to store pieces of image quality information, respectively, for a plurality of images that show the subject; a determination unit configured to determine whether image qualities of the images are favorable based on the pieces of image quality information of the images; a composition unit configured to generate a composite image using images that have been determined to have favorable image qualities by the determination unit; and a notification unit configured, in a case when there is a missing pixel area in the composite image, to provide a notification about a re-shooting method for the subject corresponding to the missing pixel area, wherein the determination unit determines whether the image qualities of the images are favorable based on a shooting resolution, focus, and blurring, wherein (1), in a case when the determination unit determines that shooting resolution is not favorable based on the pieces of the image quality information of the images, the notification unit provides a notification so that setting of focal length of a shooting unit is changed to a telephoto side, wherein (2), in a case when the determination unit determines that the focus is not favorable based on the pieces of the image quality information of the images, the notification unit provides a notification so that an f-number of the shooting unit is increased, wherein (3), in a case when the determination unit determines that the blurring is not favorable based on the pieces of the image quality information of the images, the notification unit provides a notification so that shutter speed of the shooting unit is increased, wherein (4), in a case when the determination unit determines that a reason for the missing pixel area is a shortage of overlap, the notification unit provides a notification so that speed at which the shooting range is changed when the shooting unit shoots the subject is re-set, and wherein (5), in a case when the determination unit determines that a reason for the missing pixel area is a failure in shooting, the notification unit provides a notification so that an interval at which a shutter of the shooting unit is released is reduced. ### Claim 17 A method comprising: storing pieces of image quality information, respectively, for a plurality of images that show a subject; determining whether image qualities of the images are favorable based on the pieces of image quality information of the images; generating a composite image using images that have been determined to have favorable image qualities by the determining; and, in a case when there is a missing pixel area in the composite image, providing a notification about a re-shooting method for the subject corresponding to the missing pixel area, wherein the determining determines whether the image qualities of the images are favorable based on a shooting resolution, focus, and blurring, wherein (1), in a case when the determination unit determines that shooting resolution is not favorable based on the pieces of the image quality information of the images, the notification unit provides a notification so that setting of focal length of a shooting unit is changed to a telephoto side, wherein (2), in a case when the determination unit determines that the focus is not unit provides a notification so that an f-number of the shooting unit is increased, wherein (3), in a case when the determination unit determines that the blurring is not favorable based on the pieces of the image quality information of the images, the notification unit provides a notification so that shutter speed of the shooting unit is increased, wherein (4), in a case when the determination unit determines that a reason for the missing pixel area is a shortage of overlap, the notification unit provides a notification so that speed at which the shooting range is changed when the shooting unit shoots the subject is re-set, and wherein (5), in a case when the determination unit determines that a reason for the missing pixel area is a failure in shooting, the notification unit provides a notification so that an interval at which a shutter of the shooting unit is released is reduced. ### Claim 18 A non-transitory computer-readable storage medium storing a program that, when executed by a computer, causes the computer to perform a method of controlling an information processing apparatus, the method comprising: storing pieces of image quality information, respectively, for a plurality of images that show a subject; determining whether image qualities of the images are favorable based on the pieces of image quality information of the images; generating a composite image using images that have been determined to have favorable image qualities by the determining; and, in a case when there is a missing pixel area in the composite image, providing a notification about a re-shooting method for the subject corresponding to the missing pixel area, wherein the determining determines whether the image qualities of the images are favorable based on a shooting resolution, focus, and blurring, wherein (1), in a case when the determination unit determines that shooting resolution is not favorable based on the pieces of the image quality information of the images, the notification unit provides a notification so that setting of focal length of a shooting unit is changed to a telephoto side, wherein (2), in a case when the determination unit determines that the focus is not favorable based on the pieces of the image quality information of the images, the notification unit provides a notification so that an f-number of the shooting unit is increased, wherein (3), in a case when the determination unit determines that the blurring is not favorable based on the pieces of the image quality information of the images, the notification unit provides a notification so that shutter speed of the shooting unit is increased, wherein (4), in a case when the determination unit determines that a reason for the missing pixel area is a shortage of overlap, the notification unit provides a notification so that speed at which the shooting range is changed when the shooting unit shoots the subject is re-set, and wherein (5), in a case when the determination unit determines that a reason for the missing pixel area is a failure in shooting, the notification unit provides a notification so that an interval at which a shutter of the shooting unit is released is reduced.