# US 12,112,458 B2 · Removal of objects from images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,458 B2 - Document ID: US12112458B2 - Status: Granted - Kind: B2 - Application number: 17/862,272 - Filed: 2022-07-11 - Published: 2024-01-11 - Granted: 2024-10-08 - Earliest claimed: 2022-07-11 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Shizhong Liu - Web page: https://patentgrep.com/patent/US12112458B2 ## CPC - G06T 5/77: Retouching; Inpainting; Scratch removal - H04N 5/2621: {Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/11: Region-based segmentation - G06T 7/174: involving the use of two or more images - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)} - H04N 23/632: {for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters} - H04N 23/80: Camera processing pipelines; Components thereof - G06T 2207/20092: Interactive image processing based on input by user - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Systems and techniques are provided for adjusting objects in images. For example, a process can include obtaining a first image of a scene from a camera. The scene can include a first object positioned at a first position and a second object positioned at a second position. The process can include obtaining a second image of the scene from the camera. The second image of the scene can include the first object positioned at the first position and the second object positioned at a third position different from the second position. The process can include generating an adjusted second image based on the second image. The adjusted second image can include the first object positioned at the first position. The second object at the third position is removed from the adjusted second image. The process can include displaying the adjusted second image on a display. ## Claims ### Claim 1 An apparatus for processing one or more images, comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: obtain a first image of a scene from a camera, wherein the scene includes a first object positioned at a first position and a second object positioned at a second position; obtain a second image of the scene from the camera, wherein the second image of the scene includes the first object positioned at the first position and the second object positioned at a third position, the third position being different from the second position; determine that the second object is an unwanted object based on at least one of a determination that a depth of the first object positioned at the first position in the first image is different from a depth of the second object positioned at the second position in the first image by greater than a threshold depth difference amount or a determination that a depth of the first object positioned at the first position in the second image is different from a depth of the second object positioned at the third position in the second image by greater than the threshold depth difference amount; generate, based on determining that the second object is the unwanted object, an adjusted second image based on the second image, wherein the adjusted second image includes the first object positioned at the first position and the second object at the third position is removed from the adjusted second image; and display the adjusted second image on a display, wherein the first image of the scene and the adjusted second image comprise images captured independent of obtaining a capture input. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the at least one processor is configured to: obtain a third image of the scene from the camera; obtain the capture input; and generate, based on obtaining the capture input, a capture image based on the third image of the scene, wherein the capture image includes the first object positioned at the first position and the second object positioned at the third position is removed from the capture image. ### Claim 3 (depends on 2) The apparatus of claim 2, wherein the second image of the scene and the third image of the scene are associated with a common identifier. ### Claim 4 (depends on 3) The apparatus of claim 3, wherein the third image of the scene comprises an image obtained prior to the capture input, wherein the third image of the scene is stored in the at least one memory. ### Claim 5 (depends on 4) The apparatus of claim 4, wherein the at least one memory comprises a zero shutter lag (ZSL) buffer. ### Claim 6 (depends on 1) The apparatus of claim 1, wherein the at least one processor is configured to: obtain an input associated with selecting the second object for removal from the scene; and determine that the second object is the unwanted object further based on the input associated with selecting the second object for removal from the scene. ### Claim 7 (depends on 1) The apparatus of claim 1, wherein the at least one processor is configured to: obtain an input associated with selecting the first object as a target object for inclusion in the adjusted second image; and include the first object in the adjusted second image based on obtaining the input. ### Claim 8 (depends on 7) The apparatus of claim 7, wherein the input associated with selecting the first object as the target object for inclusion in the adjusted second image comprises at least one or more of a physical contact, a gesture, a gaze direction, an input from an input device, or an audio input. ### Claim 9 (depends on 1) The apparatus of claim 1, wherein, to determine the second object is the unwanted object, the at least one processor is further configured to: determine that the first object is a previously photographed object and the second object is not a previously photographed object. ### Claim 10 (depends on 1) The apparatus of claim 1, wherein, to remove the second object from the adjusted second image, the at least one processor is configured to interpolate colors of pixels of the second image to generate pixel data for one or more pixels associated with the second object in the second image of the scene. ### Claim 11 (depends on 1) The apparatus of claim 1, wherein, to remove the second object from the adjusted second image, the at least one processor is configured to obtain pixels of the first image of the scene associated with the third position and replacing corresponding pixels of the second image of the scene associated with the third position. ### Claim 12 (depends on 1) The apparatus of claim 1, wherein the first image of the scene and the second image of the scene are preview images obtained prior to obtaining a capture input. ### Claim 13 (depends on 1) The apparatus of claim 1, wherein the second image of the scene is an image obtained subsequent to obtaining the capture input. ### Claim 14 (depends on 1) The apparatus of claim 1, wherein the at least one processor is configured to: obtain a capture input; obtain a third image of the scene, wherein the third image of the scene includes the first object at the first position and the second object; determine a fourth position of the second object in the scene; and generate an adjusted third image including the first object at the first position, wherein the second object is removed from the adjusted third image; and storing the third image. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the at least one processor is configured to: obtain a segmentation associated with the second object and the first image of the scene; associate the third image of the scene with the first image of the scene; and apply the segmentation associated with the second object and the first image of the scene to the third image of the scene. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein the first image has a first resolution and the third image has a second resolution, the second resolution being different from the first resolution. ### Claim 17 (depends on 14) The apparatus of claim 14, wherein, to generate the adjusted second image, the at least one processor is configured to: perform an inpainting process to remove the second object from the second image of the scene; and generate the adjusted third image comprises combining pixels of a fourth image of the scene associated with the fourth position with pixels of the third image of the scene, wherein the pixels of the fourth image of the scene associated with the fourth position include a portion of the scene obscured by the second object in the third image of the scene. ### Claim 18 (depends on 1) The apparatus of claim 1, wherein the at least one processor is configured to: obtain a fourth image of the scene from the camera, wherein the scene includes the first object positioned at the first position and the second object positioned at the second position, wherein the fourth image of the scene comprises an image captured independent of obtaining the capture input; generate an adjusted fourth image based on the fourth image, wherein the adjusted fourth image includes the first object positioned at the first position and the second object is removed from the adjusted fourth image; and display the adjusted fourth image on the display. ### Claim 19 (depends on 18) The apparatus of claim 18, wherein the at least one processor is configured to: obtain an input associated with removing the second object based on the first image of the scene. ### Claim 20 (depends on 18) The apparatus of claim 18, wherein the at least one processor is configured to: obtain a fifth image of the scene from the camera; obtain the capture input; generate, based on obtaining the capture input, a capture image based on the fifth image of the scene, wherein the capture image includes the first object positioned at the first position and the second object positioned at the second position is removed from the capture image; and store the capture image in the at least one memory. ### Claim 21 (depends on 1) The apparatus of claim 1, wherein, to determine that the second object is the unwanted object, the at least one processor is configured to determine whether the depth of the first object positioned at the first position in the first image is different from the depth of the second object positioned at the second position in the first image by greater than the threshold depth difference amount. ### Claim 22 (depends on 1) The apparatus of claim 1, wherein, to determine that the second object is the unwanted object, the at least one processor is configured to determine whether the depth of the first object positioned at the first position in the second image is different from the depth of the second object positioned at the third position in the second image by greater than the threshold depth difference amount. ### Claim 23 A method of processing one or more images, comprising: obtaining a first image of a scene from a camera, wherein the scene includes a first object positioned at a first position and a second object positioned at a second position; obtaining a second image of the scene from the camera, wherein the second image of the scene includes the first object positioned at the first position and the second object positioned at a third position, the third position being different from the second position; determining that the second object is an unwanted object based on at least one of a determination that a depth of the first object positioned at the first position in the first image is different from a depth of the second object positioned at the second position in the first image by greater than a threshold depth difference amount or a determination that a depth of the first object positioned at the first position in the second image is different from a depth of the second object positioned at the third position in the second image by greater than the threshold depth difference amount; generating, based on determining that the second object is the unwanted object, an adjusted second image based on the second image, wherein the adjusted second image includes the first object positioned at the first position and the second object at the third position is removed from the adjusted second image; and displaying the adjusted second image on a display, wherein the first image of the scene and the adjusted second image comprise images captured independent of obtaining a capture input. ### Claim 24 (depends on 23) The method of claim 23, further comprising: obtaining a third image of the scene from the camera; obtaining the capture input; and generating, based on obtaining the capture input, a capture image based on the third image of the scene, wherein the capture image includes the first object positioned at the first position and the second object positioned at the third position is removed from the capture image. ### Claim 25 (depends on 24) The method of claim 24, wherein the second image of the scene and the third image of the scene are associated with a common identifier. ### Claim 26 (depends on 25) The method of claim 25, wherein the third image of the scene comprises an image obtained prior to the capture input, wherein the third image of the scene is stored in a memory. ### Claim 27 (depends on 26) The method of claim 26, wherein the memory comprises a zero shutter lag (ZSL) buffer. ### Claim 28 (depends on 23) The method of claim 23, further comprising, subsequent to displaying the first image of the scene on the display, obtaining an input associated with selecting the second object for removal from the scene, wherein determining the second object is the unwanted object is further based on the input associated with selecting the second object for removal from the scene. ### Claim 29 (depends on 23) The method of claim 23, further comprising: obtaining an input associated with selecting the first object as a target object for inclusion in the adjusted second image; and include the first object in the adjusted second image based on obtaining the input. ### Claim 30 (depends on 29) The method of claim 29, wherein the input associated with selecting the first object as the target object for inclusion in the adjusted second image comprises at least one or more of a physical contact, a gesture, a gaze direction, an input from an input device, or an audio input. ### Claim 31 (depends on 29) The method of claim 29, wherein determining the second object is an unwanted object comprises: determining that the first object is a previously photographed object and the second object is not a previously photographed object. ### Claim 32 (depends on 23) The method of claim 23, wherein removing the second object from the adjusted second image comprises interpolating colors of pixels of the second image to generate pixel data for one or more pixels associated with the second object in the second image of the scene. ### Claim 33 (depends on 23) The method of claim 23, wherein removing the second object from the adjusted second image comprises obtaining pixels of the first image of the scene associated with the third position and replacing corresponding pixels of the second image of the scene associated with the third position. ### Claim 34 (depends on 23) There method of claim 23, wherein the first image of the scene and the second image of the scene are preview images obtained prior to obtaining a capture input. ### Claim 35 (depends on 23) The method of claim 23, wherein the second image of the scene is an image obtained subsequent to obtaining the capture input. ### Claim 36 (depends on 23) The method of claim 23, further comprising: obtaining a capture input; obtaining a third image of the scene, wherein the third image of the scene includes the first object at the first position and the second object; determining a fourth position of the second object in the scene; generating an adjusted third image including the first object at the first position, wherein the second object is removed from the adjusted third image; and storing the adjusted third image. ### Claim 37 (depends on 36) The method of claim 36, further comprising: obtaining a segmentation associated with the second object and the first image of the scene; and associating the third image of the scene with the first image of the scene; and applying the segmentation associated with the second object and the first image of the scene to the third image of the scene. ### Claim 38 (depends on 37) The method of claim 37, wherein the first image has a first resolution and the third image has a second resolution, the second resolution being different from the first resolution. ### Claim 39 (depends on 36) The method of claim 36, wherein: generating the adjusted second image comprises performing an inpainting process to remove the second object from the second image of the scene; and generating the adjusted third image comprises combining pixels of a fourth image of the scene associated with the fourth position with pixels of the third image of the scene, wherein the pixels of the fourth image of the scene associated with the fourth position include a portion of the scene obscured by the second object in the third image of the scene. ### Claim 40 (depends on 23) The method of claim 23, further comprising: obtaining a fourth image of the scene from the camera, wherein the scene includes the first object positioned at the first position and the second object positioned at the second position, wherein the fourth image of the scene comprises an image captured independent of obtaining the capture input; generating an adjusted fourth image based on the fourth image, wherein the adjusted fourth image includes the first object positioned at the first position and the second object is removed from the adjusted fourth image; and displaying the adjusted fourth image on the display. ### Claim 41 (depends on 40) The method of claim 40, further comprising obtaining an input associated with removing the second object based on the first image of the scene. ### Claim 42 (depends on 40) The method of claim 40, further comprising: obtaining a fifth image of the scene from the camera; obtaining the capture input; and generating, based on obtaining the capture input, a capture image based on the fifth image of the scene, wherein the capture image includes the first object positioned at the first position and the second object positioned at the second position is removed from the capture image; and storing the capture image. ### Claim 43 (depends on 23) The method of claim 23, wherein determining that the second object is the unwanted object comprises determining whether the depth of the first object positioned at the first position in the first image is different from the depth of the second object positioned at the second position in the first image by greater than the threshold depth difference amount. ### Claim 44 (depends on 23) The method of claim 23, wherein determining that the second object is the unwanted object comprises determining whether the depth of the first object positioned at the first position in the second image is different from the depth of the second object positioned at the third position in the second image by greater than the threshold depth difference amount.