# US 10,482,359 B2 · Systems and methods for removing non-stationary objects from imagery > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,359 B2 - Document ID: US10482359B2 - Status: Granted - Kind: B2 - Application number: 15/874,101 - Filed: 2018-01-18 - Published: 2019-07-18 - Granted: 2019-11-19 - Earliest claimed: 2018-01-18 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Alejandro Kauffmann; Ashley Ma; William Lindmeier; Andrew Dahley; Mark Ferdinand Bowers - Web page: https://patentgrep.com/patent/US10482359B2 ## CPC - G06T 5/77: Retouching; Inpainting; Scratch removal - G06K 15/1867: {Post-processing of the composed and rasterized print image (scanning, transmission or reproduction of documents H04N1/00)} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 11/60: Creating or editing images; Combining images with text - G11B 27/00: Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel - G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals - H04N 5/2621: {Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability} - H04N 23/60: Control of cameras or camera modules - H04N 23/633: {for displaying additional information relating to control or operation of the camera} - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/20024: Filtering details - G06T 2207/20221: Image fusion; Image merging - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The present disclosure provides image capture devices and associated methods of operation that enable the interactive editing of imagery, including, for example, removal and/or addition of non-stationary objects from and/or to captured imagery. According to one example aspect of the present disclosure, an image capture device can capture a plurality of frames of imagery that depict a scene and, while capturing the plurality of frames of imagery, provide a composite image within a viewfinder of the image capture device, where non-stationary objects included in the scene are removed from the composite image. According to another example aspect of the present disclosure, the image capture device can allow a user to edit a final composite image directly within the viewfinder. Thus, the present disclosure provides systems and methods that enable enhanced capture and interactive editing of imagery. ## Claims ### Claim 1 An image capture device, comprising: an image capture system configured to capture frames of imagery; a viewfinder configured to display at least a portion of a field of view of the image capture device; one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that when executed by the one or more processors cause the image capture device to perform operations, the operations comprising: capturing a plurality of frames of imagery over a period of time that depict a scene; and while capturing the plurality of frames of imagery and during the period of time, and for each of the plurality of frames of imagery as such frame of imagery is captured: receiving the frame of imagery; upon receipt of the frame of imagery, updating a composite image based on the frame of imagery of the plurality of frames of imagery, wherein updating the composite image comprises: detecting one or more edges in the frame of imagery that are associated with one or more non-stationary objects included in the scene; and removing from the composite image the one or more non-stationary objects included in the scene; and providing the composite image for display in the viewfinder, wherein the composite image depicts the one or more edges but not the one or more non-stationary objects. ### Claim 2 (depends on 1) The image capture device of claim 1, wherein removing from the composite image the one or more non-stationary objects included in the scene comprises, for each of a plurality of pixel positions in the composite image: identifying a first pixel value from a first frame of imagery of the plurality of frames of imagery that has a shortest distance in a color space to the other pixel values at such pixel position for the other frames of imagery; and selecting the first pixel value for inclusion in the composite image at such pixel position. ### Claim 3 (depends on 1) The image capture device of claim 1, wherein providing the composite image for display in the viewfinder comprises visualizing the one or more non-stationary objects dissolving over time. ### Claim 4 (depends on 3) The image capture device of claim 3, wherein visualizing the one or more non-stationary objects dissolving over time comprises visualizing the one or more non-stationary objects dissolving over time while also displaying an outline of each of the one or more non-stationary objects. ### Claim 5 (depends on 1) The image capture device of claim 1, wherein the operations further comprise: receiving a first user input that selects a first frame of imagery from the plurality of frames of imagery; providing the first frame of imagery for display within the viewfinder; receiving a second user input relative to a first location within the first frame of imagery; and in response to receipt of the second user input, modifying a final composite image to depict at least a first non-stationary object depicted at the first location in the first frame of imagery. ### Claim 6 (depends on 5) The image capture device of claim 5, wherein the operations further comprise: performing object segmentation on the first frame of imagery to identify a plurality of pixel values associated with the first non-stationary object in the first frame of imagery; wherein modifying the final composite image to depict at least the first non-stationary object depicted at the first location in the first frame of imagery comprises modifying the final composite image to include the plurality of pixel values associated with the first non-stationary object. ### Claim 7 (depends on 5) The image capture device of claim 5, wherein modifying the final composite image to depict at least the first non-stationary object depicted at the first location in the first frame of imagery comprises: determining a plurality of pixel values of the first frame of imagery that are included within an area around the first location associated with the second user input; and modifying the final composite image to include the plurality of pixel values. ### Claim 8 (depends on 5) The image capture device of claim 5, wherein: the viewfinder comprises a touch-sensitive display screen; and receiving the second user input relative to the first location comprises: displaying the first frame of imagery on the touch-sensitive display screen; and receiving a user touch input at a portion of the touch-sensitive display screen where the first location within the first frame of imagery is displayed. ### Claim 9 (depends on 8) The image capture device of claim 8, wherein receiving the first user input that selects the first frame of imagery from the plurality of frames of imagery comprises: receiving the first user input via an interactive tool that enables a user to scroll within the plurality of frames of imagery, wherein the first user input comprises a scroll input that scrolls to the first frame of imagery. ### Claim 10 (depends on 5) The image capture device of claim 5, wherein providing the first frame of imagery for display within the viewfinder comprises visually overlaying the first frame of imagery upon the final composite image. ### Claim 11 (depends on 5) The image capture device of claim 5, wherein providing the first frame of imagery for display within the viewfinder comprises visually highlighting at least the first non-stationary object depicted by the first frame of imagery. ### Claim 12 (depends on 1) The image capture device of claim 1, wherein the operations further comprise: providing a final composite image for display in the viewfinder; receiving a user input relative to a first location within the final composite image; and in response to receipt of the user input, modifying the final composite image to depict at least a first non-stationary object of the one or more non-stationary objects at the first location. ### Claim 13 One or more non-transitory computer-readable media that collectively store instructions that when executed by one or more processors of a computing system cause the computing system to perform operations, the operations comprising: obtaining a plurality of frames of imagery that depict a scene; generating a final composite image, wherein generating the final composite image comprises removing from the final composite image non-stationary objects included in the scene, and wherein removing from the final composite image non-stationary objects included in the scene comprises, for each of a plurality of pixel positions in the final composite image: identifying a first pixel value from a first frame of imagery of the plurality of frames of imagery that has a shortest distance in a color space to the other pixel values at such pixel position for the other frames of imagery; and selecting the first pixel value for inclusion in the final composite image at such pixel position; receiving a first user input that selects a first frame of imagery from the plurality of frames of imagery; providing the first frame of imagery for display; receiving a second user input relative to a first location within the first frame of imagery; and in response to receipt of the second user input, modifying the final composite image to depict at least a first non-stationary object depicted at the first location in the first frame of imagery. ### Claim 14 (depends on 13) The one or more non-transitory computer-readable media of claim 13, wherein: obtaining the plurality of frames of imagery comprises capturing the plurality of frames of imagery; and generating the final composite image comprises, while capturing the plurality of frames of imagery and for each of a plurality of iterations: generating an intermediate composite image from the plurality of frames of imagery, wherein generating the intermediate composite image comprises removing from the composite image one or more non-stationary objects included in the scene; and providing the intermediate composite image for display. ### Claim 15 (depends on 14) The one or more non-transitory computer-readable media of claim 14, wherein providing the intermediate composite image for display comprises displaying the generation of the intermediate composite image, wherein displaying the generation of the intermediate composite image comprises visualizing the one or more non-stationary objects dissolving over time. ### Claim 16 An apparatus comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that when executed by the one or more processors cause the apparatus to: obtain a plurality of image frames depicting a scene; for each of plural pixel positions in the image frames of the plurality of image frames, identify a pixel value at the pixel position in one of the image frames of the plurality of image frames that is the most similar in color to pixel values at the pixel position in the other image frames of the plurality of image frames, wherein the pixel value at the pixel position in the one of the image frames that is most similar in color to the pixel values at the pixel position in the other image frames is that which has a shortest distance in color space to the pixel values at the pixel position in the other image frames; and generate a composite image using the pixel values identified for each of the plural pixel positions. ### Claim 17 (depends on 16) The apparatus of claim 16, wherein the instructions, when executed by the one or more processors, cause the apparatus, while the plurality of images is being obtained, to iteratively: identify the pixel values and generate the composite image based on the image frames of the plurality of images that have already been obtained. ### Claim 18 (depends on 17) The apparatus of claim 17, wherein the instructions, when executed by the one or more processors, cause the apparatus to: render the iteratively generated composite image for display while the plurality of images is being obtained. ### Claim 19 (depends on 18) The apparatus of claim 18, wherein the instructions further cause the apparatus, while the plurality of images is being obtained, to iteratively detect one or more edges, and wherein the iteratively generated composite image depicts the one or more edges.