# US 12,217,472 B2 · Segmenting and removing objects from visual media items > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,472 B2 - Document ID: US12217472B2 - Status: Granted - Kind: B2 - Application number: 17/968,634 - Filed: 2022-10-18 - Published: 2023-04-20 - Granted: 2025-02-04 - Earliest claimed: 2021-10-18 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Orly Liba; Nikhil Karnad; Nori Kanazawa; Yael Pritch Knaan; Huizhong Chen; Longqi Cai - Web page: https://patentgrep.com/patent/US12217472B2 ## CPC - G06V 10/273: {removing elements interfering with the pattern to be recognised} - G06T 5/20: using local operators - G06T 5/60: using machine learning, e.g. neural networks - G06T 5/77: Retouching; Inpainting; Scratch removal - G06T 5/94: based on local image properties, e.g. for local contrast enhancement - G06T 7/11: Region-based segmentation - G06T 7/155: involving morphological operators - G06T 7/194: involving foreground-background segmentation - G06T 11/00: Two-dimensional [2D] image generation - G06V 10/764: using classification, e.g. of video objects - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 20/20: in augmented reality scenes - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/20036: Morphological image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30184: Infrastructure - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A media application generates training data that includes a first set of visual media items and a second set of visual media items, where the first set of visual media items correspond to the second set of visual items and include distracting objects that are manually segmented. The media application trains a segmentation machine-learning model based on the training data to receive a visual media item with one or more distracting objects and to output a segmentation mask for one or more segmented objects that correspond to the one or more distracting objects. ## Claims ### Claim 1 A computer-implemented method comprising: generating training data that includes a first set of media items and a second set of media items, wherein the first set of media items include distracting objects and the second set of media items include manual segmentations of the distracting objects; identifying one or more original media items in the first set of media items that include one or more broken powerlines; generating one or more corrected media items that correct the one or more broken powerlines; generating one or more augmented media items for the training data by blending portions of the one or more corrected media items with portions of respective one or more original media items to increase a randomness of augmentation; and training a segmentation machine-learning model based on the training data to receive a media item with one or more distracting objects and to output a segmentation mask for one or more segmented objects that correspond to the one or more distracting objects. ### Claim 2 (depends on 1) The method of claim 1, wherein generating the one or more augmented media items includes blending the one or more corrected media items and the respective one or more original media items with a checkerboard mask. ### Claim 3 (depends on 1) The method of claim 1, wherein generating the one or more corrected media items to correct the one or more broken powerlines includes: modifying a local contrast in the one or more original media items to generate corresponding one or more enhanced media items. ### Claim 4 (depends on 3) The method of claim 3, wherein the local contrast is modified using a gain curve that adds two bias curves together. ### Claim 5 (depends on 1) The method of claim 1, wherein generating training data includes augmenting one or more of the first set of media items by applying a dilation to a segmentation mask of the one or more distracting objects. ### Claim 6 (depends on 1) The method of claim 1, wherein the one or more distracting objects are organized into categories, the categories including at least one selected from a group of powerlines, power poles, towers, and combinations thereof. ### Claim 7 (depends on 1) The method of claim 1, wherein training the segmentation machine-learning model comprises: generating a first machine-learning model based on the training data; and distilling the first machine-learning model to a trained segmentation machine-learning model by running inference on the training data that is segmented by the first machine-learning model. ### Claim 8 (depends on 1) The method of claim 1, wherein the training data further includes synthesized images with the distracting objects added in front of outdoor environment objects. ### Claim 9 A computer-implemented method to remove a distracting object from a media item, the method comprising: receiving a media item from a user; identifying one or more distracting objects in the media item; providing the media item to a trained segmentation machine-learning model; outputting, with the trained segmentation machine-learning model, a segmentation mask for the one or more distracting objects in the media item; and inpainting a portion of the media item that matches the segmentation mask to obtain an output media item, wherein the one or more distracting objects are absent from the output media item; wherein the trained segmentation machine-learning model is trained by generating training data by: identifying one or more original media items in a first set of media items that include one or more broken powerlines; generating one or more corrected media items that correct the one or more broken powerlines; and generating one or more augmented media items for the training data by blending portions of the one or more corrected media items with portions of respective one or more original media items to increase a randomness of augmentation. ### Claim 10 (depends on 9) The method of claim 9, wherein the one or more distracting objects are organized into categories, the categories including at least one selected from a group of powerlines, power poles, towers, and combinations thereof. ### Claim 11 (depends on 9) The method of claim 9, further comprising providing a suggestion to a user to remove the one or more distracting objects from the media item. ### Claim 12 (depends on 9) The method of claim 9, wherein the trained segmentation machine-learning model is trained using training data that includes the first set of media items and a second set of media items, wherein the first set of media items include distracting objects and the second set of media items include manual segmentations of the distracting objects. ### Claim 13 A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations, the operations comprising: generating training data that includes a first set of media items and a second set of media items, wherein the first set of media items include distracting objects and the second set of media items include manual segmentations of the distracting objects; identifying one or more original media items in the first set of media items that include one or more broken powerlines; generating one or more corrected media items that correct the one or more broken powerlines; generating one or more augmented media items for the training data by blending portions of the one or more corrected media items with portions of respective one or more original media items to increase a randomness of augmentation; and training a segmentation machine-learning model based on the training data to receive a media item with one or more distracting objects and to output a segmentation mask for one or more segmented objects that correspond to the one or more distracting objects. ### Claim 14 (depends on 13) The computer-readable medium of claim 13, wherein generating the one or more augmented media items includes blending the one or more corrected media items and the respective one or more original media items with a checkerboard mask. ### Claim 15 (depends on 13) The computer-readable medium of claim 13, wherein generating the one or more corrected media items to correct the one or more broken powerlines includes: modifying a local contrast in the one or more original media items to generate corresponding one or more enhanced media items. ### Claim 16 (depends on 15) The computer-readable medium of claim 15, wherein the local contrast is modified using a gain curve that adds two bias curves together. ### Claim 17 (depends on 13) The computer-readable medium of claim 13, wherein generating training data includes augmenting one or more of the first set of media items by applying a dilation to a segmentation mask of the one or more distracting objects. ### Claim 18 (depends on 13) The computer-readable medium of claim 13, wherein the one or more distracting objects are organized into categories, the categories including at least one selected from a group of powerlines, power poles, towers, and combinations thereof. ### Claim 19 (depends on 13) The computer-readable medium of claim 13, wherein training the segmentation machine-learning model comprises: generating a first machine-learning model based on the training data; and distilling the first machine-learning model to a trained segmentation machine-learning model by running inference on the training data that is segmented by the first machine-learning model. ### Claim 20 (depends on 13) The computer-readable medium of claim 13, wherein the training data further includes synthesized images with the distracting objects added in front of outdoor environment objects.