# US 2019/0172236 A1 · Filling in an entity within an image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0172236 A1
- Document ID: US20190172236A1
- Status: Published application
- Kind: A1
- Application number: 15/831,090
- Filed: 2017-12-04
- Published: 2019-06-06
- Earliest claimed: 2017-12-04
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Donna K. Byron; Carmine Dimascio; Florian Pinel
- Web page: https://patentgrep.com/patent/US20190172236A1

## CPC

- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 7/00: Image analysis
- G06T 11/60: Creating or editing images; Combining images with text
- H04L 67/04: specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
- H04L 67/1097: for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10004: Still image; Photographic image
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06T 2207/20221: Image fusion; Image merging
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A computer-implemented method according to one embodiment includes identifying an entity to be filled in within an image, determining a three-dimensional (3D) model for the entity, and filling in the entity within the image, utilizing the 3D model.

## Claims

### Claim 1

A computer-implemented method, comprising:
identifying an entity within an image that includes one or more missing portions;
determining a three-dimensional (3D) model for the entity; and
filling in the one or more missing portions of the entity within the image, utilizing the 3D model.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the entity is identified in response to:
identifying a first attempt to fill in an area of the image including the entity, utilizing a tool within an image editing application for filling in regular structures; and
detecting that results of the first attempt have been undone by a user of the image editing application.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising correcting the image for distortion, wherein the one or more missing portions result from the correcting.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the entity is identified utilizing one or more image recognition techniques that utilize local image data to refine results.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein metadata is associated with the entity as a result of the identifying, the metadata including a classification of the entity and one or more specific characteristics of the entity.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein determining the 3D model includes using edge detection to determine one or more contours of the entity within the image.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein determining the 3D model includes retrieving a plurality of different 3D models from a model library.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein the plurality of different 3D models are retrieved utilizing metadata associated with the entity.

### Claim 9 (depends on 7)

The computer-implemented method of claim 7, wherein the 3D model is selected from the plurality of different 3D models utilizing additional metadata associated with the image, the additional metadata including a date and time the image was created and a geographical location where the image was created.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein filling in the entity includes applying a transformation to the 3D model for the entity.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein filling in the entity includes using the transformation to project the 3D model onto a selected area within the image that needs to be filled.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein filling in the entity includes mapping each surface of the 3D model to the image.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein filling in the entity includes applying a texture to portions each surface that need to be filled.

### Claim 14

A computer program product for filling in an entity within an image, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a processor to cause the processor to perform a method comprising:
identifying the entity within the image that includes one or more missing portions, utilizing the processor;
determining a three-dimensional (3D) model for the entity, utilizing the processor; and
filling in the one or more missing portions of the entity within the image, utilizing the 3D model and the processor.

### Claim 15 (depends on 14)

The computer program product of claim 14, wherein the entity is identified in response to:
identifying a first attempt to fill in an area of the image including the entity, utilizing a tool within an image editing application for filling in regular structures; and
detecting that results of the first attempt have been undone by a user of the image editing application.

### Claim 16 (depends on 14)

The computer program product of claim 14, wherein the entity contains one or more missing or empty portions as shown within the image that result from correcting the image for perspective or distortion.

### Claim 17 (depends on 14)

The computer program product of claim 14, wherein the entity is identified utilizing one or more image recognition techniques that utilize local image data to refine results.

### Claim 18 (depends on 1)

The computer-implemented method of claim 1, wherein filling in the one or more missing portions of the entity within the image creates a reconstructed entity and includes:
applying a transformation to the 3D model to manipulate the model so that it matches a shape and viewing angle of the entity,
using the transformation to project the 3D model onto the one or more missing portions of the entity,
mapping each surface of the 3D model to the image, and
applying a texture to the one or more missing portions of the entity that need to be filled, where the texture is learned from a texture of an adjacent surface outside of the one or more missing portions;
wherein the reconstructed entity is validated by:
performing image recognition on the reconstructed entity to obtain a first confidence score,
comparing the first confidence score to a second confidence score previously obtained for the image, and
determining a different 3D model for the entity in response to determining that the first confidence score is not greater than the second confidence score.

### Claim 19 (depends on 1)

The computer-implemented method of claim 1, wherein the entity is identified utilizing one or more image recognition techniques, and results of implementing the one or more image recognition techniques include metadata including an identification of the entity, a classification of the entity, and one or more specific characteristics of the entity.

### Claim 20

A system, comprising:
a processor; and
logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to:
identify an entity within an image that includes one or more missing portions;
determine a three-dimensional (3D) model for the entity; and
fill in the one or more missing portions of the entity within the image, utilizing the 3D model.
