# US 2019/0172237 A1 · Filling in an entity within a video > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0172237 A1 - Document ID: US20190172237A1 - Status: Published application - Kind: A1 - Application number: 15/831,142 - Filed: 2017-12-04 - Published: 2019-06-06 - Earliest claimed: 2017-12-04 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Donna K. Byron; Carmine Dimascio; Florian Pinel - Web page: https://patentgrep.com/patent/US20190172237A1 ## CPC - G06T 5/77: Retouching; Inpainting; Scratch removal - G06T 11/60: Creating or editing images; Combining images with text - G06T 2207/10016: Video; Image sequence - H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract A computer-implemented method according to one embodiment includes identifying an entity to be filled in within an image, determining a three-dimensional (3D) model for the entity, and filling in the entity within the image, utilizing the 3D model. ## Claims ### Claim 1 A computer-implemented method, comprising: identifying an entity to be filled in within a selected plurality of video frames; determining a three-dimensional (3D) model for the entity; and filling in the entity within the selected plurality of video frames, utilizing the 3D model. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the selected plurality of video frames include only a portion of a total number of video frames of a video, and the entity is identified from video frames that come before the selected plurality of video frames within the video. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the selected plurality of video frames include only a portion of a total number of video frames of a video, and the entity is identified from video frames that come after the selected plurality of video frames within the video. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, further comprising: comparing the selected plurality of video frames to other video frames that come before and after the selected plurality of video frames within a video to determine if the entity is located in one or more of the other video frames; and in response to determining that the entity is located within one or more of the other video frames, applying one or more image recognition techniques to the entity identified within the one or more of the other video frames. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the entity contains one or more missing or empty portions as shown within the selected plurality of video frames that result from correcting the selected plurality of video frames for perspective or distortion. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein metadata is associated with the entity as a result of the identifying, the metadata including a classification of the entity and one or more specific characteristics of the entity. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein determining the 3D model includes using edge detection to determine one or more contours of the entity within the selected plurality of video frames. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein determining the 3D model includes retrieving a plurality of different 3D models from a model library. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein the plurality of different 3D models are retrieved utilizing metadata associated with the entity. ### Claim 10 (depends on 8) The computer-implemented method of claim 8, wherein the 3D model is selected from the plurality of different 3D models utilizing additional metadata associated with the selected plurality of video frames, the additional metadata including a date and time the video frames were created and a geographical location where the video frames were created. ### Claim 11 (depends on 8) The computer-implemented method of claim 8, further comprising refining the plurality of different 3D models to determine the 3D model, utilizing information derived from other video frames of a video that includes the selected plurality of video frames. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein filling in the entity includes applying a transformation to the 3D model for the entity. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein filling in the entity includes using the transformation to project the 3D model onto a selected area within the selected plurality of video frames that needs to be filled. ### Claim 14 (depends on 1) The computer-implemented method of claim 1, wherein filling in the entity includes mapping each surface of the 3D model to the selected plurality of video frames. ### Claim 15 (depends on 14) The computer-implemented method of claim 14, wherein filling in the entity includes applying a texture to portions each surface that need to be filled. ### Claim 16 A computer program product for filling in an entity within a video, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a processor to cause the processor to perform a method comprising: identifying the entity to be filled in within a selected plurality of video frames, utilizing the processor; determining a three-dimensional (3D) model for the entity, utilizing the processor; and filling in the entity within the selected plurality of video frames, utilizing the 3D model and the processor. ### Claim 17 (depends on 16) The computer program product of claim 16, wherein the selected plurality of video frames include only a portion of a total number of video frames of a video, and the entity is identified from video frames that come before the selected plurality of video frames within the video. ### Claim 18 (depends on 16) The computer program product of claim 16, wherein the selected plurality of video frames include only a portion of a total number of video frames of a video, and the entity is identified from video frames that come after the selected plurality of video frames within the video. ### Claim 19 (depends on 16) The computer program product of claim 16, further comprising: comparing, utilizing the processor, the selected plurality of video frames to other video frames that come before and after the selected plurality of video frames within a video to determine if the entity is located in one or more of the other video frames; and in response to determining that the entity is located within one or more of the other video frames, applying, utilizing the processor, one or more image recognition techniques to the entity identified within the one or more of the other video frames. ### Claim 20 A system, comprising: a processor; and logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to: identify an entity to be filled in within a selected plurality of video frames; determine a three-dimensional (3D) model for the entity; and fill in the entity within the selected plurality of video frames, utilizing the 3D model.