# US 11,908,105 B2 · Image inpainting method, apparatus and device, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,105 B2 - Document ID: US11908105B2 - Status: Granted - Kind: B2 - Application number: 17/240,811 - Filed: 2021-04-26 - Published: 2021-08-12 - Granted: 2024-02-20 - Earliest claimed: 2018-12-21 - Assignees: Tencent Technology (Shenzhen) Company Limited - Applicants: Tencent Technology (Shenzhen) Company Limited - Inventors: Ye Tian; Zhibin Wang - Web page: https://patentgrep.com/patent/US11908105B2 ## CPC - G06T 5/77: Retouching; Inpainting; Scratch removal - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/11: Region-based segmentation - G06T 7/12: Edge-based segmentation - G06T 7/62: of area, perimeter, diameter or volume - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/10004: Still image; Photographic image - G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/20104: Interactive definition of region of interest [ROI] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An image inpainting method is performed at a computing device, the method including: displaying an original image, and determining a to-be-inpainted first original region and a second original region different from the first original region in the original image in accordance with a user input; identifying a target pixel and a target pixel block comprising the target pixel in the first original region; obtaining a target search region from the second original region according to the target pixel and the target pixel block comprising the target pixel, an area of the target search region being smaller than an area of the second original region; and searching the target search region for a reference pixel block matching the target pixel block and updating the original image by inpainting the target pixel block according to the reference pixel block, and displaying the updated image. ## Claims ### Claim 1 An image inpainting method, performed by a computing device, the method comprising: displaying an original image; determining a to-be-inpainted first original region and a second original region different from the first original region in the original image in accordance with a user input; identifying a target pixel and a target pixel block comprising the target pixel in the first original region; obtaining a target search region from the second original region according to the target pixel and the target pixel block comprising the target pixel, an area of the target search region being smaller than an area of the second original region; dividing the target pixel block along a boundary into a first region and a second region; in accordance with a determination that a first pixel is located in the first region having a larger intersection area with a region that is not inpainted, setting the pixel value of a pixel in a reference pixel block corresponding to the first pixel as the pixel value of the first pixel after inpainting; and in accordance with a determination that the first pixel is located in the second region, fusing the pixel value of the first pixel with the pixel value of the pixel in the reference pixel block corresponding to the first pixel as the pixel value of the first pixel after inpainting; and searching the target search region for a reference pixel block matching the target pixel block and updating the original image by inpainting the target pixel block according to the reference pixel block, and displaying the updated image. ### Claim 2 (depends on 1) The method according to claim 1, wherein the determining a to-be-inpainted first original region and a second original region different from the first original region in the original image comprises: determining, when a selection operation in the original image according to the user input is detected, a region corresponding to the selection operation as the first original region; and determining a region different from the first original region in the original image as the second original region. ### Claim 3 (depends on 2) The method according to claim 2, further comprising: displaying, during gradual enlargement of the region corresponding to the selection operation, a dynamic effect that each pixel in the region corresponding to the selection operation is switched from an original pixel value to a pixel value after inpainting, until the selection operation stops, while performing the operation of displaying the updated image. ### Claim 4 (depends on 1) The method according to claim 1, wherein the determining a to-be-inpainted first original region and a second original region different from the first original region in the original image comprises: determining, when a selection operation in the original image according to the user input is detected, a region corresponding to the selection operation; creating a labeled image according to the region corresponding to the selection operation, a size of the labeled image being equal to a size of the original image; obtaining a boundary between the region corresponding to the selection operation and an unselected region in the labeled image; and performing region division on the original image according to a position of the boundary, and determining the first original region and the second original region in the original image. ### Claim 5 (depends on 1) The method according to claim 1, wherein the obtaining a target search region from the second original region comprises: obtaining a circular region with the target pixel as a center, and determining an intersection region between the circular region and the second original region as the target search region. ### Claim 6 (depends on 5) The method according to claim 5, wherein the obtaining a circular region with the target pixel as a center, and determining an intersection region between the circular region and the second original region as the target search region comprises: obtaining a circular region with the target pixel as a center and a search distance as a radius, the search distance being equal to a product of an area of the first original region and a preset coefficient, the preset coefficient being greater than 0; and determining the intersection region between the circular region and the second original region as the target search region when the circular region intersects with the second original region; or expanding the search distance when the circular region does not intersect with the second original region, obtaining a circular region with the target pixel as a center and the expanded search distance as a radius, until the currently obtained circular region intersects with the second original region, and determining an intersection region between the currently obtained circular region and the second original region as the target search region. ### Claim 7 (depends on 1) The method according to claim 1, wherein the obtaining a target search region from the second original region comprises: obtaining a first region, the first region comprising the first original region, and the first original region being inscribed in the first region; obtaining a second region, the second region comprising the first region, an area of the second region being smaller than an area of the target image; and obtaining the target search region from a region except the first original region in the second region. ### Claim 8 (depends on 7) The method according to claim 7, wherein the obtaining the target search region from a region except the first original region in the second region comprises: obtaining a circular region with the target pixel as a center, and determining an intersection region of the circular region, the second region, and the second original region as the target search region. ### Claim 9 (depends on 8) The method according to claim 8, wherein the obtaining a circular region with the target pixel as a center, and determining an intersection region of the circular region, the second region, and the second original region as the target search region comprises: obtaining a circular region with the target pixel as a center and a search distance as a radius, the search distance being equal to a product of an area of the first original region and a preset coefficient; and determining, when the circular region intersects with a first intersection region, an intersection region between the circular region and the first intersection region as the target search region, the first intersection region being an intersection region between the second region and the second original region; or expanding the search distance when the circular region does not intersect with the first intersection region, obtaining a circular region with the target pixel as a center and the expanded search distance as a radius, until the currently obtained circular region intersects with the first intersection region, and determining an intersection region between the currently obtained circular region and the first intersection region as the target search region. ### Claim 10 (depends on 7) The method according to claim 7, wherein the obtaining the target search region from a region except the first original region in the second region comprises: extracting the second region from the target image, scaling down the second region to obtain a third region, and obtaining a first target search region from a region except a region scaled down from the first original region in the third region; the searching the target search region for a reference pixel block matching the target pixel block, and inpainting the target pixel block according to the reference pixel block comprises; searching the first target search region for a first reference pixel block matching a pixel block scaled down from the target pixel block; and inpainting, according to the first reference pixel block, the pixel block scaled down from the target pixel block; and scaling up the inpainted third region, to obtain a fourth region having the same area as the second region, and replacing the second region in the target image with the fourth region; or obtaining a second reference pixel block in the second region in the target image corresponding to the first reference pixel block, and inpainting the target pixel block according to the second reference pixel block. ### Claim 11 (depends on 1) The method according to claim 1, wherein the searching the target search region for a reference pixel block matching the target pixel block comprises: obtaining, from the target search region, a plurality of pixel blocks having the same size as the target pixel block; and obtaining similarities between the plurality of pixel blocks and the target pixel block, and determining, in the plurality of pixel blocks, a pixel block with the highest similarity to the target pixel block as the reference pixel block. ### Claim 12 (depends on 1) The method according to claim 1, wherein the searching the target search region for a reference pixel block matching the target pixel block comprises: obtaining, from the target search region, a plurality of pixel blocks matching the target pixel block; and performing a weighted fusion on the plurality of pixel blocks according to a similarity between each of the plurality of pixel blocks and the target pixel block, to obtain the reference pixel block. ### Claim 13 (depends on 1) The method according to claim 1, wherein the inpainting the target pixel block according to the reference pixel block comprises: determining, for a first pixel in the target pixel block, a pixel value of a pixel in the reference pixel block corresponding to the first pixel as a pixel value after inpainting of the first pixel; or fusing, for a first pixel in the target pixel block, a pixel value of a pixel in the reference pixel block corresponding to the first pixel with a pixel value of the first pixel, and determining a pixel value obtained through the fusion as a pixel value after inpainting of the first pixel. ### Claim 14 A computing device, comprising a memory and a processor, the memory storing computer-readable instructions that, when executed by the processor, cause the computing device to perform a plurality of operations including: displaying an original image; determining a to-be-inpainted first original region and a second original region different from the first original region in the original image in accordance with a user input; identifying a target pixel and a target pixel block comprising the target pixel in the first original region; obtaining a target search region from the second original region according to the target pixel and the target pixel block comprising the target pixel, an area of the target search region being smaller than an area of the second original region; dividing the target pixel block along a boundary into a first region and a second region; in accordance with a determination that a first pixel is located in the first region having a larger intersection area with a region that is not inpainted, setting the pixel value of a pixel in a reference pixel block corresponding to the first pixel as the pixel value of the first pixel after inpainting; and in accordance with a determination that the first pixel is located in the second region, fusing the pixel value of the first pixel with the pixel value of the pixel in the reference pixel block corresponding to the first pixel as the pixel value of the first pixel after inpainting; and searching the target search region for a reference pixel block matching the target pixel block and updating the original image by inpainting the target pixel block according to the reference pixel block, and displaying the updated image. ### Claim 15 (depends on 14) The computing device according to claim 14, wherein the determining a to-be-inpainted first original region and a second original region different from the first original region in the original image comprises: determining, when a selection operation in the original image according to the user input is detected, a region corresponding to the selection operation as the first original region; and determining a region different from the first original region in the original image as the second original region. ### Claim 16 (depends on 14) The computing device according to claim 14, wherein the determining a to-be-inpainted first original region and a second original region different from the first original region in the original image comprises: determining, when a selection operation in the original image according to the user input is detected, a region corresponding to the selection operation; creating a labeled image according to the region corresponding to the selection operation, a size of the labeled image being equal to a size of the original image; obtaining a boundary between the region corresponding to the selection operation and an unselected region in the labeled image; and performing region division on the original image according to a position of the boundary, and determining the first original region and the second original region in the original image. ### Claim 17 (depends on 14) The computing device according to claim 14, wherein the obtaining a target search region from the second original region comprises: obtaining a circular region with the target pixel as a center, and determining an intersection region between the circular region and the second original region as the target search region. ### Claim 18 (depends on 14) The computing device according to claim 14, wherein the obtaining a target search region from the second original region comprises: obtaining a first region, the first region comprising the first original region, and the first original region being inscribed in the first region; obtaining a second region, the second region comprising the first region, an area of the second region being smaller than an area of the target image; and obtaining the target search region from a region except the first original region in the second region. ### Claim 19 (depends on 14) The computing device according to claim 14, wherein the inpainting the target pixel block according to the reference pixel block comprises: determining, for a first pixel in the target pixel block, a pixel value of a pixel in the reference pixel block corresponding to the first pixel as a pixel value after inpainting of the first pixel; or fusing, for a first pixel in the target pixel block, a pixel value of a pixel in the reference pixel block corresponding to the first pixel with a pixel value of the first pixel, and determining a pixel value obtained through the fusion as a pixel value after inpainting of the first pixel. ### Claim 20 A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform a plurality of operations including: displaying an original image; determining a to-be-inpainted first original region and a second original region different from the first original region in the original image in accordance with a user input; identifying a target pixel and a target pixel block comprising the target pixel in the first original region; obtaining a target search region from the second original region according to the target pixel and the target pixel block comprising the target pixel, an area of the target search region being smaller than an area of the second original region; dividing the target pixel block along a boundary into a first region and a second region; in accordance with a determination that a first pixel is located in the first region having a larger intersection area with a region that is not inpainted, setting the pixel value of a pixel in a reference pixel block corresponding to the first pixel as the pixel value of the first pixel after inpainting; and in accordance with a determination that the first pixel is located in the second region, fusing the pixel value of the first pixel with the pixel value of the pixel in the reference pixel block corresponding to the first pixel as the pixel value of the first pixel after inpainting; and searching the target search region for a reference pixel block matching the target pixel block and updating the original image by inpainting the target pixel block according to the reference pixel block, and displaying the updated image.