# US 12,182,975 B2 · Method for processing image, related device and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,975 B2
- Document ID: US12182975B2
- Status: Granted
- Kind: B2
- Application number: 17/133,602
- Filed: 2020-12-23
- Published: 2021-12-30
- Granted: 2024-12-31
- Earliest claimed: 2020-06-30
- Assignees: Beijing Xiaomi Mobile Software Co., Ltd.
- Applicants: Beijing Xiaomi Mobile Software Co., Ltd.
- Inventors: Xiaotao Wang; Yanan Li; Guangqi Shao; Lei Lei
- Web page: https://patentgrep.com/patent/US12182975B2

## CPC

- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30196: Human being; Person
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method for processing an image includes first obtaining a media image. The media image has a target scene object at least partially shielded. A target shielding object is determined from the media image. The target shielding object at least partially shields the target scene object. Image inpainting is performed on a first region of the media image corresponding to the target shielding object to generate a target media image.

## Claims

### Claim 1

An image processing method, comprising:
obtaining a media image, the media image having a target scene object at least partially shielded;
determining a target shielding object from the media image, the target shielding object at least partially shielding the target scene object;
performing image inpainting on a first region of the media image corresponding to the target shielding object to generate a target media image;
obtaining parameters of capturing the media image, the parameters comprising at least two of time information of capturing the media image, location information of capturing the media image, orientation information of capturing the media image, season information of capturing the media image, illuminance information of capturing the media image, angle information of capturing the media image, and weather information of capturing the media image;
determining, based on at least two of the time information, the location information, the orientation information, the season information, the illuminance information, the angle information or the weather information, a subset of scene images matching the parameters from a set of scene images comprised in a cloud database; and
obtaining a scene image matching the target scene object from the subset of scene images,
wherein the obtaining the scene image matching the target ct from the subset of scene images comprises:
zeroing a feature matching weight for each scene object other than the target scene object in the media image, and determining from the subset of scene images a scene image having a highest matching degree with the target scene object as the scene image matching the target scene object.

### Claim 2 (depends on 1)

The method of claim 1, comprising:
performing removal processing on the first region of the media image corresponding to the target shielding object.

### Claim 3 (depends on 1)

The method of claim 1,
wherein the performing the image inpainting on the first region of the media image corresponding to the target shielding object to generate the target media image comprises:
performing image registration between the scene image and the target scene object, and determining from the scene image a second region corresponding to the first region; and
merging the second region and the media image to obtain the target media image.

### Claim 4 (depends on 1)

The method of claim 1, wherein the performing the image inpainting on the first region of the media image corresponding to the target shielding object to generate the target media image comprises:
performing, based on a pre-trained neural network model, image restoration on the first region of the media image corresponding to the target shielding object to generate the target media image.

### Claim 5 (depends on 1)

The method of claim 1, wherein the determining the target shielding object from the media image comprises:
performing pixel segmentation on shielding objects contained in the media image, to obtain a two-dimensional mask image of the shielding objects, the shielding objects at least partially shielding the target scene object, and a region where each shielding object is located in the two-dimensional mask image corresponding to a class label;
determining a region where a main shielding object is located in the two-dimensional mask image based on the class label; and
determining the shielding object contained in the two-dimensional mask image excluding the region where the main shielding object is located as the target shielding object.

### Claim 6 (depends on 5)

The method of claim 5, wherein the determining the region where the main shielding object is located in the two-dimensional mask image based on the class label comprises:
determining, based on a convolutional neural network, a region of pixels corresponding to a class label of the main shielding object as the region where the main shielding object is located; or
determining a region of pixels corresponding to a class label selected by a user as the region where the main shielding object is located.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining, based on at least two of the time information, the location information, the orientation information, the season information, the illuminance information, the angle information or the weather information, a subset of scene images matching the parameters from a set of scene images comprised in the cloud database comprises:
receiving, via a search box of the cloud database, the at least two of the time information, the location information, the orientation information, the season information, the illuminance information, the angle information or the weather information; and
searching from the cloud database for the subset of scene images whose characteristics matching the at least two of the time information, the location information, the orientation information, the season information, the illuminance information, the angle information or the weather information.

### Claim 8

A device for image processing, comprising:
a processor; and
a storage device configured to store an instruction executable by the processor;
wherein when the instruction stored in the storage device is executed by the processor, the processor is configured to:
obtain a media image, the media image having a target scene object at least partially shielded;
determine a target shielding object from the media image, the target shielding object at least partially shielding the target scene object;
perform image inpainting on a first region of the media image corresponding to the target shielding object to generate a target media image;
obtain parameters of capturing the media image, the parameters comprising at least two of time information of capturing the media image, location information of capturing the media image, orientation information of capturing the media image, season information of capturing the media image, illuminance information of capturing the media image, angle information of capturing the media image, and weather information of capturing the media image;
determine, based on at least two of the time information, the location information, the orientation information, the season information, the illuminance information, the angle information or the weather information, a subset of scene images matching the parameters from a set of scene images comprised in a cloud database; and
obtain the scene image matching the target scene object from the subset of scene images:
wherein the processor is configured to obtain the scene image matching the target scene object from the subset of scene images by,
zeroing a feature matching weight for each scene object other than the target scene object in the media image, and determining from the subset of scene images a scene image having a highest matching degree with the target scene object as the scene image matching the target scene object.

### Claim 9 (depends on 8)

The device of claim 8, wherein the processor is configured to perform the image inpainting on the first region of the media image corresponding to the target shielding object to generate the target media image by:
performing image registration between the scene image and the target scene object, and determining from the scene image a second region corresponding to the first region; and
merging the second region and the media image to obtain the target media image.

### Claim 10 (depends on 8)

The device of claim 8, wherein the processor is configured to perform the image inpainting on the first region of the media image corresponding to the target shielding object to generate the target media image by:
performing, based on a pre-trained neural network model, image restoration on the first region of the media image corresponding to the target shielding object to generate the target media image.

### Claim 11 (depends on 8)

The device of claim 8, wherein the processor is configured to determine the target shielding object from the media image by:
performing pixel segmentation on shielding objects contained in the media image, to obtain a two-dimensional mask image of the shielding objects, the shielding objects at least partially shielding the target scene object, and a region where each shielding object is located in the two-dimensional mask image corresponding to a class label;
determining a region where a main shielding object is located in the two-dimensional mask image based on the class label; and
determining the shielding object contained in the two-dimensional mask image excluding the region where the main shielding object is located as the target shielding object.

### Claim 12 (depends on 11)

The device of claim 11, wherein the processor is configured to determine the region where the main shielding object is located in the two-dimensional mask image based on the class label by:
determining, based on a convolutional neural network, a region of pixels corresponding to a class label of the main shielding object as the region where the main shielding object is located; or
determining a region of pixels corresponding to a class label selected by a user as the region where the main shielding object is located.

### Claim 13 (depends on 8)

A mobile terminal comprising the device of claim 8, further comprising a display screen configured to display the media image,
wherein the processor is further configured to:
perform removal processing on the first region of the media image corresponding to the target shielding object;
wherein the target media image is a pure background image without any target shielding object including humans, wherein the mobile terminal is configured to remove part of the media image within the first region in ensure that the target media image has a realistic effect as the pure background image.

### Claim 14 (depends on 8)

The device of claim 8, wherein the processor is further configured to:
receive, via a search box of the cloud database, the at least two of the time information, the location information, the orientation information, the season information, the illuminance information, the angle information or the weather information; and
search from the cloud database for the subset of scene images whose characteristics matching the at least two of the time information, the location information, the orientation information, the season information, the illuminance information, the angle information or the weather information.

### Claim 15

A non-transitory computer-readable storage medium, having an instruction stored thereon, wherein when the instruction in the storage medium is executed by a processor of a mobile terminal, the mobile terminal is configured to implement an image processing method, the method comprising:
obtaining a media image, the media image having a target scene object at least partially shielded;
determining a target shielding object from the media image, the target shielding object at least partially shielding the target scene object;
performing image inpainting on a first region of the media image corresponding to the target shielding object to generate a target media image;
obtaining parameters of capturing the media image, the parameters comprising at least two of time information of capturing the media image, location information of capturing the media image, orientation information of capturing the media image, season information of capturing the media image, illuminance information of capturing the media image, angle information of capturing the media image, and weather information of capturing the media image; and
determining, based on at least two of the time information, the location information, the orientation information, the season information, the illuminance information, the angle information or the weather information, a subset of scene images matching the parameters from a set of scene images comprised in a cloud database; and
obtaining a scene image matching the target scene object from the subset of scene images;
wherein the obtaining the scene image matching the target scene object from the subset of scene images comprises:
zeroing a feature matching weight for each scene object other than the target scene object in the media image, and determining from the subset of scene images a scene image having a highest matching degree with the target scene object as the scene image matching the target scene object.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the method further comprises:
performing removal processing on the first region of the media image corresponding to the target shielding object.

### Claim 17 (depends on 15)

The non-transitory computer-readable storage medium of claim 15,
wherein the performing the image inpainting on the first region of the media image corresponding to the target shielding object to generate the target media image comprises:
performing image registration between the scene image and the target scene object, and determining from the scene image a second region corresponding to the first region; and
merging the second region and the media image to obtain the target media image.

### Claim 18 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein determining, based on at least two of the time information, the location information, the orientation information, the season information, the illuminance information, the angle information or the weather information, a subset of scene images matching the parameters from a set of scene images comprised in the cloud database comprises:
receiving, via a search box of the cloud database, the at least two of the time information, the location information, the orientation information, the season information, the illuminance information, the angle information or the weather information; and
searching from the cloud database for the subset of scene images whose characteristics matching the at least two of the time information, the location information, the orientation information, the season information, the illuminance information, the angle information or the weather information.