# US 12,182,976 B2 · Image processing method, smart device, and computer readable storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,976 B2 - Document ID: US12182976B2 - Status: Granted - Kind: B2 - Application number: 17/711,852 - Filed: 2022-04-01 - Published: 2022-07-14 - Granted: 2024-12-31 - Earliest claimed: 2020-04-15 - Assignees: Tencent Technology (Shenzhen) Company Limited - Applicants: Tencent Technology (Shenzhen) Company Limited - Inventors: Geyang Ke; Fei Huang; Wei Xiong - Web page: https://patentgrep.com/patent/US12182976B2 ## CPC - G06T 5/70: Denoising; Smoothing - G06T 5/77: Retouching; Inpainting; Scratch removal - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/60: using machine learning, e.g. neural networks - G06T 7/90: Determination of colour characteristics - G06V 10/56: relating to colour - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods} - G06V 10/7747: {Organisation of the process, e.g. bagging or boosting} - G06V 10/776: Validation; Performance evaluation - G06N 3/047: Probabilistic or stochastic networks - G06T 2207/10024: Color image - G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Methods, apparatuses and computer-readable storage mediums for processing images to remove moiré patterns. The method entails running an image processing model to perform moiré pattern removal processing on an original image to obtain a target image. The image processing model may include a network model trained according to a moiré pattern training data set. The image processing model can also include a multi-band module, which may be configured to process the original image to obtain an N-layer Laplacian pyramid of the original image, and obtain a first processing result feature map based on feature maps corresponding to N spatial frequency bands of the N-layer Laplacian pyramid, with N being a positive integer greater than or equal to 2. The image processing model can include obtaining the target image according to the first processing result feature map. ## Claims ### Claim 1 An image processing method, performed by an electronic device, the method comprising: obtaining a moiré pattern training data set, the moiré pattern training data set including matching image pairs, the matching image pair including a training image and a supervision image, wherein the training image has moiré patterns, and the supervision image does not have the moiré patterns, wherein obtaining the moiré pattern training data set comprises: disassembling each pixel in original image data into three sub pixels side by side to obtain a sub-pixel image, each sub pixel being corresponding to a color, and colors of the original image data comprising red, green, and blue (RGB); adjusting a size of the sub-pixel image to obtain a first intermediate image with a size the same as an image size of the original image data; setting grayscale values of pixels whose grayscale values are lower than a first threshold in the first intermediate image as a second threshold, to obtain a second intermediate image; adding radial distortion to the second intermediate image to obtain a first distorted image; and performing camera imaging analog adjustment on the first distorted image to obtain a training image with moiré patterns added; training an initial model according to each matching image pair in the moiré pattern training data set to obtain an image processing model; obtaining an original image; and running the image processing model to perform moiré pattern removal processing on the original image to obtain a target image, wherein the image processing model includes a multi-band module, the multi-band module being configured to process the original image to obtain an N-layer Laplacian pyramid of the original image, and obtain a first processing result feature map based on feature maps corresponding to N spatial frequency bands of the N-layer Laplacian pyramid, the target image being obtained according to the first processing result feature map, and N being a positive integer greater than or equal to 2. ### Claim 2 (depends on 1) The method according to claim 1, wherein running the image processing model comprises: running the image processing model to perform M times of initial analysis processing on the original image by using the multi-band module, to obtain initial feature maps in N scales, M being a positive integer greater than or equal to 2, M being greater than or equal to N, and the initial analysis processing comprising: down-sampling and performing convolution processing on the original image, or performing down-scaled convolution processing on the original image; performing convolution processing on the initial feature maps in N scales to obtain intermediate feature maps in N scales; obtaining the feature maps corresponding to the N spatial frequency bands of the N-layer Laplacian pyramid according to the intermediate feature maps in N scales; obtaining the first processing result feature map according to the feature maps corresponding to the N spatial frequency bands; and obtaining the target image according to the first processing result feature map. ### Claim 3 (depends on 2) The method according to claim 2, wherein in the feature maps corresponding to the N spatial frequency bands, a feature map of an Nth spatial frequency band is obtained according to an Nth intermediate feature map with a smallest scale in the intermediate feature maps; and in the feature maps corresponding to the N spatial frequency bands, a feature map of an (N-i)th spatial frequency band is obtained according to the Nth intermediate feature map, an (N-i)th intermediate feature map, and all intermediate feature maps between the Nth intermediate feature map and the (N-i)th intermediate feature map, i being a positive integer greater than or equal to 1 and less than N. ### Claim 4 (depends on 2) The method according to claim 2, wherein obtaining the first processing result feature map comprises: obtaining a predicted feature map, the predicted feature map being obtained according to an Nth intermediate feature map, a 1st intermediate feature map, and all intermediate feature maps between the Nth intermediate feature map and the 1st intermediate feature map; performing size adjustment on feature maps whose scales are smaller than a scale of the predicted feature map in the feature maps corresponding to the N spatial frequency bands, so that scales of the adjusted feature maps corresponding to the spatial frequency bands are equal to the scale of the predicted feature map; and obtaining the first processing result feature map according to the predicted feature map and the adjusted feature maps corresponding to the spatial frequency bands. ### Claim 5 (depends on 2) The method according to claim 2, wherein the image processing model further includes a prediction module, and obtaining the target image comprises: obtaining a blurred feature map obtained by blurring the original image; obtaining a second processing result feature map according to the first processing result feature map and the blurred feature map by using the prediction module; and obtaining the target image according to the second processing result feature map. ### Claim 6 (depends on 5) The method according to claim 5, wherein the image processing model further includes a super-resolution module, and obtaining the target image comprises: obtaining a reference feature map according to the first processing result feature map, and obtaining an intermediate result feature map by processing the second processing result feature map, a scale of the reference feature map being the same as a scale of the original image, and a scale of the intermediate result feature map being the same as the scale of the original image; and obtaining the target image according to the reference feature map and the intermediate result feature map by using the super-resolution module. ### Claim 7 (depends on 1) The method according to claim 1, wherein performing camera imaging analog adjustment comprises: performing perspective transformation processing on the first distorted image according to a first perspective transformation parameter to obtain a tilt analog image; processing the tilt analog image according to an image brightness distribution feature of camera imaging to obtain a brightness analog image; adding image noise to the brightness analog image to obtain a noise analog image with image noise; and processing the noise analog image according to a preset lighting coefficient to obtain a brightness-darkness analog image. ### Claim 8 (depends on 1) The method according to claim 1, wherein obtaining the moiré pattern training data set further comprises: displaying original image data on an electronic display screen, and photographing the original image data displayed on the electronic display screen to obtain a training image; performing feature matching and optical flow alignment processing on the original image data according to the training image, to obtain a supervision image; and constructing a matching image pair based on the training image and the corresponding supervision image. ### Claim 9 (depends on 1) The method according to claim 1, wherein the initial model includes: a multi-band module to be trained, a prediction module to be trained, and a super-resolution module to be trained; and training the initial model comprises: processing the training image in the matching image pair of the moiré pattern training data set as an input of the initial model by using the initial model, to obtain a first result outputted by the multi-band module, a second result outputted by the prediction module, and a third result outputted by the super-resolution module, the first result including feature maps of N spatial frequency bands corresponding to an N-layer Laplacian pyramid of the training image, the second result including a second result feature map corresponding to the training image, and the third result including a target image corresponding to the training image; performing N-layer Laplacian pyramid processing on the supervision image in the matching image pair of the moiré pattern training data set to obtain feature maps of N spatial frequency bands corresponding to the supervision image; obtaining N basic loss function values of the feature maps of the N spatial frequency bands corresponding to the supervision image and the feature maps of the N spatial frequency bands in the first result, obtaining a first loss function value between an image obtained after resolution processing is performed on the supervision image and the second result, and obtaining a second loss function value between the supervision image and the third result; and adjusting model parameters of the initial model according to the N basic loss function values, the first loss function value, and the second loss function value, to obtain the image processing model. ### Claim 10 (depends on 1) The method according to claim 1, wherein training the initial model comprises: performing size adjustment on images in a target matching image pair of the moiré pattern training data set to obtain P deformed matching image pairs of different sizes; and training the initial model according to the P deformed matching image pairs to obtain P loss function values, and adjusting model parameters of the initial model according to the P loss function values, to obtain the image processing model. ### Claim 11 (depends on 1) The method according to claim 1, wherein the model obtained through training according to each matching image pair in the moiré pattern training data set is used as a first image processing model, and the method further comprises: performing size enlargement on the training image in the moiré pattern training data set to obtain an enlarged moiré pattern image of a target size; running the first image processing model to perform moiré pattern removal processing on the enlarged moiré pattern image to obtain a moiré pattern supervision image; scaling the moiré pattern supervision image and the enlarged moiré pattern image to obtain a plurality of moiré pattern image pairs; and adjusting model parameters of the first image processing model according to the plurality of moiré pattern image pairs to obtain the image processing model. ### Claim 12 (depends on 11) The method according to claim 11, wherein the model obtained by adjusting the model parameters of the first image processing model according to the plurality of moiré pattern image pairs is used as a first version of image processing model; the first version of image processing model is used as a new first image processing model, to train the new first image processing model according to the moiré pattern training data set to obtain a second version of image processing model; the image processing model includes: the first version of image processing model and the second version of image processing model; and running the image processing model comprises: running the first version of image processing model and the second version of image processing model to perform moiré pattern removal processing on the original image to obtain a first image and a second image; and obtaining the target image according to the first image and the second image. ### Claim 13 An image processing method, performed by an electronic device, the method comprising: obtaining a moiré pattern training data set, the moiré pattern training data set including matching image pairs, the matching image pair including a training image and a supervision image, wherein the training image has moiré patterns, and the supervision image does not have the moiré patterns; training an initial model according to each matching image pair in the moiré pattern training data set to obtain an image processing model, wherein the initial model includes a multi-band module to be trained, a prediction module to be trained, and a super-resolution module to be trained, and wherein training the initial model comprises: processing the training image in the matching image pair of the moiré pattern training data set as an input of the initial model by using the initial model, to obtain a first result outputted by the multi-band module, a second result outputted by the prediction module, and a third result outputted by the super-resolution module, the first result including feature maps of N spatial frequency bands corresponding to an N-layer Laplacian pyramid of the training image, the second result including a second result feature map corresponding to the training image, and the third result including a target image corresponding to the training image, performing N-layer Laplacian pyramid processing on the supervision image in the matching image pair of the moiré pattern training data set to obtain feature maps of N spatial frequency bands corresponding to the supervision image, obtaining N basic loss function values of the feature maps of the N spatial frequency bands corresponding to the supervision image and the feature maps of the N spatial frequency bands in the first result, obtaining a first loss function value between an image obtained after resolution processing is performed on the supervision image and the second result, and obtaining a second loss function value between the supervision image and the third result, and adjusting model parameters of the initial model according to the N basic loss function values, the first loss function value, and the second loss function value, to obtain the image processing model; obtaining an original image; and running the image processing model to perform moiré pattern removal processing on the original image to obtain a target image, wherein the image processing model includes a multi-band module, the multi-band module being configured to process the original image to obtain an N-layer Laplacian pyramid of the original image, and obtain a first processing result feature map based on feature maps corresponding to N spatial frequency bands of the N-layer Laplacian pyramid, the target image being obtained according to the first processing result feature map, and N being a positive integer greater than or equal to 2. ### Claim 14 (depends on 13) The method according to claim 13, wherein running the image processing model comprises: running the image processing model to perform M times of initial analysis processing on the original image by using the multi-band module, to obtain initial feature maps in N scales, M being a positive integer greater than or equal to 2, M being greater than or equal to N, and the initial analysis processing comprising: down-sampling and performing convolution processing on the original image, or performing down-scaled convolution processing on the original image; performing convolution processing on the initial feature maps in N scales to obtain intermediate feature maps in N scales; obtaining the feature maps corresponding to the N spatial frequency bands of the N-layer Laplacian pyramid according to the intermediate feature maps in N scales; obtaining the first processing result feature map according to the feature maps corresponding to the N spatial frequency bands; and obtaining the target image according to the first processing result feature map. ### Claim 15 (depends on 14) The method according to claim 14, wherein in the feature maps corresponding to the N spatial frequency bands, a feature map of an Nth spatial frequency band is obtained according to an Nth intermediate feature map with a smallest scale in the intermediate feature maps; and in the feature maps corresponding to the N spatial frequency bands, a feature map of an (N-i)th spatial frequency band is obtained according to the Nth intermediate feature map, an (N-i)th intermediate feature map, and all intermediate feature maps between the Nth intermediate feature map and the (N-i)th intermediate feature map, i being a positive integer greater than or equal to 1 and less than N. ### Claim 16 (depends on 14) The method according to claim 14, wherein obtaining the first processing result feature map comprises: obtaining a predicted feature map, the predicted feature map being obtained according to an Nth intermediate feature map, a 1st intermediate feature map, and all intermediate feature maps between the Nth intermediate feature map and the 1st intermediate feature map; performing size adjustment on feature maps whose scales are smaller than a scale of the predicted feature map in the feature maps corresponding to the N spatial frequency bands, so that scales of the adjusted feature maps corresponding to the spatial frequency bands are equal to the scale of the predicted feature map; and obtaining the first processing result feature map according to the predicted feature map and the adjusted feature maps corresponding to the spatial frequency bands. ### Claim 17 (depends on 14) The method according to claim 14, wherein the image processing model further includes a prediction module, and obtaining the target image comprises: obtaining a blurred feature map obtained by blurring the original image; obtaining a second processing result feature map according to the first processing result feature map and the blurred feature map by using the prediction module; and obtaining the target image according to the second processing result feature map. ### Claim 18 (depends on 17) The method according to claim 17, wherein the image processing model further includes a super-resolution module, and obtaining the target image comprises: obtaining a reference feature map according to the first processing result feature map, and obtaining an intermediate result feature map by processing the second processing result feature map, a scale of the reference feature map being the same as a scale of the original image, and a scale of the intermediate result feature map being the same as the scale of the original image; and obtaining the target image according to the reference feature map and the intermediate result feature map by using the super-resolution module.