# US 12,182,971 B2 · Image processing method and apparatus, storage medium, and electronic device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,971 B2 - Document ID: US12182971B2 - Status: Granted - Kind: B2 - Application number: 17/524,387 - Filed: 2021-11-11 - Published: 2022-03-03 - Granted: 2024-12-31 - Earliest claimed: 2019-11-26 - Assignees: Tencent Technology (Shenzhen) Company Limited - Applicants: Tencent Technology (Shenzhen) Company Limited - Inventors: Xiaoming Yu; Yang Yi; Feng Li; Kaijie Cai - Web page: https://patentgrep.com/patent/US12182971B2 ## CPC - G06T 5/70: Denoising; Smoothing - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/82: using neural networks - G06V 40/16: Human faces, e.g. facial parts, sketches or expressions - G06V 40/161: {Detection; Localisation; Normalisation} - G06T 2207/20012: Locally adaptive - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A plurality of groups of portrait components and a region in which the plurality of groups of portrait components are located are recognized from a first image, each group of portrait components corresponding to one human body, and a target region that includes a human face is determined in the region in which the plurality of groups of portrait components are located, to blur regions other than the target region in the first target image. A target group of portrait components including the human face is recognized from the first image, so that the target region in which the target group of portrait components is located is determined as a foreground region, and limbs of other people without a human face are determined as a background region. The disclosed system and method improve the accuracy of recognizing a foreground person and reducing incorrect detection in portrait recognition. ## Claims ### Claim 1 An image processing method comprising: recognizing, from a first image, a plurality of groups of portrait components and a region in which the plurality of groups of portrait components are located, each group of portrait components corresponding to one human body; determining, by processing circuitry of an electronic device, in the recognized region in which the plurality of groups of portrait components are located, which of the plurality of groups of portrait components includes a respective human face; determining a target region in the recognized region that is separated from other regions in the recognized region based on which of the plurality of groups of portrait components is determined to include the respective human face, the target region including each of the plurality of groups of portrait components that includes the respective human face; and blurring regions of the first image other than the target region, to obtain a second image. ### Claim 2 (depends on 1) The method according to claim 1, wherein the determining the target region comprises: determining M groups of portrait components comprising a human face from N groups of portrait components, the N groups of portrait components being the plurality of groups of portrait components, N≥M≥1; and determining, in a region in which the M groups of portrait components are located, the target region in which the respective human face of each of the plurality of groups of portrait components including the respective human face is located is larger than or equal to a first threshold, or the target region in which each of the plurality of groups of portrait components including the respective human face is located being larger than or equal to a second threshold. ### Claim 3 (depends on 2) The method according to claim 2, wherein the first threshold and the second threshold are positively correlated to a size of the first image. ### Claim 4 (depends on 2) The method according to claim 2, wherein the determining, in the region in which the M groups of portrait components are located, comprises: setting pixel values of pixels corresponding to the M groups of portrait components to a first pixel value, and setting pixel values of pixels in the first image other than the pixels corresponding to the M groups of portrait components to a second pixel value, to obtain a binary image, the first pixel value being different from the second pixel value; and performing a region recognition on the binary image, to obtain the target region, the target region comprising pixels of the respective human face of each of the plurality of groups of portrait components determined to include the respective human face. ### Claim 5 (depends on 1) The method according to claim 1, wherein the recognizing comprises: processing the first image by using a recognition model, and determining the plurality of groups of portrait components and the recognized region in which the plurality of groups of portrait components are located. ### Claim 6 (depends on 1) The method according to claim 1, wherein, before the recognizing, the method further comprises: acquiring a first group of training images, a second group of training images, a group of region division results, and a group of training recognition results, the first group of training images corresponding one to one to the group of region division results, each region division result representing a known portrait region in an image in the first group of training images, the second group of training images corresponding one to one to the group of training recognition results, each training recognition result representing a known portrait component in an image in the second group of training images; and training an initial recognition model based on the first group of training images and the second group of training images, to obtain a trained recognition model, an error between an estimated portrait region recognized from the first group of training images by using the trained recognition model and the known portrait region in the group of region division results satisfying a first convergence condition, an error between an estimated portrait component recognized from the second group of training images by using the trained recognition model and the known portrait component in the group of training recognition results satisfying a second convergence condition, the trained recognition model comprising: an encoding network configured to encode an image to obtain encoded data, a portrait region recognition network configured to recognize a portrait region according to the encoded data, and a portrait component recognition network configured to recognize a portrait component according to the encoded data. ### Claim 7 (depends on 6) The method according to claim 6, wherein the training comprises: selecting a first training image from the first group of training images, and selecting a second training image from the second group of training images; inputting the first training image and the second training image into the initial recognition model, the initial recognition model comprising an initial encoding network, an initial portrait region recognition network, and an initial portrait component recognition network, the initial encoding network comprising cascaded first convolutional layers, the initial portrait region recognition network comprising cascaded second convolutional layers, the initial portrait component recognition network comprising cascaded third convolutional layers; and receiving, by a first convolutional layer in the initial encoding network, encoded data obtained after a cascaded previous first convolutional layer encodes the first training image and the second training image, and transmitting the encoded data to a corresponding second convolutional layer, third convolutional layer, and cascaded next first convolutional layer; receiving, by the initial portrait region recognition network, encoded data transmitted by a corresponding first convolutional layer and cascaded previous second convolutional layer, and performing a portrait region recognition on the received encoded data; and receiving, by the initial portrait component recognition network, encoded data transmitted by a corresponding first convolutional layer and cascaded previous third convolutional layer, and performing a portrait component recognition on the received encoded data. ### Claim 8 (depends on 1) The method according to claim 1, wherein, in response to a determination that the first image is a video frame image in a video, after the blurring the regions other than the target region in the first image, the method further comprises: replacing the first image in the video with the second image; and playing the second image in a process of playing the video. ### Claim 9 An image processing apparatus, comprising: processing circuitry configured to recognize, from a first image, a plurality of groups of portrait components and a region in which the plurality of groups of portrait components are located, each group of portrait components corresponding to one human body; determine, in the recognized region in which the plurality of groups of portrait components are located, which of the plurality of groups of portrait components includes a respective human face; determine a target region in the recognized region that is separated from other regions in the recognized region based on which of the plurality of groups of portrait components is determined to include the respective human face, the target region including each of the plurality of groups of portrait components that includes the respective human face; and blur regions of the first image other than the target region, to obtain a second image. ### Claim 10 (depends on 9) The apparatus according to claim 9, wherein the processing circuitry performs the determining by: determining M groups of portrait components from N groups of portrait components, the N groups of portrait components being the plurality of groups of portrait components, N≥M≥1; and determining, in a region in which the M groups of portrait components are located, the target region in which the respective human face of each of the plurality of groups of portrait components including the respective human face is located is larger than or equal to a first threshold, or, the target region in which the plurality of groups of portrait components including the respective human face is located being larger than or equal to a second threshold. ### Claim 11 (depends on 10) The apparatus according to claim 10, wherein the first threshold and the second threshold is positively correlated to a size of the first image. ### Claim 12 (depends on 10) The apparatus according to claim 10, wherein the processing circuitry is configured to determine the target region by: setting pixel values of pixels corresponding to the M groups of portrait components to a first pixel value, and setting pixel values of pixels in the first image other than the pixels corresponding to the M groups of portrait components to a second pixel value, to obtain a binary image, the first pixel value being different from the second pixel value; and performing a region recognition on the binary image, to obtain the target region, the target region comprising pixels of the respective human face of each of the plurality of groups of portrait components determined to include the respective human face. ### Claim 13 (depends on 9) The apparatus according to claim 9, wherein the processing circuitry is configured to recognize the plurality of groups of portrait components by: processing the first image by using a recognition model, and determining the plurality of groups of portrait components and the recognized region in which the plurality of groups of portrait components are located. ### Claim 14 (depends on 9) The apparatus according to claim 9, wherein, before recognizing the plurality of groups of portrait components, the processing circuitry is further configured to: acquire a first group of training images, a second group of training images, a group of region division results, and a group of training recognition results, the first group of training images corresponding one to one to the group of region division results, each region division result representing a known portrait region in an image in the first group of training images, the second group of training images corresponding one to one to the group of training recognition results, each training recognition result representing a known portrait component in an image in the second group of training images; and train an initial recognition model based on the first group of training images and the second group of training images, to obtain a trained recognition model, an error between an estimated portrait region recognized from the first group of training images by using the trained recognition model and the known portrait region in the group of region division results satisfying a first convergence condition, an error between an estimated portrait component recognized from the second group of training images by using the trained recognition model and the known portrait component in the group of training recognition results satisfying a second convergence condition, the trained recognition model comprising: an encoding network configured to encode an image to obtain encoded data, a portrait region recognition network configured to recognize a portrait region according to the encoded data, and a portrait component recognition network configured to recognize a portrait component according to the encoded data. ### Claim 15 (depends on 14) The apparatus according to claim 14, wherein the processing circuitry is configured to train the initial recognition model by: selecting a first training image from the first group of training images, and selecting a second training image from the second group of training images; inputting the first training image and the second training image into the initial recognition model, the initial recognition model comprising an initial encoding network, an initial portrait region recognition network, and an initial portrait component recognition network, the initial encoding network comprising cascaded first convolutional layers, the initial portrait region recognition network comprising cascaded second convolutional layers, the initial portrait component recognition network comprising cascaded third convolutional layers; and receiving, by a first convolutional layer in the initial encoding network, encoded data obtained after a cascaded previous first convolutional layer encodes the first training image and the second training image, and transmitting the encoded data to a corresponding second convolutional layer, third convolutional layer, and cascaded next first convolutional layer; receiving, by the initial portrait region recognition network, encoded data transmitted by a corresponding first convolutional layer and cascaded previous second convolutional layer, and performing a portrait region recognition on the received encoded data; and receiving, by the initial portrait component recognition network, encoded data transmitted by a corresponding first convolutional layer and cascaded previous third convolutional layer, and performing a portrait component recognition on the received encoded data. ### Claim 16 (depends on 9) The apparatus according to claim 9, wherein, in response to a determination that the first image is a video frame image in a video and after blurring the regions other than the target region in the first image, the processing circuitry is further configured to: replace the first image in the video with the second image; and play the second image in a process of playing the video. ### Claim 17 A non-transitory computer-readable storage medium storing computer-readable instructions thereon, which, when executed by a processor, cause the processor to perform an image processing method comprising: recognizing, from a first image, a plurality of groups of portrait components and a region in which the plurality of groups of portrait components are located, each group of portrait components corresponding to one human body; determining, by processing circuitry of an electronic device, in the recognized region in which the plurality of groups of portrait components are located, which of the plurality of groups of portrait components includes a respective human face; determining a target region in the recognized region that is separated from other regions in the recognized region based on which of the plurality of groups of portrait components is determined to include the respective human face, the target region including each of the plurality of groups of portrait components that includes the respective human face; and blurring regions of the first image other than the target region, to obtain a second image. ### Claim 18 (depends on 17) The non-transitory computer-readable storage medium according to claim 17, wherein the determining comprises: determining M groups of portrait components from N groups of portrait components, the N groups of portrait components being the plurality of groups of portrait components, N≥M≥1; and determining, in a region in which the M groups of portrait components are located, the target region in which the respective human face of each of the plurality of groups of portrait components including the respective human face is located is larger than or equal to a first threshold, or the target region in which the plurality of groups of portrait components including the respective human face is located being larger than or equal to a second threshold. ### Claim 19 (depends on 18) The non-transitory computer-readable storage medium according to claim 18, wherein the first threshold and the second threshold are positively correlated to a size of the first image. ### Claim 20 (depends on 18) The non-transitory computer-readable storage medium according to claim 18, wherein the determining, in the region in which the M groups of portrait components are located, comprises: setting pixel values of pixels corresponding to the M groups of portrait components to a first pixel value, and setting pixel values of pixels in the first image other than the pixels corresponding to the M groups of portrait components to a second pixel value, to obtain a binary image, the first pixel value being different from the second pixel value; and performing a region recognition on the binary image, to obtain the target region, the target region comprising pixels of the respective human face of each of the plurality of groups of portrait components determined to include the respective human face.