# US 2022/0044358 A1 · Image processing method and apparatus, device, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0044358 A1 - Document ID: US20220044358A1 - Status: Published application - Kind: A1 - Application number: 17/505,889 - Filed: 2021-10-20 - Published: 2022-02-10 - Earliest claimed: 2021-01-20 - Assignees: Beijing Baidu Netcom Science Technology Co., Ltd. - Applicants: Beijing Baidu Netcom Science Technology Co., Ltd. - Inventors: Jian WANG; Xiang LONG; Hao SUN; Zhiyong JIN; Errui DING - Web page: https://patentgrep.com/patent/US20220044358A1 ## CPC - G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining - G06T 3/4046: using neural networks - G06F 18/253: {of extracted features} - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06V 10/82: using neural networks - G06V 30/18057: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Provided are an image processing method and apparatus, a device, and a storage medium, relating to the technical field of image processing, in particular to the artificial intelligence fields such as computer vision and deep learning. The specific implementation scheme is as follows: inputting a to-be-processed image into an encoding network to obtain a basic image feature, wherein the encoding network includes at least two cascaded overlapping encoding sub-networks which perform encoding and fusion processing on input data at at least two resolutions; and inputting the basic image feature into a decoding network to obtain a target image feature for pixel point classification, wherein the decoding network includes at least one cascaded overlapping decoding sub-network to perform decoding and fusion processing on input data at at least two resolutions respectively. ## Claims ### Claim 1 An image processing method, comprising: inputting a to-be-processed image into an encoding network to obtain a basic image feature, wherein the encoding network comprises at least two cascaded overlapping encoding sub-networks, wherein each of non-head overlapping encoding sub-networks of the at least two overlapping encoding sub-networks performs encoding and fusion processing on input data of the each of non-head overlapping encoding sub-networks at at least two resolutions; and inputting the basic image feature into a decoding network to obtain a target image feature for pixel point classification, wherein the decoding network comprises at least one cascaded overlapping decoding sub-network, wherein each overlapping decoding sub-network of the at least one cascaded overlapping decoding sub-network performs decoding and fusion processing on input data of the each overlapping decoding sub-network at at least two resolutions. ### Claim 2 (depends on 1) The method according to claim 1, wherein input data of a head overlapping encoding sub-network in the encoding network is the to-be-processed image; and input data of the each of non-head overlapping encoding sub-networks in the encoding network is determined according to output data of a previous overlapping encoding sub-network of the each of the non-head overlapping encoding sub-networks, and resolutions of pieces of input data of the non-head overlapping encoding sub-networks sequentially decrease. ### Claim 3 (depends on 2) The method according to claim 2, wherein the each of the non-head overlapping encoding sub-networks in the encoding network comprises at least two encoding convolutional layers, and resolutions of pieces of input data of the at least two encoding convolutional layers are different from each other. ### Claim 4 (depends on 3) The method according to claim 3, wherein for each of the at least two encoding convolutional layers of the each of the non-head overlapping encoding sub-networks in the encoding network, performing, by an encoding convolutional layer, feature extraction on input data of the encoding convolutional layer to obtain an encoding feature; adjusting, by the encoding convolutional layer, encoding features outputted by other encoding convolutional layers to obtain a first adjustment result, wherein the encoding convolutional layer and the other encoding convolutional layers belong to a same overlapping encoding sub-network; and performing, by the encoding convolutional layer, feature fusion on the first adjustment result and the encoding feature to obtain output data of the encoding convolutional layer. ### Claim 5 (depends on 4) The method according to claim 4, wherein the each of the non-head overlapping encoding sub-networks comprises a high-resolution encoding convolutional layer and a low-resolution encoding convolutional layer; wherein the high-resolution encoding convolutional layer comprises a high-resolution encoding feature extraction unit and a high-resolution encoding feature fusion unit; and the low-resolution encoding convolutional layer comprises a low-resolution encoding feature extraction unit and a low-resolution encoding feature fusion unit; the high-resolution encoding feature extraction unit performs feature extraction on input data of the high-resolution encoding feature extraction unit to obtain a high-resolution encoding feature; the low-resolution encoding feature extraction unit performs feature extraction on input data of the low-resolution encoding feature extraction unit to obtain a low-resolution encoding feature; the high-resolution encoding feature fusion unit performs feature fusion on an upsampling result of the low-resolution encoding feature and the high-resolution encoding feature to obtain output data of the high-resolution encoding feature fusion unit, wherein the upsampling result has a same resolution as the high-resolution encoding feature; and the low-resolution encoding feature fusion unit performs feature fusion on a downsampling result of the high-resolution encoding feature and the low-resolution encoding feature to obtain output data of the low-resolution encoding feature fusion unit, wherein the downsampling result has a same resolution as the low-resolution encoding feature. ### Claim 6 (depends on 1) The method according to claim 1, wherein input data of a head overlapping decoding sub-network in the decoding network is determined according to output data of a tail overlapping encoding sub-network in the encoding network; and input data of each of non-head overlapping decoding sub-networks in the decoding network is determined according to output data of a previous overlapping decoding sub-network of each of the non-head overlapping decoding sub-networks, and resolutions of pieces of input data of the non-head overlapping decoding sub-networks sequentially increase. ### Claim 7 (depends on 6) The method according to claim 6, wherein the each of non-tail overlapping decoding sub-networks in the decoding network comprises at least two decoding convolutional layers, and resolutions of pieces of input data of the at least two decoding convolutional layers are different from each other. ### Claim 8 (depends on 7) The method according to claim 7, wherein for each of the at least two decoding convolutional layers of the each of the non-tail overlapping decoding sub-networks in the decoding network, performing, by an decoding convolutional layer, feature reconstruction on input data of the decoding convolutional layer to obtain a decoding feature; and adjusting, by the decoding convolutional layer, decoding features outputted by other decoding convolutional layers to obtain a second adjustment result, wherein the decoding convolutional layer and the other decoding convolutional layers belong to a same overlapping decoding sub-network; and performing, by the decoding convolutional layer, feature fusion on the second adjustment result and the decoding feature to obtain output data of the decoding convolutional layer. ### Claim 9 (depends on 8) The method according to claim 8, wherein the each of the non-tail overlapping decoding sub-networks comprises a high-resolution decoding convolutional layer and a low-resolution decoding convolutional layer; wherein the high-resolution decoding convolutional layer comprises a high-resolution decoding feature reconstruction unit and a high-resolution decoding feature fusion unit; and the low-resolution decoding convolutional layer comprises a low-resolution decoding feature reconstruction unit and a low-resolution decoding feature fusion unit; the high-resolution decoding feature reconstruction unit performs feature reconstruction on input data of the high-resolution decoding feature reconstruction unit to obtain a high-resolution decoding feature; the low-resolution decoding feature reconstruction unit performs feature reconstruction on input data of the low-resolution decoding feature reconstruction unit to obtain a low-resolution decoding feature; the high-resolution decoding feature fusion unit performs feature fusion on an upsampling result of the low-resolution decoding feature and the high-resolution decoding feature to obtain output data of the high-resolution decoding feature fusion unit, wherein the upsampling result has a same resolution as the high-resolution decoding feature; and the low-resolution decoding feature fusion unit performs feature fusion on a downsampling result of the high-resolution decoding feature and the low-resolution decoding feature to obtain output data of the low-resolution decoding feature fusion unit, wherein the downsampling result has a same resolution as the low-resolution decoding feature. ### Claim 10 An image processing apparatus, comprising a processor and a memory for storing execution instructions that when executed by the processor cause the processor to perform steps in following modules: a basic image feature obtaining module, which is configured to input a to-be-processed image into an encoding network to obtain a basic image feature, wherein the encoding network comprises at least two cascaded overlapping encoding sub-networks, wherein each overlapping encoding sub-network of the at least two overlapping encoding sub-networks performs encoding and fusion processing on input data of the each overlapping encoding sub-network at at least two resolutions; and a target image feature obtaining module, which is configured to input the basic image feature into a decoding network to obtain a target image feature for pixel point classification, wherein the decoding network comprises at least one cascaded overlapping decoding sub-network, wherein each overlapping decoding sub-network of the at least one cascaded overlapping decoding sub-network performs decoding and fusion processing on input data of the each overlapping decoding sub-network at at least two resolutions. ### Claim 11 (depends on 10) The apparatus according to claim 10, wherein input data of a head overlapping encoding sub-network in the encoding network is the to-be-processed image; and input data of each of non-head overlapping encoding sub-networks in the encoding network is determined according to output data of a previous overlapping encoding sub-network of the each of the non-head overlapping encoding sub-networks, and resolutions of pieces of input data of the non-head overlapping encoding sub-networks sequentially decrease. ### Claim 12 (depends on 11) The apparatus according to claim 11, wherein the each of the non-head overlapping encoding sub-networks in the encoding network comprises at least two encoding convolutional layers, and resolutions of pieces of input data of the at least two encoding convolutional layers are different from each other. ### Claim 13 (depends on 12) The apparatus according to claim 12, wherein for each of the at least two encoding convolutional layers of the each of the non-head overlapping encoding sub-networks in the encoding network, an encoding convolutional layer performs feature extraction on input data of the encoding convolutional layer to obtain an encoding feature; the encoding convolutional layer adjusts encoding features outputted by other encoding convolutional layers to obtain a first adjustment result, wherein the encoding convolutional layer and the other encoding convolutional layers belong to a same overlapping encoding sub-network; and the encoding convolutional layer performs feature fusion on the first adjustment result and the encoding feature to obtain output data of the encoding convolutional layer. ### Claim 14 (depends on 13) The apparatus according to claim 13, wherein the each of the non-head overlapping encoding sub-networks comprises a high-resolution encoding convolutional layer and a low-resolution encoding convolutional layer; wherein the high-resolution encoding convolutional layer comprises a high-resolution encoding feature extraction unit and a high-resolution encoding feature fusion unit; and the low-resolution encoding convolutional layer comprises a low-resolution encoding feature extraction unit and a low-resolution encoding feature fusion unit; the high-resolution encoding feature extraction unit performs feature extraction on input data of the high-resolution encoding feature extraction unit to obtain a high-resolution encoding feature; the low-resolution encoding feature extraction unit performs feature extraction on input data of the low-resolution encoding feature extraction unit to obtain a low-resolution encoding feature; the high-resolution encoding feature fusion unit performs feature fusion on an upsampling result of the low-resolution encoding feature and the high-resolution encoding feature to obtain output data of the high-resolution encoding feature fusion unit, wherein the upsampling result has a same resolution as the high-resolution encoding feature; and the low-resolution encoding feature fusion unit performs feature fusion on a downsampling result of the high-resolution encoding feature and the low-resolution encoding feature to obtain output data of the low-resolution encoding feature fusion unit, wherein the downsampling result has a same resolution as the low-resolution encoding feature. ### Claim 15 (depends on 10) The apparatus according to claim 10, wherein input data of a head overlapping decoding sub-network in the decoding network is determined according to output data of a tail overlapping encoding sub-network in the encoding network; and input data of each of non-head overlapping decoding sub-networks in the decoding network is determined according to output data of a previous overlapping decoding sub-network of each of the non-head overlapping decoding sub-networks, and resolutions of pieces of input data of the non-head overlapping decoding sub-networks sequentially increase. ### Claim 16 (depends on 15) The apparatus according to claim 15, wherein each of non-tail overlapping decoding sub-networks in the decoding network comprises at least two decoding convolutional layers, and resolutions of pieces of input data of the at least two decoding convolutional layers are different from each other. ### Claim 17 (depends on 16) The apparatus according to claim 16, wherein for each of the at least two decoding convolutional layers of the each of the non-tail overlapping decoding sub-networks in the decoding network, an decoding convolutional layer performs feature reconstruction on input data of the decoding convolutional layer to obtain a decoding feature; and the decoding convolutional layer adjusts decoding features outputted by other decoding convolutional layers to obtain a second adjustment result, wherein the decoding convolutional layer and the other decoding convolutional layers belong to a same overlapping decoding sub-network; and the decoding convolutional layer performs feature fusion on the second adjustment result and the decoding feature to obtain output data of the decoding convolutional layer. ### Claim 18 (depends on 17) The apparatus according to claim 17, wherein the each of the non-tail overlapping decoding sub-networks comprises a high-resolution decoding convolutional layer and a low-resolution decoding convolutional layer; wherein the high-resolution decoding convolutional layer comprises a high-resolution decoding feature reconstruction unit and a high-resolution decoding feature fusion unit; and the low-resolution decoding convolutional layer comprises a low-resolution decoding feature reconstruction unit and a low-resolution decoding feature fusion unit; the high-resolution decoding feature reconstruction unit performs feature reconstruction on input data of the high-resolution decoding feature reconstruction unit to obtain a high-resolution decoding feature; the low-resolution decoding feature reconstruction unit performs feature reconstruction on input data of the low-resolution decoding feature reconstruction unit to obtain a low-resolution decoding feature; the high-resolution decoding feature fusion unit performs feature fusion on an upsampling result of the low-resolution decoding feature and the high-resolution decoding feature to obtain output data of the high-resolution decoding feature fusion unit, wherein the upsampling result has a same resolution as the high-resolution decoding feature; and the low-resolution decoding feature fusion unit performs feature fusion on a downsampling result of the high-resolution decoding feature and the low-resolution decoding feature to obtain output data of the low-resolution decoding feature fusion unit, wherein the downsampling result has a same resolution as the low-resolution decoding feature. ### Claim 19 (depends on 1) A non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are configured to cause a computer to perform the image processing method according to claim 1.