# US 10,373,292 B2 · Method and system for providing high resolution image through super-resolution reconstruction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,373,292 B2 - Document ID: US10373292B2 - Status: Granted - Kind: B2 - Application number: 15/805,197 - Filed: 2017-11-07 - Published: 2018-05-17 - Granted: 2019-08-06 - Earliest claimed: 2016-11-07 - Assignees: Umbo Cv Inc. - Applicants: Umbo Cv Inc. - Inventors: Pai-Heng Hsiao; Ping-Lin Chang - Web page: https://patentgrep.com/patent/US10373292B2 ## CPC - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - G06T 7/13: Edge detection - G06T 7/90: Determination of colour characteristics - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - G06T 2207/10024: Color image - G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform - G06T 2207/20076: Probabilistic image processing - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system and method for providing high resolution image through super-resolution reconstruction. The system comprises a frame buffer, a convolutional neural network module, an image processing module, and a bi-directional prediction module. After the frame buffer has received high-resolution image with low frames per second and low-resolution image with high frames per second, both are captured by an image capturing module, image processing with super-resolution reconstruction can be performed through the cooperation of the convolutional neural network module, the image processing module and the bi-directional prediction module. In other words, a frame with super-resolution can be calculated through a well-trained convolutional neural network. ## Claims ### Claim 1 An implementation method for providing high resolution image through super-resolution reconstruction, comprising the following steps: Capturing image step: capturing a plurality of high-resolution frames and a plurality of low-resolution frames by an image capturing device; Generating candidate pixel set and selecting pixel set step: a convolutional neural network module performs multiple convolution operations on said high-resolution frame so as to calculate out a plurality of candidate high-resolution pixel sets; thereafter, at least one said convolution operation is conducted simultaneously on each of said candidate high-resolution pixel sets so as to generate a selected high-resolution pixel set; Generating post-processing residual value step: said convolutional neural network module performs at least one said convolution operation on said selected high-resolution pixel set and said low-resolution frame, so as to generate a post-processing residual value; Generating prediction high-resolution frame step: a pixel addition unit performs a pixel addition operation on said post-processing residual value and said low-resolution frame so as to calculate out a forward prediction high-resolution frame and a reverse prediction high-resolution frame of the same time point; and Generating super-resolution frame step: after a bi-directional prediction module receives said forward prediction high-resolution frame and said reverse prediction high-resolution frame, said convolutional neural network module performs at least one said convolution operation on said forward prediction high-resolution frame and said reverse prediction high-resolution frame, so as to generate a super-resolution frame. ### Claim 2 (depends on 1) The implementation method for providing high resolution image through super-resolution reconstruction as claimed in claim 1, after one said high-resolution frame and one said low-resolution frame of next time point are performed with said generating candidate pixel set and selecting pixel set step, said generating post-processing residual value step, and said generating prediction high-resolution frame step sequentially, said forward prediction high-resolution frame is generated thereon. ### Claim 3 (depends on 1) The implementation method for providing high resolution image through super-resolution reconstruction as claimed in claim 1, after one said high-resolution frame and one said low-resolution frame of last time point are performed with said generating candidate pixel set and selecting pixel set step, said generating post-processing residual value step, and said generating prediction high-resolution frame step sequentially, said reverse prediction high-resolution frame is generated thereon. ### Claim 4 (depends on 1) The implementation method for providing high resolution image through super-resolution reconstruction as claimed in claim 1, after said convolutional neural network module performs at least one said convolution operation on each of said candidate high-resolution pixel sets, a displacement calculation unit firstly from the operation result of said convolution operation filters out the pixel set having a smaller pixel displacement quantity with said low-resolution frame to serve as said selected high-resolution pixel set. ### Claim 5 (depends on 1) The implementation method for providing high resolution image through super-resolution reconstruction as claimed in claim 1, wherein the frames per second of a plurality of said high-resolution frames captured by said image capturing device is lower than or equal to that of said low-resolution frames. ### Claim 6 (depends on 1) The implementation method for providing high resolution image through super-resolution reconstruction as claimed in claim 1, wherein said image capturing device determines the quantity of continuously captured frame by the number of samples, and at least one high-resolution frame is present in the captured frames. ### Claim 7 (depends on 1) The implementation method for providing high resolution image through super-resolution reconstruction as claimed in claim 1, wherein said convolutional neural network module has an input layer, a hidden layer and an output layer, said input layer is employed to input the frame or the residual value to be conducted with convolution operation, said hidden layer is employed to store a plurality of parameters for determining which convolution kernel to be used in said convolution operation, said output layer is used to output the operation result of said convolution operation. ### Claim 8 (depends on 7) The implementation method for providing high resolution image through super-resolution reconstruction as claimed in claim 7, wherein said plurality of parameters determine said different convolution kernel which said input layer can use to perform multiple said convolution operation on said high-resolution frame separately, so as to calculate out said plurality of candidate high-resolution pixel sets, and said input layer can employ again said convolution kernel with 1×1 pixel to perform one said convolution operation on said each of said candidate high-resolution pixel sets simultaneously, so as to calculate out said selected high-resolution pixel set. ### Claim 9 (depends on 7) The implementation method for providing high resolution image through super-resolution reconstruction as claimed in claim 7, wherein said plurality of parameters determine said convolution kernel which said input layer can use to perform one said convolution operation on said selected high-resolution pixel set and said low-resolution frame simultaneously, so as to calculate out an image fusion residual value. ### Claim 10 (depends on 9) The implementation method for providing high resolution image through super-resolution reconstruction as claimed in claim 9, wherein said plurality of parameters are in such a manner that said input layer can perform one said convolution operation on said image fusion residual value so as to generate a first post-processing residual value; thereafter, said input layer can perform multiple said convolution operation sequentially on said first post-processing residual value, and the last resulting post-processing residual value is served as said post-processing residual value. ### Claim 11 (depends on 7) The implementation method for providing high resolution image through super-resolution reconstruction as claimed in claim 7, wherein said plurality of parameters are in such a manner that said input layer can perform at least one said convolution operation on said forward prediction high-resolution frame and said reverse prediction high-resolution frame so as to calculate out said super-resolution frame respectively corresponding to R, G, B color frequency signals, or calculate out said super-resolution frame respectively corresponding to C, M, Y, K color frequency signals, or calculate out said super-resolution frame respectively corresponding to Y, U, V color frequency signals. ### Claim 12 (depends on 1) The implementation method for providing high resolution image through super-resolution reconstruction as claimed in claim 1, wherein a memory module is separately in data linking with a frame buffer and a streaming buffer so as to store each of said decoded high-resolution frames and each of said decoded low-resolution frames, and store un-decoded plural packets, each of said packets is decoded to generate each of said high-resolution frames and each of said low-resolution frames. ### Claim 13 A system for providing high resolution image through super-resolution reconstruction, which can be in data linking with an image capturing device to obtain a plurality of high-resolution frames and a plurality of low-resolution frames captured by said image capturing device so as to allow viewing a monitored image through an user terminal device after data linking, said system comprising: a frame buffer for receiving, temporarily storing one said high-resolution frame, one said low-resolution frame and a super-resolution frame; a convolutional neural network module in data linking with said frame buffer to perform multiple convolution operations on said high-resolution frame, so as to generate a plurality of candidate high-resolution pixel sets; and to perform at least one said convolution operation on said plurality of candidate high-resolution pixel sets so as to generate a selected high-resolution pixel set; and to perform at least one said convolution operation on said selected high-resolution pixel set and said low-resolution frame simultaneously so as to generate a post-processing residual value; an image processing module in data linking with said convolutional neural network module, said image processing module has an image addition unit for performing image addition operation on said post-processing residual value and said low-resolution frame so as to generate a forward prediction high-resolution frame and a reverse prediction high-resolution frame of the same time point; a bi-directional prediction module in data linking with said convolutional neural network module for receiving said forward prediction high-resolution frame and said reverse prediction high-resolution frame, and then transmitting these frames to said convolutional neural network module to perform said convolution operation so as to generate said super-resolution frame. ### Claim 14 (depends on 13) The system for providing high resolution image through super-resolution reconstruction as claimed in claim 13, after one said high-resolution frame and one said low-resolution frame of next time point are performed with multiple said convolution operation by said convolutional neural network module and said pixel addition operation by said image processing module sequentially, said forward prediction high-resolution frame is generated thereon. ### Claim 15 (depends on 13) The system for providing high resolution image through super-resolution reconstruction as claimed in claim 13, after one said high-resolution frame and one said low-resolution frame of last time point are performed with multiple said convolution operation by said convolutional neural network module and said pixel addition operation by said image processing module sequentially, said reverse prediction high-resolution frame is generated thereon. ### Claim 16 (depends on 13) The system for providing high resolution image through super-resolution reconstruction as claimed in claim 13, wherein said image processing module has a displacement calculation unit, with respect to the result obtained from said convolution operation conducted on said plurality of high-resolution pixel set, said displacement calculation unit firstly conducts a pixel displacement quantity calculation and then selects out the pixel set having a smaller pixel displacement quantity with said low-resolution frame to serve as said selected high-resolution pixel set. ### Claim 17 (depends on 13) The system for providing high resolution image through super-resolution reconstruction as claimed in claim 13, wherein the frames per second of a plurality of said high-resolution frames captured by said image capturing device is lower than or equal to that of said low-resolution frames. ### Claim 18 (depends on 13) The system for providing high resolution image through super-resolution reconstruction as claimed in claim 13, wherein said image capturing device determines the quantity of continuously captured frame with number of samples, and at least one high-resolution frame is present in the captured frames. ### Claim 19 (depends on 13) The system for providing high resolution image through super-resolution reconstruction as claimed in claim 13, wherein said convolutional neural network module has an input layer, a hidden layer and an output layer, said input layer is employed to input the frame or the residual value to be conducted with convolution operation, said hidden layer is employed to store a plurality of parameters for determining which convolution kernel to be used in said convolution operation, said output layer is used to output the operation result of said convolution operation. ### Claim 20 (depends on 19) The system for providing high resolution image through super-resolution reconstruction as claimed in claim 19, wherein said plurality of parameters determine said different convolution kernel which said input layer can use to perform multiple said convolution operation on said high-resolution frame separately, so as to calculate out said plurality of candidate high-resolution pixel sets, and said input layer can employ again said convolution kernel with 1×1 pixel to perform one said convolution operation on each of said candidate high-resolution pixel sets simultaneously, so as to calculate out said selected high-resolution pixel set. ### Claim 21 (depends on 19) The system for providing high resolution image through super-resolution reconstruction as claimed in claim 19, wherein said plurality of parameters are in such a manner that said input layer can perform one said convolution operation on said image fusion residual value so as to generate a first post-processing residual value; thereafter, said input layer can perform multiple said convolution operations sequentially on said first post-processing residual value, and the last resulting post-processing residual value is served as said post-processing residual value. ### Claim 22 (depends on 19) The system for providing high resolution image through super-resolution reconstruction as claimed in claim 19, wherein said plurality of parameters are in such a manner that said input layer can perform at least one said convolution operation on said forward prediction high-resolution frame and said reverse prediction high-resolution frame so as to calculate out said super-resolution frame respectively corresponding to R, G; B color frequency signals, or calculate out said super-resolution frame respectively corresponding to C, M, Y, K color frequency signals, or calculate out said super-resolution frame respectively corresponding to Y, U, V color frequency signals. ### Claim 23 (depends on 13) The system for providing high resolution image through super-resolution reconstruction as claimed in claim 13, wherein a memory module is separately in data linking with a frame buffer and a streaming buffer so as to store each of said decoded high-resolution frames and each of said decoded low-resolution frames, and store un-decoded plural packets, each of said packets is decoded to generate each of said high-resolution frames and each of said low-resolution frames.