# US 2014/0193092 A1 · Superresolution image processing using an invertible sparse matrix > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0193092 A1 - Document ID: US20140193092A1 - Status: Published application - Kind: A1 - Application number: 13/950,384 - Filed: 2013-07-25 - Published: 2014-07-10 - Earliest claimed: 2013-01-09 - Assignees: LSI Corporation - Applicants: LSI Corporation - Inventors: Alexander Alexandrovich Petyushko; Dmitry Nikolaevich Babin; Ivan Leonidovich Mazurenko; Alexander Borisovich Kholodenko - Web page: https://patentgrep.com/patent/US20140193092A1 ## CPC - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Superresolution image processing that can be applied when two image frames of the same scene are available so that image information from one frame can be used to enhance the image from the other frame. The superresolution image processing uses a sparse matrix generated based on a Markov random field defined over these two image frames. The sparse matrix is inverted and applied to the image data from the image frame that is being enhanced to generate a corresponding enhanced image. ## Claims ### Claim 1 A machine-implemented method of image processing comprising (A) applying superresolution processing to a first image frame having a first spatial resolution to generate a second image frame having a second spatial resolution that is higher than the first spatial resolution, wherein step (A) comprises: (A1) generating a sparse matrix based on a Markov random field defined over the first image frame and a third image frame having a third spatial resolution that is higher than the first spatial resolution; (A2) generating an inverse matrix by inverting the sparse matrix; and (A3) generating the second image frame using the inverse matrix, wherein: the first image frame covers a first scene; and each of the second and third image frames covers the first scene. ### Claim 2 (depends on 1) The method of claim 1, wherein the second spatial resolution is lower than the third spatial resolution. ### Claim 3 (depends on 1) The method of claim 1, wherein: the third image frame represents a rectangular array of M×N pixels; and the sparse matrix is a square matrix having MN rows and MN columns, with no more than 5MN non-zero matrix elements. ### Claim 4 (depends on 3) The method of claim 3, wherein the no more than 5MN non-zero matrix elements are located on a set of diagonals consisting of no more than five diagonals of the sparse matrix. ### Claim 5 (depends on 4) The method of claim 4, wherein the set of diagonals includes a main diagonal and two shorter diagonals immediately adjacent to the main diagonal, one on either side thereof. ### Claim 6 (depends on 5) The method of claim 5, wherein the set of diagonals further includes two additional shorter diagonals that are offset from the main diagonal by M or N matrix elements. ### Claim 7 (depends on 3) The method of claim 3, wherein the second image frame represents a rectangular array of M×N pixels. ### Claim 8 (depends on 3) The method of claim 3, wherein: step (A) further comprises (A4) generating a vector based on the first image frame, said vector having MN elements; and step (A3) comprises: generating a string of MN pixel values by calculating a product of the inverse matrix and the vector; slicing the string of pixel values into M substrings, each having N pixel values; and generating the second image frame by arranging the M substrings as rows or columns of a two-dimensional array of pixels. ### Claim 9 (depends on 1) The method of claim 1, further comprising: (B) generating the first image frame using a first image sensor; and (C) generating the third image using a second image sensor different from the first image sensor. ### Claim 10 (depends on 1) The method of claim 1, further comprising: (B) partitioning a fourth image frame into a first plurality of sub-frames, wherein the first image frame is a sub-frame of the first plurality; and (C) partitioning a fifth image frame into a second plurality of sub-frames, wherein the third image frame is a sub-frame of the second plurality. ### Claim 11 (depends on 10) The method of claim 10, wherein: the fourth image frame covers a second scene, wherein the first scene is a sub-scene of the second scene; and the fifth image frame covers the second scene. ### Claim 12 (depends on 11) The method of claim 11, further comprising: (D) applying superresolution processing to another sub-frame of the first plurality to generate a sixth image frame covering another sub-scene of the second scene, said another sub-scene being different from the first scene; and (E) combining the second image frame and the sixth image frame to generate a combined image frame corresponding to the second scene. ### Claim 13 (depends on 12) The method of claim 12, wherein step (E) comprises arranging the second image frame and the sixth image frame as tiles of the combined image frame. ### Claim 14 (depends on 12) The method of claim 12, wherein step (D) comprises: (D1) generating a second sparse matrix based on a Markov random field defined over said another sub-frame of the first plurality and a sub-frame of the second plurality corresponding to said another sub-frame of the first plurality; (D2) generating a second inverse matrix by inverting the second sparse matrix; and (D3) generating the sixth image frame using the second inverse matrix. ### Claim 15 (depends on 12) The method of claim 12, wherein step (D) comprises applying a conjugate-gradient algorithm to said another sub-frame of the first plurality to generate the sixth image frame. ### Claim 16 (depends on 1) The method of claim 1, wherein: the first image frame is a depth map of the first scene captured by a range sensor; the second image frame is an upsampled depth map of the first scene; and the third image frame is a photograph of the first scene captured by a luminosity sensor. ### Claim 17 (depends on 1) The method of claim 1, further comprising (B) performing at least one of image registration and image trimming to generate the first image frame and the third image frame. ### Claim 18 A non-transitory machine-readable medium, having encoded thereon program code, wherein, when the program code is executed by a machine, the machine implements a method of image processing, the method comprising: (A) applying superresolution processing to a first image frame having a first spatial resolution to generate a second image frame having a second spatial resolution that is higher than the first spatial resolution, wherein step (A) comprises: (A1) generating a sparse matrix based on a Markov random field defined over the first image frame and a third image frame having a third spatial resolution that is higher than the first spatial resolution; (A2) generating an inverse matrix by inverting the sparse matrix; and (A3) generating the second image frame using the inverse matrix, wherein: the first image frame covers a first scene; and each of the second and third image frames covers the first scene. ### Claim 19 An apparatus comprising: a memory configured to store (i) a first image frame covering a first scene, said first image frame having a first spatial resolution, and (ii) a second image frame covering the first scene, said second image frame having a second spatial resolution that is higher than the first spatial resolution; and a processor configured to apply superresolution processing to the first image frame to generate a third image frame covering the first scene, said third image frame having a third spatial resolution that is higher than the first spatial resolution, wherein the processor is further configured to: generate a sparse matrix based on a Markov random field defined over the first image frame and the second image frame; generate an inverse matrix by inverting the sparse matrix; and generate the third image frame using the inverse matrix. ### Claim 20 (depends on 19) The apparatus of claim 19, further comprising: a range sensor configured to generate image data for the first image frame; and a luminosity sensor configured to generate image data for the second image frame.