# US 2014/0153819 A1 · Learned Piece-Wise Patch Regression for Image Enhancement

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0153819 A1
- Document ID: US20140153819A1
- Status: Published application
- Kind: A1
- Application number: 13/691,190
- Filed: 2012-11-30
- Published: 2014-06-05
- Earliest claimed: 2012-11-30
- Assignees: Adobe Systems Incorporated
- Applicants: Adobe Systems Incorporated
- Inventors: Zhe Lin; Xin Lu; Jonathan Brandt; Hailin Jin
- Web page: https://patentgrep.com/patent/US20140153819A1

## CPC

- G06T 5/70: Denoising; Smoothing
- G06N 20/00: Machine learning
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods are provided for providing learned, piece-wise patch regression for image enhancement. In one embodiment, an image manipulation application generates training patch pairs that include training input patches and training output patches. Each training patch pair includes a respective training input patch from a training input image and a respective training output patch from a training output image. The training input image and the training output image include at least some of the same image content. The image manipulation application determines patch-pair functions from at least some of the training patch pairs. Each patch-pair function corresponds to a modification to a respective training input patch to generate a respective training output patch. The image manipulation application receives an input image generates an output image from the input image by applying at least some of the patch-pair functions based on at least some input patches of the input image.

## Claims

### Claim 1

A method comprising:
generating, by an image manipulation application executed by a processing device, a plurality of training patch pairs comprising a plurality of training input patches and a plurality of training output patches, wherein each training patch pair comprises a respective training input patch from a training input image and a respective training output patch from a training output image, wherein the training input image and the training output image include at least some of the same image content;
determining, by the image manipulation application, a plurality of patch-pair functions from at least some of the plurality of training patch pairs, wherein each patch-pair function of the plurality of patch-pair functions corresponds to a modification to a respective training input patch to generate a respective training output patch;
receiving, by the image manipulation application, an input image; and
generating, by the image manipulation application, an output image from the input image by applying at least some of the plurality of patch-pair functions based on at least some of a plurality of input patches of the input image.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the plurality of patch-pair functions comprises:
partitioning the plurality of training input patches into a plurality of groups of training input patches;
determining a plurality of anchor points, wherein each anchor point represents a respective one of the plurality of groups of training input patches; and
determining a respective patch-pair function for each of the plurality of anchor points.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining the plurality of patch-pair functions from at least some of the plurality of training patch pairs comprises applying a least squares algorithm to determine a relationship between a respective one of a respective group of training input patches represented by a respective anchor point and a corresponding training output patch.

### Claim 4 (depends on 2)

The method of claim 2, wherein determining the plurality of patch-pair functions from at least some of the plurality of training patch pairs comprises applying a non-negative least squares algorithm to determine a relationship between a respective one of a respective group of training input patches represented by a respective anchor point and a corresponding training output patch.

### Claim 5 (depends on 2)

The method of claim 2, wherein applying the corresponding one of the plurality of patch-pair functions comprises, for each input patch of the plurality of input patches:
determining a respective anchor point corresponding to the input patch; and
applying, to the input patch, the respective patch-pair function corresponding to the respective anchor point.

### Claim 6 (depends on 2)

The method of claim 2, wherein applying the at least some of the plurality of patch-pair functions based on the at least some of the plurality of input patches of the input image comprises, for each input patch of the at least some of a plurality of input patches:
determining a distance between the input patch and a respective anchor point corresponding to the input patch;
determining whether the distance exceeds a threshold distance; and
performing at least one of:
applying the respective patch-pair function to the respective anchor point based on determining that the distance does not exceed the threshold distance; and
applying the respective patch-pair function to the input patch based on determining that the distance exceeds the threshold distance.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining the plurality of patch-pair functions comprises:
generating a hierarchy of the plurality of training input patches representing a plurality of groups of training input patches; and
determining a plurality of anchor points selecting a plurality of leaf nodes from the hierarchy, wherein each anchor point represents a respective one of the plurality of groups of training input patches; and
determining a respective patch-pair function for each of the plurality of anchor points.

### Claim 8 (depends on 7)

The method of claim 7, wherein generating the hierarchy comprises, for each level of a plurality of levels in the hierarchy specified by a depth factor for the hierarchy, determining a plurality of centroids, each centroid representative of a respective subset of the plurality of training input patches closest to the centroid, wherein the number of centroids for each level is specified by a branch factor for the hierarchy.

### Claim 9 (depends on 1)

The method of claim 1, wherein each training input patch is represented by a multi-dimensional vector.

### Claim 10

A non-transitory computer-readable medium embodying program code executable by a processing device, the non-transitory computer-readable medium comprising:
program code for generating a plurality of training patch pairs comprising a plurality of training input patches and a plurality of training output patches, wherein each training patch pair comprises a respective training input patch from a training input image and a respective training output patch from a training output image, wherein the training input image and the training output image include at least some of the same image content;
program code for determining a plurality of patch-pair functions from at least some of the plurality of training patch pairs, wherein each of the plurality of patch-pair functions corresponds to a modification to a respective training input patch to generate a respective training output patch;
program code for receiving an input image; and
program code for generating an output image from the input image by applying at least some of the plurality of patch-pair functions based on at least some of a plurality of input patches of the input image.

### Claim 11 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein determining the plurality of patch-pair functions comprises:
partitioning the plurality of training input patches into a plurality of groups of training input patches;
determining a plurality of anchor points, wherein each anchor point represents a respective one of the plurality of groups of training input patches; and
determining a respective patch-pair function for each of the plurality of anchor points.

### Claim 12 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein determining the plurality of patch-pair functions from at least some of the plurality of training patch pairs comprises applying a least squares algorithm to determine a relationship between a respective one of a respective group of training input patches represented by a respective anchor point and a corresponding training output patch.

### Claim 13 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein determining the plurality of patch-pair functions from at least some of the plurality of training patch pairs comprises applying a non-negative least squares algorithm to determine a relationship between a respective one of a respective group of training input patches represented by a respective anchor point and a corresponding training output patch.

### Claim 14 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein applying the corresponding one of the plurality of patch-pair functions comprises, for each input patch of the plurality of input patches:
determining a respective anchor point corresponding to the input patch; and
applying, to the input patch, the respective patch-pair function corresponding to the respective anchor point.

### Claim 15 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein determining the plurality of patch-pair functions comprises:
generating a hierarchy of the plurality of training input patches representing a plurality of groups of training input patches; and
determining a plurality of anchor points selecting a plurality of leaf nodes from the hierarchy, wherein each anchor point represents a respective one of the plurality of groups of training input patches; and
determining a respective patch-pair function for each of the plurality of anchor points.

### Claim 16

A system comprising:
a processor configured to execute instructions stored in a non-transitory computer-readable medium;
wherein the instructions comprise an image manipulation application configured to perform operations comprising:
generating a plurality of training patch pairs comprising a plurality of training input patches and a plurality of training output patches, wherein each training patch pair comprises a respective training input patch from a training input image and a respective training output patch from a training output image, wherein the training input image and the training output image include at least some of the same image content;
determining a plurality of patch-pair functions from at least some of the plurality of training patch pairs, wherein each of the plurality of patch-pair functions corresponds to a modification to a respective training input patch to generate a respective training output patch;
receiving an input image; and
generating an output image from the input image by applying at least some of the plurality of patch-pair functions based on at least some of a plurality of input patches of the input image.

### Claim 17 (depends on 16)

The system of claim 16, further comprising an imaging device configured to record the input image, wherein the processor is configured to receive the input image from the imaging device.

### Claim 18 (depends on 16)

The system of claim 16, wherein determining the plurality of patch-pair functions comprises:
partitioning the plurality of training input patches into a plurality of groups of training input patches, wherein partitioning the plurality of training input patches into the plurality of groups of training input patches comprises generating a hierarchy of the plurality of training input patches;
determining a plurality of anchor points by selecting a plurality of leaf nodes from the hierarchy, wherein each anchor point represents a respective one of the plurality of groups of training input patches; and
determining the respective patch-pair function for each of the plurality of anchor points.

### Claim 19 (depends on 16)

The system of claim 16, wherein determining the plurality of patch-pair functions comprises:
generating a hierarchy of the plurality of training input patches representing a plurality of groups of training input patches; and
determining a plurality of anchor points selecting a plurality of leaf nodes from the hierarchy, wherein each anchor point represents a respective one of the plurality of groups of training input patches; and
determining a respective patch-pair function for each of the plurality of anchor points.

### Claim 20 (depends on 19)

The system of claim 19, wherein applying the at least some of the plurality of patch-pair functions based on the at least some of the plurality of input patches of the input image comprises, for each input patch of the at least some of a plurality of input patches:
determining a distance between the input patch and a respective anchor point corresponding to the input patch;
determining whether the distance exceeds a threshold distance; and
performing at least one of:
applying the respective patch-pair function to the respective anchor point based on determining that the distance does not exceed the threshold distance; and
applying the respective patch-pair function to the input patch based on determining that the distance exceeds the threshold distance.
