# US 2015/0178591 A1 · System, method and computer-accessible medium for restoring an image taken through a window

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0178591 A1
- Document ID: US20150178591A1
- Status: Published application
- Kind: A1
- Application number: 14/575,179
- Filed: 2014-12-18
- Published: 2015-06-25
- Earliest claimed: 2013-12-18
- Assignees: New York University
- Applicants: New York University
- Inventors: ROB FERGUS; DAVID EIGEN; DILIP KRISHNAN
- Web page: https://patentgrep.com/patent/US20150178591A1

## CPC

- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30248: Vehicle exterior or interior
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems, methods and computer-accessible mediums for modifying an image(s) can be provided. For example, first image information for the image(s) can be received. Second image information can be generated by separating the first image information into at least two overlapping images. The image(s) can be modified using a prediction procedure based on the second image information.

## Claims

### Claim 1

A non-transitory computer-accessible medium having stored thereon computer-executable instructions for modifying at least one image, wherein, when a computer hardware arrangement executes the instructions, the computer arrangement is configured to perform procedures comprising:
receiving first image information for the at least one image;
generating second image information by separating the first image information into at least two overlapping images; and
modifying the at least one image using a prediction procedure based on the second image information.

### Claim 2 (depends on 1)

The non-transitory computer-accessible medium of claim 1, wherein the prediction procedure includes an average prediction determination for each overlapping pixels of the at least two overlapping images.

### Claim 3 (depends on 1)

The non-transitory computer-accessible medium of claim 1, wherein the prediction procedure includes predicting a clean image for at least one of the overlapping images.

### Claim 4 (depends on 1)

The non-transitory computer-accessible medium of claim 1, wherein the prediction procedure is associated with a neural network that receives each of the overlapping images as an input.

### Claim 5 (depends on 4)

The non-transitory computer-accessible medium of claim 4, wherein the neural network includes a series of layers.

### Claim 6 (depends on 5)

The non-transitory computer-accessible medium of claim 5, wherein each of the layers applies a linear map to at least one of the overlapping images.

### Claim 7 (depends on 6)

The non-transitory computer-accessible medium of claim 6, wherein each of the layers further applies an element-wise sigmoid to at least one of the overlapping images.

### Claim 8 (depends on 4)

The non-transitory computer-accessible medium of claim 4, wherein the neural network is a convolutional neural network.

### Claim 9 (depends on 8)

The non-transitory computer-accessible medium of claim 8, wherein the modification includes removing components of the at least one image associated with at least one of dirt, debris, or water from a picture taken through a window.

### Claim 10 (depends on 9)

The non-transitory computer-accessible medium of claim 9, wherein the neural network includes at least three neural networks, a first neural network for the removal of the dirt, a second neural network for the removal of the debris and a third neural network for the removal of the water.

### Claim 11 (depends on 1)

The non-transitory computer-accessible medium of claim 1, wherein the first image information includes information from a picture taken through a window having at least one of dirt, debris or water thereon.

### Claim 12 (depends on 1)

The non-transitory computer-accessible medium of claim 1, wherein the computer arrangement is further configured to generate at least one weight for use by the prediction procedure.

### Claim 13 (depends on 12)

The non-transitory computer-accessible medium of claim 12, wherein the computer arrangement is further configured to generate the weights by minimizing a mean squared error over a dataset of corresponding noisy and clean image pairs.

### Claim 14 (depends on 13)

The non-transitory computer-accessible medium of claim 13, wherein the computer arrangement is further configured to reduce a loss of the mean squared error.

### Claim 15 (depends on 14)

The non-transitory computer-accessible medium of claim 14, wherein the computer arrangement is further configured to reduce the loss using a Stochastic Gradient Descent.

### Claim 16 (depends on 15)

The non-transitory computer-accessible medium of claim 15, wherein the computer arrangement is further configured to determine a gradient of the error by backpropagating a depatchifying procedure.

### Claim 17 (depends on 12)

The non-transitory computer-accessible medium of claim 12, wherein the computer arrangement is further configured to initialize the at least one weight by randomly drawing from a normal distribution with a mean of about 0 and a standard deviation of about 0.001.

### Claim 18 (depends on 12)

The non-transitory computer-accessible medium of claim 12, wherein the computer arrangement is further configured to decorrelate the at least one weight.

### Claim 19

A method for modifying at least one image, comprising:
receiving first image information for the at least one image;
generating second image information by separating the first image information into at least two overlapping images; and
using a computer hardware arrangement, modifying the at least one image using a prediction procedure based on the second image information.

### Claim 20

A system for modifying at least one image, comprising
a computer hardware arrangement configured to:
receive first image information for the at least one image;
generate second image information by separating the first image information into at least two overlapping images; and
modify the at least one image using a prediction procedure based on the second image information.
