# US 12,118,754 B2 · Image signal processor for processing images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,754 B2
- Document ID: US12118754B2
- Status: Granted
- Kind: B2
- Application number: 18/494,099
- Filed: 2023-10-25
- Published: 2024-02-15
- Granted: 2024-10-15
- Earliest claimed: 2017-10-11
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Hau Hwang; Tushar Sinha Pankaj; Vishal Gupta; Jisoo Lee
- Web page: https://patentgrep.com/patent/US12118754B2

## CPC

- G06T 7/90: Determination of colour characteristics
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 3/06: Topological mapping of higher dimensional structures onto lower dimensional surfaces
- G06T 3/4015: Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
- G06T 3/4046: using neural networks
- G06N 3/048: Activation functions
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Techniques are provided for using one or more machine learning systems to process input data including image data. The input data including the image data can be obtained, and at least one machine learning system can be applied to at least a portion of the image data to determine at least one color component value for one or more pixels of at least the portion of the image data. Based on application of the at least one machine learning system to at least the portion of the image data, output image data for a frame of output image data can be generated. The output image data includes at least one color component value for one or more pixels of the frame of output image data. Application of the at least one machine learning system causes the output image data to have a reduced dimensionality relative to the input data.

## Claims

### Claim 1

A method of processing image data using one or more machine learning systems, the method comprising:
obtaining a first input patch of a frame of input image data;
generating, based on application of at least one machine learning system to the first input patch, a first output patch of image data for a frame of output image data, the first output patch including at least one color component value for one or more pixels of the first output patch;
obtaining a second input patch of the frame of input image data;
generating, based on application of the at least one machine learning system to the second input patch, a second output patch of image data for the frame of output image data, the second output patch including at least one color component value for one or more pixels of the second output patch; and
combining at least the first output patch and the second output patch to generate the frame of output image data.

### Claim 2 (depends on 1)

The method of claim 1, wherein the frame of input image data includes a frame of raw image data output from an image sensor.

### Claim 3 (depends on 2)

The method of claim 2, wherein the frame of raw image data is filtered by a color filter array.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first input patch includes a first patch of pixels from the frame of input image data, and wherein the second input patch includes a second patch of pixels from the frame of input image data.

### Claim 5 (depends on 4)

The method of claim 4, wherein the first patch of pixels of the first input patch overlaps with the second patch of pixels of the second input patch.

### Claim 6 (depends on 4)

The method of claim 4, wherein the first output patch and the second output patch are non-overlapping.

### Claim 7 (depends on 4)

The method of claim 4, wherein the first patch of pixels of the first input patch includes a same number of pixels as the second patch of pixels of the second input patch.

### Claim 8 (depends on 7)

The method of claim 7, wherein the first output patch includes a same number of pixels as the second output patch.

### Claim 9 (depends on 1)

The method of claim 1, wherein the at least one machine learning system includes at least one neural network.

### Claim 10 (depends on 1)

The method of claim 1, wherein application of the at least one machine learning system to the first input patch causes the first output patch to have a reduced dimensionality relative to the first input patch, and wherein application of the at least one machine learning system to the second input patch causes the second output patch to have a reduced dimensionality relative to the second input patch.

### Claim 11 (depends on 1)

The method of claim 1, wherein applying the at least one machine learning system to the first input patch includes:
applying one or more convolutional filters to the first input patch to generate reduced resolution data representative of the first input patch, each convolutional filter of the one or more convolutional filters including an array of weights.

### Claim 12 (depends on 1)

The method of claim 1, wherein application of the at least one machine learning system to the first input patch results in a plurality of color component values being determined for each pixel of the one or more pixels of the first output patch, and wherein application of the at least one machine learning system to the second input patch results in a plurality of color component values being determined for each pixel of the one or more pixels of the second output patch.

### Claim 13 (depends on 1)

The method of claim 1, wherein the first output patch of image data includes a first subset of pixels of the frame of output image data, and wherein the second output patch of image data includes a second subset of pixels of the frame of output image data.

### Claim 14 (depends on 1)

The method of claim 1, wherein the image data is captured by an image sensor.

### Claim 15 (depends on 1)

The method of claim 1, wherein the image data is captured by a plurality of image sensors.

### Claim 16

An apparatus for processing image data using one or more machine learning systems, the apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
obtain a first input patch of a frame of input image data;
generate, based on application of at least one machine learning system to the first input patch, a first output patch of image data for a frame of output image data, the first output patch including at least one color component value for one or more pixels of the first output patch;
obtain a second input patch of the frame of input image data;
generate, based on application of the at least one machine learning system to the second input patch, a second output patch of image data for the frame of output image data, the second output patch including at least one color component value for one or more pixels of the second output patch; and
combine at least the first output patch and the second output patch to generate the frame of output image data.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the frame of input image data includes a frame of raw image data output from an image sensor.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein the frame of raw image data is filtered by a color filter array.

### Claim 19 (depends on 16)

The apparatus of claim 16, wherein the first input patch includes a first patch of pixels from the frame of input image data, and wherein the second input patch includes a second patch of pixels from the frame of input image data.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the first patch of pixels of the first input patch overlaps with the second patch of pixels of the second input patch.

### Claim 21 (depends on 19)

The apparatus of claim 19, wherein the first output patch and the second output patch are non-overlapping.

### Claim 22 (depends on 19)

The apparatus of claim 19, wherein the first patch of pixels of the first input patch includes a same number of pixels as the second patch of pixels of the second input patch.

### Claim 23 (depends on 22)

The apparatus of claim 22, wherein the first output patch includes a same number of pixels as the second output patch.

### Claim 24 (depends on 16)

The apparatus of claim 16, wherein the at least one machine learning system includes at least one neural network.

### Claim 25 (depends on 16)

The apparatus of claim 16, wherein application of the at least one machine learning system to the first input patch causes the first output patch to have a reduced dimensionality relative to the first input patch, and wherein application of the at least one machine learning system to the second input patch causes the second output patch to have a reduced dimensionality relative to the second input patch.

### Claim 26 (depends on 16)

The apparatus of claim 16, wherein the at least one processor is configured to apply the at least one machine learning system to the first input patch, wherein, to apply the at least one machine learning system to the first input patch, the at least one processor is configured to:
apply one or more convolutional filters to the first input patch to generate reduced resolution data representative of the first input patch, each convolutional filter of the one or more convolutional filters including an array of weights.

### Claim 27 (depends on 16)

The apparatus of claim 16, wherein application of the at least one machine learning system to the first input patch results in a plurality of color component values being determined for each pixel of the one or more pixels of the first output patch, and wherein application of the at least one machine learning system to the second input patch results in a plurality of color component values being determined for each pixel of the one or more pixels of the second output patch.

### Claim 28 (depends on 16)

The apparatus of claim 16, wherein the first output patch of image data includes a first subset of pixels of the frame of output image data, and wherein the second output patch of image data includes a second subset of pixels of the frame of output image data.

### Claim 29 (depends on 16)

The apparatus of claim 16, wherein the image data is captured by an image sensor.

### Claim 30 (depends on 16)

The apparatus of claim 16, wherein the image data is captured by a plurality of image sensors.