# US 10,373,291 B1 · Image transformation for machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,373,291 B1
- Document ID: US10373291B1
- Status: Granted
- Kind: B1
- Application number: 15/885,178
- Filed: 2018-01-31
- Published: 2019-08-01
- Granted: 2019-08-06
- Earliest claimed: 2018-01-31
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Carrell Daniel Killebrew; Ravi Narayanaswami; Dong Hyuk Woo
- Web page: https://patentgrep.com/patent/US10373291B1

## CPC

- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06N 3/063: using electronic means
- G06N 20/00: Machine learning
- G06T 1/60: Memory management
- G06T 3/4046: using neural networks
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including an apparatus for determining pixel coordinates for image transformation and memory addresses for storing the transformed image data. In some implementations, a system includes a processing unit configured to perform machine learning computations for images using a machine learning model and pixel values for the images, a storage medium configured to store the pixel values for the images, and a memory address computation unit that includes one or more hardware processors. The processor(s) are configured to receive image data for an image and determine that the dimensions of the image do not match the dimensions of the machine learning model. In response, the processor(s) determine pixel coordinates for a transformed version of the input image and, for each of the pixel coordinates, memory address(es), in the storage medium, for storing pixel value(s) that will be used to generate an input to the machine learning model.

## Claims

### Claim 1

A system, comprising:
a processing unit configured to perform machine learning computations for images using a machine learning model and pixel values for the images, wherein the machine learning model is configured to process images having particular dimensions;
a storage medium configured to store the pixel values for the images; and
a memory address computation unit that includes one or more hardware processors configured to:
receive image data for an image to be processed as input by the machine learning model, the image data specifying dimensions of the image, the dimensions of the image specifying a horizontal pixel dimension and a vertical pixel dimension, wherein the horizontal pixel dimension is a number of pixels along the horizontal dimension of the image and the vertical pixel dimension is a number of pixels along the vertical dimension of the image;
determine that the dimensions of the image do not match the particular dimensions of images that the machine learning model is configured to process;
in response to determining that the dimensions of the image do not match the particular dimensions, determining memory addresses for a transformed version of the image that has dimensions that match the particular dimensions, including:
determine, based on the dimensions of the image and the particular dimensions, a horizontal pixel stride and a vertical pixel stride for the image; and
determine, using the horizontal pixel stride and the vertical pixel stride, a plurality of pixel coordinates for the transformed version of the image, each pixel coordinate including a vertical coordinate and a horizontal coordinate;
for each of the plurality of pixel coordinates, determine one or more memory addresses, in the storage medium, for storing one or more pixel values that will be used to generate an input to the machine learning model for the pixel coordinate, the one or more memory addresses for each pixel coordinate being based on the vertical coordinate and the horizontal coordinate of the pixel coordinate; and
output the memory address for each pixel value of each pixel coordinate to the processing unit, wherein the processing unit uses each memory address to obtain the pixel value stored in memory at the memory address and performs one or more machine learning computations using the obtained pixel value.

### Claim 2 (depends on 1)

The system of claim 1, wherein:
the memory address computation unit comprises:
one or more first hardware storage circuits for storing a current value of the horizontal coordinate; and
one or more second hardware storage circuits for storing a current value of the vertical coordinate;
the one or more hardware processors determine the plurality of pixel coordinates for the image by:
determining an initial pixel coordinate that includes an initial horizontal coordinate and an initial vertical coordinate;
storing the initial horizontal coordinate in the one or more first hardware storage circuits;
storing the initial vertical coordinate in the one or more second hardware storage circuits;
updating the current value of the horizontal coordinate for each iteration of a first loop used to traverse the horizontal pixel dimension of the image by adding the horizontal pixel stride to the current value of the horizontal coordinate after each iteration of the first loop; and
updating the current value of the vertical coordinate for each iteration of a second loop used to traverse the vertical pixel dimension of the image by adding the vertical pixel stride to the current value of the vertical coordinate after each iteration of the first second loop.

### Claim 3 (depends on 2)

The system of claim 2, wherein the determined memory address for each pixel coordinate corresponds to a location in the storage medium for storing a pixel value for a nearest neighbor pixel for the pixel coordinate, and wherein the pixel value at each nearest neighbor pixel is used by the processing unit as an input to the machine learning model.

### Claim 4 (depends on 2)

The system of claim 2, wherein determining the one or more memory addresses, in the storage medium, for storing the one or more pixel values that will be used to generate an input to the machine learning model for the pixel coordinate comprises:
for each iteration of the first loop or the second loop:
identifying the current value of the horizontal coordinate stored in the one or more first hardware storage circuits;
identifying the current value of the vertical coordinate stored in the one or more second hardware storage circuits; and
determining the memory address for the pixel coordinate that corresponds to the current value of the horizontal coordinate and the current value of the vertical coordinate based on the current value of the horizontal coordinate and the current value of the vertical coordinate.

### Claim 5 (depends on 4)

The system of claim 4, wherein determining the memory address for the pixel coordinate that corresponds to the current value of the horizontal coordinate and the current value of the vertical coordinate comprises:
determining a first partial address value based on a product of the current value of the horizontal coordinate and a first dimension multiplier;
determining a second partial address value based on a product of the current value of the vertical coordinate and a second dimension multiplier; and
determining the memory address for the pixel coordinate that corresponds to the current value of the horizontal coordinate and the current value of the vertical coordinate by determining a sum of the first partial address value and the second partial address value.

### Claim 6 (depends on 5)

The system of claim 5, wherein the memory address for the pixel coordinate that corresponds to the current value of the horizontal coordinate and the current value of the vertical coordinate equals the sum of the first partial address value, the second partial address value, and a base memory address.

### Claim 7 (depends on 2)

The system of claim 2, wherein the first loop is nested within the second loop or the second loop is nested within the first loop.

### Claim 8 (depends on 3)

The system of claim 3, wherein determining the one or more memory addresses, in the storage medium, for storing the one or more pixel values that will be used to generate an input to the machine learning model for the pixel coordinate comprises determining a respective memory address for each of four pixel coordinates for use in bilinear interpolation of a pixel value for the pixel coordinate, including:
for each iteration of the first loop or the second loop:
identifying the current value of the horizontal coordinate stored in the one or more first hardware storage circuits;
identifying the current value of the vertical coordinate stored in the one or more second hardware storage circuits;
identifying a vertical offset value;
identifying a horizontal offset value;
determining, for a first pixel coordinate of the four pixel coordinates, a first memory address for storing a first pixel value for the first pixel coordinate based on a sum of the current value of the horizontal coordinate and the current value of the vertical coordinate;
determining, for a second pixel coordinate of the four pixel coordinates, a second memory address for storing a second pixel value the second pixel coordinate based on a sum of the current value of the horizontal coordinate, the current value of the vertical coordinate, and the vertical offset value;
determining, for a third pixel coordinate of the four pixel coordinates, a third memory address for storing a third pixel value for the third pixel coordinate based on a sum of the current value of the horizontal coordinate, the current value of the vertical coordinate, and the horizontal offset value; and
determining, for a fourth pixel coordinate of the four pixel coordinates, a fourth memory address for storing a fourth pixel value for the fourth pixel coordinate based on a sum of the current value of the horizontal coordinate, the current value of the vertical coordinate, the vertical offset value, and the horizontal offset value.

### Claim 9 (depends on 8)

The system of claim 8, wherein the vertical offset value is based at least on the number of horizontal pixels in the image and the horizontal offset value is equal to one.

### Claim 10 (depends on 8)

The system of claim 8, wherein:
the one or more processors determines a bilinear interpolation weight for each of the four pixel coordinates and outputs each weight to the processing unit; and
the processing unit determines an input pixel value for the machine learning model using the pixel value at each of the four pixel coordinates and the weight for each of the four pixel coordinates.

### Claim 11 (depends on 10)

The system of claim 10, wherein the weight for each of the four pixel coordinates is based on a fractional portion of the horizontal coordinate of the pixel coordinate and a fractional portion of the vertical coordinate of the pixel coordinate.

### Claim 12 (depends on 1)

The system of claim 1, wherein the processing unit is configured to:
generate the input to the machine learning model by, for each of the plurality of pixel coordinates, determining one or more color values from the pixel values stored in the one or more memory addresses for the pixel coordinates; and
process the generated input using the machine learning model to generate a machine learning output.

### Claim 13

A method performed by one or more hardware processors, the method comprising:
receiving, by a memory address computation unit that includes the one or more hardware processors, image data for an image to be processed as input by a machine learning model, the image data specifying dimensions of the image, the dimensions of the image specifying a horizontal pixel dimension and a vertical pixel dimension, wherein the horizontal pixel dimension is a number of pixels along the horizontal dimension of the image and the vertical pixel dimension is a number of pixels along the vertical dimension of the image;
determining, by the memory address computation unit, that the dimensions of the image do not match particular dimensions of images that the machine learning model is configured to process;
in response to determining that the dimensions of the image do not match the particular dimensions, determining, by the memory address computation unit, memory addresses for a transformed version of the image that has dimensions that match the particular dimensions, including:
determining, by the memory address computation unit and based on the dimensions of the image and the particular dimensions, a horizontal pixel stride and a vertical pixel stride for the image; and
determining, by the memory address computation unit and using the horizontal pixel stride and the vertical pixel stride, a plurality of pixel coordinates for a transformed version of the image, each pixel coordinate including a vertical coordinate and a horizontal coordinate;
for each of the plurality of pixel coordinates, determining, by the memory address computation unit, one or more memory addresses, in a storage medium, for storing one or more pixel values that will be used to generate an input to the machine learning model for the pixel coordinate, the memory address for each pixel coordinate being based on the vertical coordinate and the horizontal coordinate of the pixel coordinate; and
outputting, by the memory address computation unit, the memory address for each pixel value of each pixel coordinate to a processing unit configured to perform machine learning computations for images using the machine learning model, wherein the processing unit uses each memory address to obtain, from a storage medium that stores the pixel values for the images, the pixel value stored in memory at the memory address and performs one or more machine learning computations using the obtained pixel value.

### Claim 14 (depends on 13)

The method of claim 13, wherein determining the plurality of pixel coordinates for the image comprises:
determining an initial pixel coordinate that includes an initial horizontal coordinate and an initial vertical coordinate;
storing the initial horizontal coordinate in one or more first hardware storage circuits;
storing the initial vertical coordinate in one or more second hardware storage circuits;
updating a current value of the horizontal coordinate for each iteration of a first loop used to traverse the horizontal pixel dimension of the image by adding the horizontal pixel stride to the current value of the horizontal coordinate after each iteration of the first loop and, for each update to the current value of the vertical coordinate, storing the current value of the horizontal coordinate in the one or more first hardware storage circuits; and
updating the current value of the vertical coordinate for each iteration of a second loop used to traverse the vertical pixel dimension of the image by adding the vertical pixel stride to the current value of the vertical coordinate after each iteration of the first second loop and, for each update to the current value of the vertical coordinate, storing the current value of the vertical coordinate in the one or more second hardware storage circuits.

### Claim 15 (depends on 14)

The method of claim 14, wherein the determined memory address for each pixel coordinate corresponds to a location in the storage medium for storing a pixel value for a nearest neighbor pixel for the pixel coordinate, and wherein the pixel value at each nearest neighbor pixel is used by the processing unit as an input to the machine learning model.

### Claim 16

An apparatus, comprising:
one or more first hardware storage circuits for storing a current value of a horizontal coordinate for an image; and
one or more second hardware storage circuits for storing a current value of a vertical coordinate for an image to be processed as input by a machine learning model;
one or more hardware processors configured to:
receive image data for the image, the image data specifying dimensions of the image, the dimensions of the image specifying a horizontal pixel dimension and a vertical pixel dimension, wherein the horizontal pixel dimension is a number of pixels along the horizontal dimension of the image and the vertical pixel dimension is a number of pixels along the vertical dimension of the image;
determine that the dimensions of the image do not match particular dimensions of images that the machine learning model is configured to process;
in response to determining that the dimensions of the image do not match the particular dimensions, determine memory addresses for a transformed version of the image that has dimensions that match the particular dimensions, including:
determine, based on the dimensions of the image and the particular dimensions, a horizontal pixel stride and a vertical pixel stride for the image;
determine, using the horizontal pixel stride and the vertical pixel stride, a plurality of pixel coordinates for a transformed version of the image, each pixel coordinate including a vertical coordinate and a horizontal coordinate;
store the horizontal coordinate in the one or more first hardware storage circuits; and
store the vertical coordinate in one or more second hardware storage circuits;
for each of the plurality of pixel coordinates, determine one or more memory addresses, in the storage medium, for storing one or more pixel values that will be used to generate an input to the machine learning model for the pixel coordinate, the memory address for each pixel coordinate being based on the vertical coordinate and the horizontal coordinate of the pixel coordinate; and
output the memory address for each pixel value of each pixel coordinate to a processing unit configured to perform machine learning computations for images using the machine learning model, wherein the processing unit uses each memory address to obtain the pixel value stored in memory at the memory address and performs one or more machine learning computations using the obtained pixel value.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein determining the plurality of pixel coordinates for the image comprises:
determining an initial pixel coordinate that includes an initial horizontal coordinate and an initial vertical coordinate;
storing the initial horizontal coordinate in the one or more first hardware storage circuits;
storing the initial vertical coordinate in the one or more second hardware storage circuits;
updating a current value of the horizontal coordinate for each iteration of a first loop used to traverse the horizontal pixel dimension of the image by adding the horizontal pixel stride to the current value of the horizontal coordinate after each iteration of the first loop and, for each update to the current value of the vertical coordinate, storing the current value of the horizontal coordinate in the one or more first hardware storage circuits; and
updating the current value of the vertical coordinate for each iteration of a second loop used to traverse the vertical pixel dimension of the image by adding the vertical pixel stride to the current value of the vertical coordinate after each iteration of the first second loop and, for each update to the current value of the vertical coordinate, storing the current value of the vertical coordinate in the one or more second hardware storage circuits.