# US 10,455,211 B2 · Method and apparatus of image processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,455,211 B2
- Document ID: US10455211B2
- Status: Granted
- Kind: B2
- Application number: 15/605,474
- Filed: 2017-05-25
- Published: 2018-11-29
- Granted: 2019-10-22
- Earliest claimed: 2017-05-25
- Assignees: Advanced Micro Devices, Inc.
- Applicants: Advanced Micro Devices, Inc.
- Inventors: Michael L. Schmit; Radhakrishna Giduthuri; Kiriti Nagesh Gowda
- Web page: https://patentgrep.com/patent/US10455211B2

## CPC

- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
- H04N 23/55: Optical parts specially adapted for electronic image sensors; Mounting thereof
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method and apparatus of precomputing includes capturing a first image by a first image capturing device. An image space for the first image is defined and pixels in the image space are analyzed for validity. Valid pixels are stored as valid pixel groups and the valid pixel groups are processed.

## Claims

### Claim 1

A method of precomputing that reduces a number of computations required by a graphics pipeline, comprising:
capturing a first image of a plurality of images by a first image capturing device of a plurality of image capturing devices;
capturing a second image of the plurality of images by a second image capturing device of the plurality of image capturing devices;
defining a first image space for the first image that is a subset of the first image;
defining a second image space for the second image that is a subset of the second image;
capturing, by a respective image capturing device of the plurality of image capturing devices, a respective remaining image of each of the remaining plurality of images;
defining a respective remaining image space for each of the respective remaining images that is a subset of that image;
determining first valid pixels by analyzing pixels in the first image space for validity for processing of pixels captured by the first image capturing device;
determining second valid pixels by analyzing pixels in the second image space for validity for processing of pixels captured by the second image capturing device;
determining respective remaining valid pixels by analyzing pixels in each of the remaining respective image spaces for validity for processing of pixels captured by the respective image capturing devices;
storing the first valid pixels, the second valid pixels and the respective remaining valid pixels as a valid pixel group; and
once all valid pixels for all images spaces are determined and stored as the valid pixel group, receiving and processing, in a graphics pipeline, the valid pixel group thereby reducing the number of computations required.

### Claim 2 (depends on 1)

The method of claim 1 wherein defining the first and second image spaces includes defining the first and second image spaces as a subset of a total image to be projected into a three-hundred sixty degree space.

### Claim 3 (depends on 2)

The method of claim 2 wherein the projection is an equirectangular projection.

### Claim 4 (depends on 2)

The method of claim 2 wherein the defined first image space includes image space in a view of the first image capturing device and the defined second image space includes image space in a view of the second image capturing device.

### Claim 5 (depends on 2)

The method of claim 2, further comprising generating a mask that identifies the defined image spaces and excludes from processing pixels in areas of the total image outside the defined image spaces for the respective first or second image devices.

### Claim 6 (depends on 1)

The method of claim 1 wherein a boundary of the defined first image space is located in an overlap area between the first image and one or more adjacent images, and a boundary of the defined second image space is located in an overlap area between the second image and one or more adjacent images.

### Claim 7 (depends on 6)

The method of claim 6 wherein the boundary is defined by one or more seams formed in the overlap areas where the first image or second image and respective adjacent images are combined during a stitching process.

### Claim 8 (depends on 6)

The method of claim 6 wherein analyzing pixels in the first or second image spaces for validity includes determining if the pixel is within the boundary for that image space.

### Claim 9 (depends on 6)

The method of claim 6, further comprising storing a pixel within the boundary in the valid pixel group.

### Claim 10 (depends on 9)

The method of claim 9, further comprising storing a mirror pixel at the edge of the boundary in the valid pixel group.

### Claim 11 (depends on 10)

The apparatus of claim 10 wherein a boundary of the defined first image space is located in an overlap area between the first image and one or more adjacent images, and a boundary of the defined second image space is located in an overlap area between the second image and one or more adjacent images.

### Claim 12 (depends on 11)

The apparatus of claim 11 wherein the boundary is defined by one or more seams formed in the overlap areas where the first image or second image and respective adjacent images are combined during a stitching process.

### Claim 13 (depends on 11)

The apparatus of claim 11 wherein the computing device determines if the pixel is within the boundary and stores pixels within the boundary in the valid pixel group.

### Claim 14 (depends on 6)

The method of claim 6 wherein analyzing pixels in the first or second image spaces for validity includes determining pixels at the edge of the boundary and generating a mirror pixel for each pixel at the edge of the boundary.

### Claim 15

An apparatus that reduces a number of computations required by a graphics pipeline, comprising:
a computing device in communication with all image capturing devices of an image capturing system, wherein the computing device receives a first image of a plurality of images captured by a first image capturing device, receives a second image of the plurality of images captured by a second image capturing device, and a respective remaining image of each of the remaining plurality of images captured by each of the remaining respective image capturing devices, the computing device defines a first image space for the first image that is a subset of the first image, defines a second image space for the second image that is a subset of the second image, defines a respective remaining image space for each of the respective remaining images that is a subset of that image, determines first valid pixels by analyzing pixels in the first image space for validity for processing of pixels captured by the first image capturing device, determines second valid pixels by analyzing pixels in the second image space for validity for processing of pixels captured by the second image capturing device, determines respective remaining valid pixels by analyzing pixels in each of the remaining respective image spaces for validity for processing of pixels captured by the respective image capturing devices, stores the first valid pixels, the second valid pixels, and the respective remaining valid pixels as a valid pixel group, and processes, once all valid pixels for all images spaces are determined and stored as the valid pixel group, in a graphics pipeline of the computing device, the valid pixel group thereby reducing the number of computations required.

### Claim 16 (depends on 15)

The apparatus of claim 15 wherein the computing device defines the first and second image spaces as a subset of a total image to be projected into a three-hundred sixty degree space.

### Claim 17 (depends on 16)

The apparatus of claim 16 wherein the projection is an equirectangular projection.

### Claim 18 (depends on 16)

The apparatus of claim 16 wherein the defined first image space includes image space in a view of the first image capturing device and the defined second image space includes image space in a view of the second image capturing device.

### Claim 19 (depends on 16)

The apparatus of claim 16, further comprising the computing device generating a mask that identifies the defined image spaces and excludes from processing pixels in areas of the total image outside the defined image spaces for the respective first or second image devices.

### Claim 20

A non-transitory computer-readable medium that reduces a number of computations required by a graphics pipeline, the non-transitory computer-readable medium having instructions recorded thereon, that when executed by a computing device, cause the computing device to perform operations including:
receiving a first image of a plurality of images, captured by a first image capturing device of a plurality of image capturing devices;
receiving a second image of the plurality of images, captured by a second image capturing device of the plurality of image capturing devices;
receiving a respective remaining image of each of the remaining plurality of images, captured by a respective image capturing device of the plurality of image capturing devices;
defining a first image space for the first image that is a subset of the first image;
defining a second image space for the second image that is a subset of the second image;
defining a respective remaining image space for each of the respective remaining images that is a subset of that image;
determining first valid pixels by analyzing pixels in the first image space for validity for processing of pixels captured by the first image capturing device;
determining second valid pixels by analyzing pixels in the second image space for validity for processing of pixels captured by the second image capturing device;
determining respective remaining valid pixels by analyzing pixels in each of the remaining respective image spaces for validity for processing of pixels captured by the respective image capturing devices;
storing the first valid pixels, the second valid pixels and the respective remaining valid pixels as a valid pixel group; and
once all valid pixels for all images spaces are determined and stored as the valid pixel group, receiving and processing, in a graphics pipeline, the valid pixel group thereby reducing the number of computations required.