# US 2017/0287200 A1 · Dual fisheye image stitching for spherical image content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0287200 A1
- Document ID: US20170287200A1
- Status: Published application
- Kind: A1
- Application number: 15/367,061
- Filed: 2016-12-01
- Published: 2017-10-05
- Earliest claimed: 2016-04-05
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Bijan Forutanpour; Phi Hung Le Nguyen; Ning Bi
- Web page: https://patentgrep.com/patent/US20170287200A1

## CPC

- G06T 15/04: Texture mapping
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T 3/12: Panospheric to cylindrical image transformations
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Examples are described for overlaying primitives, arranged as concentric circles, in circular images onto respective mesh models to generate rectangular images representative of a 360-degree video or image. Portions of the rectangular images are blended to generate a stitched rectangular image, and image content for display is generated based on the stitched rectangular image.

## Claims

### Claim 1

A method of generating image content, the method comprising:
overlaying first primitives, arranged as first concentric circles, in a first circular image onto a first mesh model to generate a first rectangular image;
overlaying second primitives, arranged as second concentric circles, in a second circular image onto a second mesh model to generate a second rectangular image, wherein the first circular image and the second circular image include overlapping image content;
blending a portion of the first rectangular image with a portion of the second rectangular image to generate a stitched rectangular image; and
generating image content for display based on the stitched rectangular image.

### Claim 2 (depends on 1)

The method of claim 1,
wherein overlaying the first primitives in the first circular image comprises textured rendering a first set of the first primitives of the first circular image to a first border of the first mesh model, and textured rendering a second, different set of the first primitives of the first circular image to a second border of the first mesh model to generate the first rectangular image,
wherein overlaying the second primitives in the second circular image comprises textured rendering a first set of the second primitives of the second circular image to a first border of the second mesh model, and textured rendering a second, different set of the second primitives of the second circular image to a second border of the second mesh model to generate the second rectangular image, and
wherein blending the portion of the first rectangular image with the portion of the second rectangular image comprises blending pixels located along the first border of the first rectangular image with pixels located along the first border of the second rectangular image to generate the stitched rectangular image.

### Claim 3 (depends on 2)

The method of claim 2,
wherein textured rendering the first set of the first primitives of the first circular image to the first border of the first mesh model comprises textured rendering primitives of the first circular image located along a perimeter of the first circular image to a bottom border of the first mesh model to generate primitives along a bottom border of the first rectangular image,
wherein textured rendering the second set of the first primitives of the first circular image to the second border of the first mesh model comprises textured rendering primitives of the first circular image connected to a center of the first circular image to a top border of the first mesh model to generate primitives along a top border of the first rectangular image,
wherein textured rendering the first set of the second primitives of the second circular image to the first border of the second mesh model comprises textured rendering primitives of the second circular image located along a perimeter of the second circular image to a top border of the second mesh model to generate primitives along a top border of the second rectangular image, and
wherein textured rendering the second set of the second primitives of the second circular image to the second border of the second mesh model comprises textured rendering primitives of the second circular image connected to a center of the second circular image to a bottom border of the second mesh model to generate primitives along a bottom border of the second rectangular image.

### Claim 4 (depends on 3)

The method of claim 3, wherein blending the portion of the first rectangular image with the portion of the second rectangular image comprises blending the primitives along the bottom border of the first rectangular image with the primitives along the top border of the second rectangular image to generate the stitched rectangular image.

### Claim 5 (depends on 2)

The method of claim 2, further comprising:
rotating image content of the stitched rectangular image by 90-degrees,
wherein generating image content for display based on the stitched rectangular image comprises generating image content for display based on the rotated stitched rectangular image.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
determining similarity between the overlapping image content in the first rectangular image and the second rectangular image;
adjusting at least one of size or shape of one or more of the first primitives arranged as first concentric circles in the first circular image, or size or shape of one or more of the first primitives arranged as first concentric circles in the first circular image and second primitives arranged as second concentric circles in the second circular image;
overlaying the adjusted primitives in the first circular image to generate a first adjusted rectangular image, or the adjusted primitives in the first and the second circular images to generate the first adjusted rectangular image and a second adjusted rectangular image; and
repeating the adjusting and overlaying until similarity between overlapping image content in the first adjusted rectangular image and the second adjusted rectangular image or the second rectangular image is within a similarity threshold,
wherein blending the portion of the first rectangular image with the portion of the second rectangular image comprises blending a portion of the first adjusted rectangular image with one of a portion of the second adjusted rectangular image or the portion of the second rectangular image.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
determining portions of the image content for display having visual artifacts;
executing one or more instances of a pixel shader for each pixel in the determined portions, wherein the pixel shader is configured to determine color values for a pixel in the determined portions based at least in part on color values of neighboring pixels.

### Claim 8 (depends on 1)

The method of claim 1, wherein the first circular image is captured by a first fisheye lens, and the second circular image is captured by a second fisheye lens.

### Claim 9 (depends on 8)

The method of claim 8,
wherein blending the portion of the first rectangular image with the portion of the second rectangular image comprises blending, with the fisheye camera device that includes the first and second fisheye lenses, the portion of the first rectangular image with the portion of the second rectangular image.

### Claim 10 (depends on 8)

The method of claim 8, further comprising:
receiving, with a processing device and from a fisheye camera device that includes the first fisheye lens and the second fisheye lens, the first circular image and the second circular image,
wherein blending the portion of the first rectangular image with the portion of the second rectangular image comprises blending, with the processing device, the portion of the first rectangular image with the portion of the second rectangular image.

### Claim 11

A device for generating image content, the device comprising:
a memory configured to store a first circular image and a second circular image; and
a graphics processing unit (GPU) comprising at least one of fixed-function circuitry or programmable circuitry, the GPU configured to:
overlay first primitives, arranged as first concentric circles, in the first circular image stored in the memory onto a first mesh model to generate a first rectangular image;
overlay second primitives, arranged as second concentric circles, in the second circular image stored in the memory onto a second mesh model to generate a second rectangular image, wherein the first circular image and the second circular image include overlapping image content;
blend a portion of the first rectangular image with a portion of the second rectangular image to generate a stitched rectangular image; and
generate image content for display based on the stitched rectangular image.

### Claim 12 (depends on 11)

The device of claim 11,
wherein to overlay the first primitives in the first circular image, the GPU is configured to textured render a first set of the first primitives of the first circular image to a first border of the first mesh model, and textured render a second, different set of the first primitives of the first circular image to a second border of the first mesh model to generate the first rectangular image,
wherein to overlay the second primitives in the second circular image, the GPU is configured to textured render a first set of the second primitives of the second circular image to a first border of the second mesh model, and textured render a second, different set of the second primitives of the second circular image to a second border of the second mesh model to generate the second rectangular image, and
wherein to blend the portion of the first rectangular image with the portion of the second rectangular image, the GPU is configured to blend pixels located along the first border of the first rectangular image with pixels located along the first border of the second rectangular image to generate the stitched rectangular image.

### Claim 13 (depends on 12)

The device of claim 12,
wherein to textured render the first set of the first primitives of the first circular image to the first border of the first mesh model, the GPU is configured to textured render primitives of the first circular image located along a perimeter of the first circular image to a bottom border of the first mesh model to generate primitives along a bottom border of the first rectangular image,
wherein to textured render the second set of the first primitives of the first circular image to the second border of the first mesh model, the GPU is configured to textured render primitives of the first circular image connected to a center of the first circular image to a top border of the first mesh model to generate primitives along a top border of the first rectangular image,
wherein to textured render the first set of the second primitives of the second circular image to the first border of the second mesh model, the GPU is configured to textured render primitives of the second circular image located along a perimeter of the second circular image to a top border of the second mesh model to generate primitives along a top border of the second rectangular image, and
wherein to textured render the second set of the second primitives of the second circular image to the second border of the second mesh model, the GPU is configured to textured render primitives of the second circular image connected to a center of the second circular image to a bottom border of the second mesh model to generate primitives along a bottom border of the second rectangular image.

### Claim 14 (depends on 13)

The device of claim 13, wherein to blend the portion of the first rectangular image with the portion of the second rectangular image, the GPU is configured to blend the primitives along the bottom border of the first rectangular image with the primitives along the top border of the second rectangular image to generate the stitched rectangular image.

### Claim 15 (depends on 12)

The device of claim 12, wherein the GPU is configured to:
rotate image content of the stitched rectangular image by 90-degrees,
wherein to generate image content for display based on the stitched rectangular image, the GPU is configured to generate image content for display based on the rotated stitched rectangular image.

### Claim 16 (depends on 11)

The device of claim 11, wherein the GPU is configured to:
determine similarity between the overlapping image content in the first rectangular image and the second rectangular image;
adjust at least one of size or shape of one or more of the first primitives arranged as first concentric circles in the first circular image, or size or shape of one or more of the first primitives arranged as first concentric circles in the first circular image and second primitives arranged as second concentric circles in the second circular image;
overlay the adjusted primitives in the first circular image to generate a first adjusted rectangular image, or the adjusted primitives in the first and the second circular images to generate the first adjusted rectangular image and a second adjusted rectangular image; and
repeat the adjusting and overlaying until similarity between overlapping image content in the first adjusted rectangular image and the second adjusted rectangular image or the second rectangular image is within a similarity threshold,
wherein to blend the portion of the first rectangular image with the portion of the second rectangular image, the GPU is configured to blend a portion of the first adjusted rectangular image with one of a portion of the second adjusted rectangular image or the portion of the second rectangular image.

### Claim 17 (depends on 11)

The device of claim 11, wherein the GPU is configured to:
determine portions of the image content for display having visual artifacts;
execute one or more instances of a pixel shader for each pixel in the determined portions, wherein the pixel shader is configured to determine color values for a pixel in the determined portions based at least in part on color values of neighboring pixels.

### Claim 18 (depends on 11)

The device of claim 11, further comprising a first fisheye lens and a second fisheye lens, wherein the first circular image is captured by the first fisheye lens, and the second circular image is captured by the second fisheye lens.

### Claim 19 (depends on 18)

The device of claim 18,
wherein to blend the portion of the first rectangular image with the portion of the second rectangular image, the GPU is configured to blend, with the fisheye camera device that includes the first and second fisheye lenses, the portion of the first rectangular image with the portion of the second rectangular image.

### Claim 20 (depends on 11)

The device of claim 11, wherein the memory is configured to store the first circular image and the second circular image from a fisheye camera device that includes a first fisheye lens and a second fisheye lens that generated the first circular image and the second circular image, respectively.

### Claim 21

A computer-readable storage medium storing instructions that when executed cause one or more processors of a device for generating image content to:
overlay first primitives, arranged as first concentric circles, in a first circular image onto a first mesh model to generate a first rectangular image;
overlay second primitives, arranged as second concentric circles, in a second circular image onto a second mesh model to generate a second rectangular image, wherein the first circular image and the second circular image include overlapping image content;
blend a portion of the first rectangular image with a portion of the second rectangular image to generate a stitched rectangular image; and
generate image content for display based on the stitched rectangular image.

### Claim 22 (depends on 21)

The computer-readable storage medium of claim 21,
wherein the instructions that cause the one or more processors to overlay the first primitives in the first circular image comprise instructions that cause the one or more processors to textured render a first set of the first primitives of the first circular image to a first border of the first mesh model, and textured render a second, different set of the first primitives of the first circular image to a second border of the first mesh model to generate the first rectangular image,
wherein the instructions that cause the one or more processors to overlay the second primitives in the second circular image comprise instructions that cause the one or more processors to textured render a first set of the second primitives of the second circular image to a first border of the second mesh model, and textured render a second, different set of the second primitives of the second circular image to a second border of the second mesh model to generate the second rectangular image, and
wherein the instructions that cause the one or more processors to blend the portion of the first rectangular image with the portion of the second rectangular image comprise instructions that cause the one or more processors to blend pixels located along the first border of the first rectangular image with pixels located along the first border of the second rectangular image to generate the stitched rectangular image.

### Claim 23 (depends on 22)

The computer-readable storage medium of claim 22,
wherein the instructions that cause the one or more processors to textured render the first set of the first primitives of the first circular image to the first border of the first mesh model comprise instructions that cause the one or more processors to textured render primitives of the first circular image located along a perimeter of the first circular image to a bottom border of the first mesh model to generate primitives along a bottom border of the first rectangular image,
wherein the instructions that cause the one or more processors to textured render the second set of the first primitives of the first circular image to the second border of the first mesh model comprise instructions that cause the one or more processors to textured render primitives of the first circular image connected to a center of the first circular image to a top border of the first mesh model to generate primitives along a top border of the first rectangular image,
wherein the instructions that cause the one or more processors to textured render the first set of the second primitives of the second circular image to the first border of the second mesh model comprise instructions that cause the one or more processors to textured render primitives of the second circular image located along a perimeter of the second circular image to a top border of the second mesh model to generate primitives along a top border of the second rectangular image, and
wherein the instructions that cause the one or more processors to textured render the second set of the second primitives of the second circular image to the second border of the second mesh model comprise instructions that cause the one or more processors to textured render primitives of the second circular image connected to a center of the second circular image to a bottom border of the second mesh model to generate primitives along a bottom border of the second rectangular image.

### Claim 24 (depends on 23)

The computer-readable storage medium of claim 23, wherein the instructions that cause the one or more processors to blend the portion of the first rectangular image with the portion of the second rectangular image comprise instructions that cause the one or more processors to blend the primitives along the bottom border of the first rectangular image with the primitives along the top border of the second rectangular image to generate the stitched rectangular image.

### Claim 25

A device for generating image content, the device comprising:
means for overlaying first primitives, arranged as first concentric circles, in a first circular image onto a first mesh model to generate a first rectangular image;
means for overlaying second primitives, arranged as second concentric circles, in a second circular image onto a second mesh model to generate a second rectangular image, wherein the first circular image and the second circular image include overlapping image content;
means for blending a portion of the first rectangular image with a portion of the second rectangular image to generate a stitched rectangular image; and
means for generating image content for display based on the stitched rectangular image.

### Claim 26 (depends on 25)

The device of claim 25,
wherein the means for overlaying the first primitives in the first circular image comprises means for textured rendering a first set of the first primitives of the first circular image to a first border of the first mesh model, and means for textured rendering a second, different set of the first primitives of the first circular image to a second border of the first mesh model to generate the first rectangular image,
wherein the means for overlaying the second primitives in the second circular image comprises means for textured rendering a first set of the second primitives of the second circular image to a first border of the second mesh model, and means for textured rendering a second, different set of the second primitives of the second circular image to a second border of the second mesh model to generate the second rectangular image, and
wherein means for blending the portion of the first rectangular image with the portion of the second rectangular image comprises means for blending pixels located along the first border of the first rectangular image with pixels located along the first border of the second rectangular image to generate the stitched rectangular image.

### Claim 27 (depends on 26)

The device of claim 26,
wherein the means for textured rendering the first set of the first primitives of the first circular image to the first border of the first mesh model comprises means for textured rendering primitives of the first circular image located along a perimeter of the first circular image to a bottom border of the first mesh model to generate primitives along a bottom border of the first rectangular image,
wherein the means for textured rendering the second set of the first primitives of the first circular image to the second border of the first mesh model comprises means for textured rendering primitives of the first circular image connected to a center of the first circular image to a top border of the first mesh model to generate primitives along a top border of the first rectangular image,
wherein the means for textured rendering the first set of the second primitives of the second circular image to the first border of the second mesh model comprises means for textured rendering primitives of the second circular image located along a perimeter of the second circular image to a top border of the second mesh model to generate primitives along a top border of the second rectangular image, and
wherein the means for textured rendering the second set of the second primitives of the second circular image to the second border of the second mesh model comprises means for textured rendering primitives of the second circular image connected to a center of the second circular image to a bottom border of the second mesh model to generate primitives along a bottom border of the second rectangular image.

### Claim 28 (depends on 27)

The device of claim 27, wherein the means for blending the portion of the first rectangular image with the portion of the second rectangular image comprises means for blending the primitives along the bottom border of the first rectangular image with the primitives along the top border of the second rectangular image to generate the stitched rectangular image.
