# US 10,021,299 B2 · System and method for image stitching

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,021,299 B2
- Document ID: US10021299B2
- Status: Granted
- Kind: B2
- Application number: 15/168,251
- Filed: 2016-05-31
- Published: 2017-11-30
- Granted: 2018-07-10
- Earliest claimed: 2016-05-31
- Assignees: Tower Spring Global Limited
- Applicants: Tower Spring Global Limited
- Inventors: Eric Kwong Hang Tsang; Yuntao Sun
- Web page: https://patentgrep.com/patent/US10021299B2

## CPC

- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- H04N 5/265: Mixing
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Digital logic circuitry includes a plurality of logic blocks and memories that execute a method to stitch images. The method includes capturing the images with a plurality of pixel sensors, providing blocks of intensity values captured by the pixel sensors to input/output (I/O) pins of the digital logic circuitry, processing the intensity values with the digital logic circuitry, and stitching the processed images into a stitched image.

## Claims

### Claim 1

A method executed by a digital logic circuitry that includes a plurality of logic blocks and memories that stitch fisheye images into a 360° panoramic image, the method comprising:
capturing, with a camera, the fisheye images with a plurality of pixel sensors;
providing blocks of intensity values captured by the pixel sensors to input/output (I/O) pins of the digital logic circuitry without processing the intensity values with a processor;
processing, by the digital logic circuitry and with a mapping algorithm, the intensity values by converting the blocks from the fisheye images to rectilinear images; and
stitching, by the digital logic circuitry and with blending data obtained from edges of the rectilinear images, the rectilinear images into the 360° panoramic image,
wherein the mapping algorithm is based on a cubic interpolating coefficient generator represented by:
pixel
⁡
(
ux
,
uy
)
=
∑
i
=
0
3
⁢
∑
j
=
0
3
⁢
wx
⁡
(
i
,
ux
)
⁢
wy
⁡
(
j
,
uy
)
⁢
d
16
⁡
(
i
,
j
)
where d16 are the intensity values captured by the pixel sensors, and wx(i)=f(i, ux) and wy(j)=f(i, uy) are interpolation coefficients generated by the cubic interpolating coefficient generator with the following cubic interpolation functions:
f
⁡
(
0
,
x
)
=
-
3
×
x
3
+
6
×
x
2
-
3
×
x
4
,
f
⁡
(
1
,
x
)
=
5
×
x
3
-
9
×
x
2
+
4
4
,
f
⁡
(
2
,
x
)
=
-
5
×
x
3
+
6
×
x
2
+
3
×
x
4
,
f
⁡
(
3
,
x
)
=
3
×
x
3
-
3
×
x
2
4
.

### Claim 2 (depends on 1)

The method of claim 1, wherein the mapping algorithm includes:
dividing each of the blocks into tiles each having a predetermined number of pixels;
saving the intensity values of each tile to the memories in the digital logic circuitry;
obtaining the intensity values of pre-selected pixels of the tiles from the memories of the digital logic circuitry;
calculating, with the logic blocks of the digital logic circuitry, an interpolated pixel value based on the pre-selected pixels of the fisheye image; and
generating, by the digital logic circuitry and with the interpolated pixel value, the rectilinear images that have a number of output pixels with the interpolated pixel value.

### Claim 3 (depends on 1)

The method of claim 1, wherein the blending data are generated by:
dividing pixels on edges of the rectilinear images into a plurality of groups;
saving the intensity values of each of the groups to the memories of the digital logic circuitry;
applying a matrix to coordinates of the pixels of the groups; and
generating, with the logic blocks of the digital logic circuitry and by performing logic calculations to each of the groups, the blending data,
wherein the logic calculations include subtraction, summation, and multiplication.

### Claim 4 (depends on 1)

The method of claim 1, wherein the mapping algorithm is executed by a Field Programmable Gate Array (FPGA) to convert one of the fisheye images with M pixels to one of the rectilinear images with N pixels by mapping pixel locations in the rectilinear image to pixel locations in the fisheye image, wherein M is less than N.

### Claim 5 (depends on 1)

The method of claim 1, wherein a number of the blocks of the fisheye image are dependent upon a number of the logic blocks in the digital logic circuitry.

### Claim 6 (depends on 1)

The method of claim 1 further comprising:
calculating, with the digital logic circuitry and for each output pixel of the rectilinear images, coordinate points in the blocks of the fisheye image that correspond to the output pixel of the rectilinear image.