# US 10,313,656 B2 · Image stitching for three-dimensional video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,313,656 B2
- Document ID: US10313656B2
- Status: Granted
- Kind: B2
- Application number: 14/856,507
- Filed: 2015-09-16
- Published: 2016-03-24
- Granted: 2019-06-04
- Earliest claimed: 2014-09-22
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Company, Ltd.
- Inventors: Sajid Sadi; Sergio Perdices-Gonzalez; Rahul Budhiraja; Brian Dongwoo Lee; Ayesha Mudassir Khwaja; Pranav Mistry
- Web page: https://patentgrep.com/patent/US10313656B2

## CPC

- H04N 13/194: Transmission of image signals
- G02B 27/017: {Head mounted}
- G03B 35/08: by simultaneous recording
- G03B 37/04: with cameras or projectors providing touching or overlapping fields of view
- H04N 5/265: Mixing
- H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597)
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/254: in combination with electromagnetic radiation sources for illuminating objects
- H04N 13/261: with monoscopic-to-stereoscopic image conversion
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 19/54: using feature points or meshes
- H04N 19/597: specially adapted for multi-view video sequence encoding
- H04N 19/61: in combination with predictive coding
- H04N 23/80: Camera processing pipelines; Components thereof
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio
- H04N 23/957: Light-field or plenoptic cameras or camera modules
- H04N 23/959: by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics
- G02B 2027/0134: {of stereoscopic type}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G06T 2200/04: involving 3D image data
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

In one embodiment, a method includes receiving multiple images from multiple cameras, where the multiple cameras include a first camera having a first orientation and a second camera having a second orientation, and the multiple images include a first image from the first camera and a second image from the second camera. The method also includes detecting multiple feature points within the first and second images. The method further includes determining one or more pairs of corresponding feature points located within a region of overlap between the first and second images, where the pairs of corresponding feature points include a respective one of the feature points from each of the first and second images. The method also includes spatially adjusting the first or second image based on a calculated offset between each pair of corresponding feature points.

## Claims

### Claim 1

A method comprising:
receiving a plurality of images from a plurality of cameras, wherein the plurality of cameras comprises a first camera having a first orientation and a second camera having a second orientation, wherein the plurality of images comprises a first image from the first camera and a second image from the second camera;
detecting a plurality of feature points within the first and second images;
determining one or more pairs of corresponding feature points located within a region of overlap between the first and second images, wherein the pairs of corresponding feature points comprise a respective one of the feature points from each of the first and second images;
spatially adjusting one or more pixels of the first or second image based on a calculated offset between each pair of the corresponding feature points, wherein at least some of the one or more pixels are in the region of overlap;
comparing, prior to combining the first and second images, a portion of the first or second image containing the spatially adjusted one or more pixels with a third image containing the region of overlap, wherein the third image is taken by a third camera having the same orientation as either the first or the second camera;
either:
spatially adjusting, based on the comparison of the portion of the first or second image containing the spatially adjusted one or more pixels with the third image, at least one pixel within the portion; or
confirming, based on the comparison of the portion of the first or second image containing the spatially adjusted one or more pixels with the third image, the accuracy of the spatial adjustment made to the one or more pixels of the first or second image; and
combining the first and second images as spatially adjusted into a merged image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the region of overlap between the first and second images is proportional to an overlap in a respective field of view of the first and second cameras.

### Claim 3 (depends on 1)

The method of claim 1, wherein the detection comprises:
detecting one or more features of the first and second images based on a local contrast gradient; and
extracting one or more feature points of the detected features.

### Claim 4 (depends on 3)

The method of claim 3, wherein the extraction comprises using a scale-invariant feature transform (SIFT), speeded up robust features (SURF), or oriented FAST and Rotated BRIEF (ORB) feature detection algorithm.

### Claim 5 (depends on 1)

The method of claim 1, wherein the determination comprises:
partitioning the first and second images into a pre-determined number of regions in a pre-determined grid;
comparing feature points located within each region along an edge of the first image with a corresponding region along an opposite edge of the second image; and
identifying the corresponding pairs of feature points based on a pattern of feature points within each region of the edges.

### Claim 6 (depends on 5)

The method of claim 5, wherein the comparison within each region along the edge is performed on a pre-determined area around each feature point, wherein the pre-determined area is based on the number of pixels of the region of overlap.

### Claim 7 (depends on 6)

The method of claim 6, further comprising determining whether a sum of square of differences in the pre-determined area is less than a pre-determined threshold value.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
the third image is captured during a same time period as the first and second images;
the third camera has the first orientation; and
the second camera is located between the first and third cameras.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
detecting a plurality of feature points within the third image;
determining one or more pairs of corresponding feature points located within the third image and the region of overlap between the first and third images, wherein the pairs of corresponding feature points of the first and third images comprise a respective one of the feature points from each of the first and third images; and
determining one or more pairs of corresponding feature points located within a region of overlap between the second and third images, wherein the pairs of corresponding feature points of the second and third images comprise a respective one of the feature points from each of the second and third images.

### Claim 10 (depends on 8)

The method of claim 8, wherein the first and third camera comprise a stereoscopic pair of cameras.

### Claim 11 (depends on 1)

The method of claim 1, further comprising receiving a fourth image from a fourth camera, wherein the fourth camera has an orientation that is approximately orthogonal to a plane defined by the first and second cameras.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
determining one or more pairs of corresponding feature points located within a region of overlap between the merged image and the fourth image, wherein the pairs of corresponding feature points comprise a respective one of the feature points from each of the merged image and the fourth image; and
applying a radial warp around a center of the fourth image, such the center of the fourth image is fixed.

### Claim 13 (depends on 1)

The method of claim 1, wherein the calculation comprises calculating a cost function that is proportional to an absolute value of a spatial offset between each pair of corresponding feature points.

### Claim 14 (depends on 1)

The method of claim 1, wherein the calculation comprises calculating a cost function that is proportional to a rotational offset between each pair of corresponding feature points along one or more axes of rotation.

### Claim 15 (depends on 1)

The method of claim 1, wherein the second orientation is different from the first orientation, and wherein each image is captured during a same period of time.

### Claim 16 (depends on 1)

The method of claim 1, wherein combining the first and second images comprises:
detecting an edge in the region of overlap; and
applying a gradient on the edge.

### Claim 17 (depends on 1)

The method of claim 1, wherein the adjustment comprises a rotation or translation of the first or second image.

### Claim 18 (depends on 1)

The method of claim 1, wherein the first image, the second image, and the third image have not been combined with any other image prior to combining the first and second images into a merged image.

### Claim 19 (depends on 1)

The method of claim 1, further comprising:
accessing a fourth image from a fourth camera, wherein the fourth image is taken by a fourth camera having a different orientation than the third camera and the same orientation as either the first or the second camera;
detecting a second plurality of feature points within the third and fourth images;
determining one or more second pairs of corresponding feature points located within a region of overlap between the third and fourth images, wherein the second pairs of corresponding feature points comprise a respective one of the second feature points from each of the third and fourth images;
spatially adjusting the third or fourth image based on a calculated offset between each pair of the corresponding second feature points;
comparing the third and fourth images with either the first image or the second image containing the region of overlap between the third and fourth images; and
combining the third and fourth images into a merged image based on the spatial adjustment and on the comparison with the first or second image.

### Claim 20

One or more non-transitory computer-readable storage media embodying instructions that are operable when executed to:
receive a plurality of images from a plurality of cameras, wherein the plurality of cameras comprise a first camera having a first orientation and a second camera having a second orientation, wherein the plurality of images comprises a first image from the first camera and a second image from the second camera;
detect a plurality of feature points within the first and second images;
determine one or more pairs of corresponding feature points located within a region of overlap between the first and second images, wherein the pairs of corresponding feature points comprise a respective one of the feature points from each of the first and second images;
spatially adjust one or more pixels of the first or second image based on a calculated offset between each pair of the corresponding feature points;
compare, prior to combining the first and second images, a portion of the first or second image containing the spatially adjusted one or more pixels with a third image containing the region of overlap, wherein the third image is taken by a third camera having the same orientation as either the first or the second camera;
either:
spatially adjust, based on the comparison of the portion of the first or second image containing the spatially adjusted one or more pixels with the third image, at least one pixel within the portion; or
confirm, based on the comparison of the portion of the first or second image containing the spatially adjusted one or more pixels with the third image, the accuracy of the spatial adjustment made to the one or more pixels of the first or second image; and
combine the first and second images as spatially adjusted into a merged image.

### Claim 21 (depends on 20)

The media of claim 20, wherein the region of overlap between the first and second images is proportional to an overlap in a respective field of view of the first and second cameras.

### Claim 22 (depends on 20)

The media of claim 20, wherein the detect the plurality of feature points within the first and second images comprises:
detect one or more features of the first and second images based on a local contrast gradient; and
extract one or more feature points of the detected features.

### Claim 23 (depends on 20)

The media of claim 20, wherein the determine the one or more pairs of corresponding feature points located within the region of overlap between the first and second images comprises:
partition the first and second images into a pre-determined number of regions in a pre-determined grid;
compare feature points located within each region along an edge of the first image with a corresponding region along an opposite edge of the second image; and
identify the corresponding pairs of feature points based on a pattern of feature points within each region of the edges.

### Claim 24 (depends on 20)

The media of claim 20, wherein:
the third image is captured during a same time period as the first and second images;
the third camera has the first orientation; and
the second camera is located between the first and third cameras.

### Claim 25 (depends on 24)

The media of claim 24, wherein the instructions are further operable when executed to:
detect a plurality of feature points within the third image;
determine one or more pairs of corresponding feature points located within the third image and the region of overlap between the first and third images, wherein the pairs of corresponding feature points of the first and third images comprise a respective one of the feature points from each of the first and third images; and
determine one or more pairs of corresponding feature points located within a region of overlap between the second and third images, wherein the pairs of corresponding feature points of the second and third images comprise a respective one of the feature points from each of the second and third images.

### Claim 26 (depends on 20)

The media of claim 20, wherein the calculated offset between each pair of corresponding feature points comprises a cost function that is proportional to an absolute value of a spatial offset between each pair of corresponding feature points.

### Claim 27

An apparatus comprising:
one or more non-transitory computer-readable storage media embodying instructions; and
one or more processors coupled to the storage media and configured to execute the instructions to:
receive a plurality of images from a plurality of cameras, wherein the plurality of cameras comprise a first camera having a first orientation and a second camera having a second orientation, wherein the plurality of images comprises a first image from the first camera and a second image from the second camera;
detect a plurality of feature points within the first and second images;
determine one or more pairs of corresponding feature points located within a region of overlap between the first and second images, wherein the pairs of corresponding feature points comprise a respective one of the feature points from each of the first and second images;
spatially adjust one or more pixels of the first or second image based on a calculated offset between each pair of the corresponding feature points;
compare, prior to combining the first and second images, a portion of the first or second image containing the spatially adjusted one or more pixels with a third image containing the region of overlap, wherein the third image is taken by a third camera having the same orientation as either the first or the second camera;
either:
spatially adjust, based on the comparison of the portion of the first or second image containing the spatially adjusted one or more pixels with the third image, at least one pixel within the portion; or
confirm, based on the comparison of the portion of the first or second image containing the spatially adjusted one or more pixels with the third image, the accuracy of the spatial adjustment made to the one or more pixels of the first or second image; and
combine the first and second images as spatially adjusted into a merged image.

### Claim 28 (depends on 27)

The apparatus of claim 27, wherein the region of overlap between the first and second images is proportional to an overlap in a respective field of view of the first and second cameras.

### Claim 29 (depends on 27)

The apparatus of claim 27, wherein the detect the plurality of feature points within the first and second images comprises:
detect one or more features of the first and second images based on a local contrast gradient; and
extract one or more feature points of the detected features.

### Claim 30 (depends on 27)

The apparatus of claim 27, wherein the determine the one or more pairs of corresponding feature points located within the region of overlap between the first and second images comprises:
partition the first and second images into a pre-determined number of regions in a pre-determined grid;
compare feature points located within each region along an edge of the first image with a corresponding region along an opposite edge of the second image; and
identify the corresponding pairs of feature points based on a pattern of feature points within each region of the edges.

### Claim 31 (depends on 27)

The apparatus of claim 27, wherein the processors are further configured to execute the instructions to:
detect a plurality of feature points within the third image;
determine one or more pairs of corresponding feature points located within the third image and the region of overlap between the first and third images, wherein the pairs of corresponding feature points of the first and third images comprise a respective one of the feature points from each of the first and third images; and
determine one or more pairs of corresponding feature points located within a region of overlap between the second and third images, wherein the pairs of corresponding feature points of the second and third images comprise a respective one of the feature points from each of the second and third images.