# US 2018/0027178 A1 · Mapping of spherical image data into rectangular faces for transport and decoding across networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0027178 A1
- Document ID: US20180027178A1
- Status: Published application
- Kind: A1
- Application number: 15/357,963
- Filed: 2016-11-21
- Published: 2018-01-25
- Earliest claimed: 2016-07-19
- Assignees: GoPro, Inc.
- Applicants: GoPro, Inc.
- Inventors: Timothy Macmillan; Adeel Abbas
- Web page: https://patentgrep.com/patent/US20180027178A1

## CPC

- G06T 3/073: Transforming surfaces of revolution to planar images, e.g. cylindrical surfaces to planar images
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 3/14: Transformations for image registration, e.g. adjusting or mapping for alignment of images
- G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/38: Registration of image sequences
- G06T 7/55: from multiple images
- G06T 7/593: from stereo images
- G06T 7/73: using feature-based methods
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 15/20: Perspective computation
- H04N 5/265: Mixing
- H04N 13/10: Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 13/302: for viewing without the aid of special glasses, i.e. using autostereoscopic displays
- H04N 13/334: using spectral multiplexing
- H04N 13/344: with head-mounted left-right displays
- H04N 19/136: Incoming video signal characteristics or properties
- H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06T 2200/04: involving 3D image data
- G06T 2207/20221: Image fusion; Image merging
- H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A system captures a first hemispherical image and a second hemispherical image, each hemispherical image including an overlap portion, the overlap potions capturing a same field of view, the two hemispherical images collectively comprising a spherical FOV and separated along a longitudinal plane. The system maps a modified first hemispherical image to a first portion of the 2D projection of a cubic image, the modified first hemispherical image including a non-overlap portion of the first hemispherical image, and maps a modified second hemispherical image to a second portion of the 2D projection of the cubic image, the modified second hemispherical image also including a non-overlap portion. The system maps the overlap portions of the first hemispherical image and the second hemispherical image to the 2D projection of the cubic image, and encodes the 2D projection of the cubic image to generate an encoded image representative of the spherical FOV.

## Claims

### Claim 1

A method for mapping spherical images to a 2D projection of a cubic representation of a spherical field of view (FOV) comprising:
capturing a first hemispherical image and a second hemispherical image, each hemispherical image including an overlap portion, the overlap potions capturing a same field of view, the two hemispherical images collectively comprising a spherical FOV and separated along a longitudinal plane;
mapping a modified first hemispherical image to a first portion of the 2D projection of a cubic image, the modified first hemispherical image including a non-overlap portion of the first hemispherical image;
mapping a modified second hemispherical image to a second portion of the 2D projection of the cubic image, the modified second hemispherical image including a non-overlap portion of the second hemispherical image;
mapping the overlap portions of the first hemispherical image and the second hemispherical image to the 2D projection of the cubic image; and
encoding the 2D projection of the cubic image to generate an encoded image representative of the spherical FOV.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first hemispherical image is captured by a first hemispherical camera with a hyper-hemispherical FOV, wherein the second hemispherical image is captured by a second hemispherical image with the hyper-hemispherical FOV, and wherein the first hemispherical camera is structurally fixed at a substantially opposite direction to a direction of the second hemispherical camera.

### Claim 3 (depends on 1)

The method of claim 1, wherein mapping the modified first hemispherical image to the first portion of the 2D projection comprises:
warping the modified first hemispherical image into a first square image;
extracting a first central portion from the first square image, the first central portion being a square image;
dividing the remainder of the first square image into four remaining portions of the first square image, the four remaining portions of the first square image having equal size;
mapping the first central portion to a first face of the projection of cubic image; and
mapping each of the four remaining portions of the first square image to one half of each of a set of four faces of the 2D projection of the cubic image, the set of four faces of the cubic image including a second, third, fourth, and fifth face of the 2D projection of cubic image.

### Claim 4 (depends on 3)

The method of claim 3, wherein the mapping the modified second hemispherical image further comprises:
warping the modified second hemispherical image into a second square image;
extracting a second central portion from the second square image, the second central portion being a square image;
dividing the remainder of the second square image into four remaining portions of the second square image, the four remaining portions of the second square image having equal size;
mapping the second central portion to a sixth face of the 2D projection of cubic image; and
mapping each of the four remaining portions of the second square image to a second half of each of the set of four faces of the 2D projection of cubic image, each of the four remaining portions of the second square image mapped to one of the set of four faces of the 2D projection of cubic image on which a remaining portion of the first square image representing an adjacent FOV is mapped.

### Claim 5 (depends on 4)

The method of claim 4, wherein the mapping the overlap portions of the first hemispherical image and the second hemispherical image to the 2D projection of the cubic image further comprises:
warping the overlap portions of the first hemispherical image and the second hemispherical image into rectangular images;
mapping each of the warped overlap portions of the first hemispherical image and the second hemispherical image to an additional portion of the 2D projection of cubic image.

### Claim 6 (depends on 5)

The method of claim 5, wherein two or more of the additional portions of the 2D projection of cubic image are arranged adjacent to each other in the 2D projection of cubic image.

### Claim 7 (depends on 5)

The method of claim 5, wherein the set of four faces of the 2D projection of the cubic image are arranged laterally adjacent to each other, and wherein a corresponding additional portion of the 2D projection of the cubic image is inserted between the two halves of each face of the set of four faces, each of the set of four faces representing a FOV that overlaps the FOV represented by the corresponding additional portion.

### Claim 8 (depends on 5)

The method of claim 5, wherein the faces of the 2D projection of the cubic image and the additional portions of the 2D projection of the cubic image are arranged to completely fill a rectangular bounding box.

### Claim 9 (depends on 8)

The method of claim 8, wherein at least one of the faces of the 2D projection of the cubic image is arranged adjacent to another face of the 2D projection of the cubic image such that the two faces represent adjacent FOVs of the original spherical FOV.

### Claim 10 (depends on 4)

The method of claim 4, wherein mapping the overlap portions of the first hemispherical image and the second hemispherical image to the 2D projection of the cubic image further comprises:
warping the overlap portions of the first hemispherical image and the second hemispherical image into rectangular images; and
encoding each of the warped overlap portions of the first hemispherical image and the second hemispherical image as a digital watermark in a corresponding face of the 2D projection of the projection of cubic image.

### Claim 11

A specially-configured hardware system for mapping spherical images to a projection of a cubic representation of a spherical field of view (FOV) configured to:
capture a first hemispherical image and a second hemispherical image, each hemispherical image including an overlap portion, the overlap potions capturing a same field of view, the two hemispherical images collectively comprising a spherical FOV and separated along a longitudinal plane;
map a modified first hemispherical image to a first portion of the 2D projection of a cubic image, the modified first hemispherical image including a non-overlap portion of the first hemispherical image;
map a modified second hemispherical image to a second portion of the 2D projection of the cubic image, the modified second hemispherical image including a non-overlap portion of the second hemispherical image;
map the overlap portions of the first hemispherical image and the second hemispherical image to the 2D projection of the cubic image; and
encode the 2D projection of the cubic image to generate an encoded image representative of the spherical FOV.

### Claim 12 (depends on 11)

The system of claim 11, wherein the first hemispherical image is captured by a first hemispherical camera with a hyper-hemispherical FOV, wherein the second hemispherical image is captured by a second hemispherical image with the hyper-hemispherical FOV, and wherein the first hemispherical camera is structurally fixed at a substantially opposite direction to a direction of the second hemispherical camera.

### Claim 13 (depends on 11)

The system of claim 11, wherein the system is further configured to:
warp the modified first hemispherical image into a first square image;
extract a first central portion from the first square image, the first central portion being a square image;
divide the remainder of the first square image into four remaining portions of the first square image, the four remaining portions of the first square image having equal size;
map the first central portion to a first face of the projection of cubic image; and
map each of the four remaining portions of the first square image to one half of each of a set of four faces of the 2D projection of the cubic image, the set of four faces of the cubic image including a second, third, fourth, and fifth face of the 2D projection of cubic image.

### Claim 14 (depends on 13)

The system of claim 13, wherein the system is further configured to:
warp the modified second hemispherical image into a second square image;
extract a second central portion from the second square image, the second central portion being a square image;
divide the remainder of the second square image into four remaining portions of the second square image, the four remaining portions of the second square image having equal size;
map the second central portion to a sixth face of the 2D projection of cubic image; and
map each of the four remaining portions of the second square image to a second half of each of the set of four faces of the 2D projection of cubic image, each of the four remaining portions of the second square image mapped to one of the set of four faces of the 2D projection of cubic image on which a remaining portion of the first square image representing an adjacent FOV is mapped.

### Claim 15 (depends on 14)

The system of claim 14, wherein the system is further configured to:
warp the overlap portions of the first hemispherical image and the second hemispherical image into rectangular images;
warp each of the warped overlap portions of the first hemispherical image and the second hemispherical image to an additional portion of the 2D projection of cubic image.

### Claim 16 (depends on 15)

The system of claim 15, wherein two or more of the additional portions of the 2D projection of cubic image are arranged adjacent to each other in the 2D projection of cubic image.

### Claim 17 (depends on 15)

The system of claim 15, wherein the set of four faces of the 2D projection of the cubic image are arranged laterally adjacent to each other, and wherein a corresponding additional portion of the 2D projection of the cubic image is inserted between the two halves of each face of the set of four faces, each of the set of four faces representing a FOV that overlaps the FOV represented by the corresponding additional portion.

### Claim 18 (depends on 15)

The system of claim 15, wherein the faces of the 2D projection of the cubic image and the additional portions of the 2D projection of the cubic image are arranged to completely fill a rectangular bounding box.

### Claim 19 (depends on 18)

The system of claim 18, wherein at least one of the faces of the 2D projection of the cubic image is arranged adjacent to another face of the 2D projection of the cubic image such that the two faces represent adjacent FOVs of the original spherical FOV.

### Claim 20 (depends on 14)

The system of claim 14, wherein the system is further configured to:
warp the overlap portions of the first hemispherical image and the second hemispherical image into rectangular images; and
encode each of the warped overlap portions of the first hemispherical image and the second hemispherical image as a digital watermark in a corresponding face of the 2D projection of the projection of cubic image.
