# US 10,484,664 B2 · Mapping of spherical image data into rectangular faces for transport and decoding across networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,484,664 B2 - Document ID: US10484664B2 - Status: Granted - Kind: B2 - Application number: 16/208,327 - Filed: 2018-12-03 - Published: 2019-06-13 - Granted: 2019-11-19 - Earliest claimed: 2016-07-19 - Assignees: GoPro, Inc. - Applicants: GoPro, Inc. - Inventors: Timothy Macmillan; Adeel Abbas - Web page: https://patentgrep.com/patent/US10484664B2 ## CPC - G06T 3/073: Transforming surfaces of revolution to planar images, e.g. cylindrical surfaces to planar images - G06F 18/22: Matching criteria, e.g. proximity measures - G06T 3/14: Transformations for image registration, e.g. adjusting or mapping for alignment of images - G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning - G06T 7/0002: {Inspection of images, e.g. flaw detection} - G06T 7/38: Registration of image sequences - G06T 7/55: from multiple images - G06T 7/593: from stereo images - G06T 7/73: using feature-based methods - G06T 11/60: Creating or editing images; Combining images with text - G06T 15/20: Perspective computation - H04N 5/265: Mixing - H04N 13/10: Processing, recording or transmission of stereoscopic or multi-view image signals - H04N 13/243: using three or more two-dimensional [2D] image sensors - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - H04N 13/302: for viewing without the aid of special glasses, i.e. using autostereoscopic displays - H04N 13/334: using spectral multiplexing - H04N 13/344: with head-mounted left-right displays - H04N 19/136: Incoming video signal characteristics or properties - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - G06T 2200/04: involving 3D image data - G06T 2207/20221: Image fusion; Image merging - H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system captures a first hemispherical image and a second hemispherical image, each hemispherical image including an overlap portion, the overlap portion capturing a same field of view, the two hemispherical images collectively comprising a spherical FOV and separated along a longitudinal plane. The system maps a modified first hemispherical image to a first portion of the 2D projection of a cubic image, the modified first hemispherical image including a non-overlap portion of the first hemispherical image, and maps a modified second hemispherical image to a second portion of the 2D projection of the cubic image, the modified second hemispherical image also including a non-overlap portion. The system maps the overlap portions of the first hemispherical image and the second hemispherical image to the 2D projection of the cubic image, and encodes the 2D projection of the cubic image to generate an encoded image representative of the spherical FOV. ## Claims ### Claim 1 A method for mapping spherical images to a two-dimensional projection of a cubic representation of a spherical field of view (FOV), the method comprising: capturing a first hyper-hemispherical image and a second hyper-hemispherical image, each of the first hyper-hemispherical image and the second hyper-hemispherical image comprising an overlap portion, the overlap portions capturing a same area within the spherical FOV; modifying the first hyper-hemispherical image, the modifying comprising removing the overlap portion of the first hyper-hemispherical image, the modifying of the first hyper-hemispherical image resulting in a modified first hemispherical image; modifying the second hyper-hemispherical image, the modifying comprising removing the overlap portion of the second hyper-hemispherical image, the modifying of the second hyper-hemispherical image resulting in a modified second hemispherical image; mapping a first portion of the modified first hemispherical image into first, second and third facets of the two-dimensional projection of the cubic representation, the first, the second and the third facets of the two-dimensional projection of the cubic representation defining a first extent of contiguous space within the spherical FOV; mapping a first portion of the modified second hemispherical image into the first, the second and the third facets of the two-dimensional projection of the cubic representation of the spherical FOV; mapping a first portion of the overlap portions into the first, the second and the third facets of the two-dimensional projection of the cubic representation of the spherical FOV; and encoding the two-dimensional projection of the cubic representation of the spherical FOV to generate an encoded image representative of the spherical FOV. ### Claim 2 (depends on 1) The method of claim 1, further comprising: mapping a second portion of the modified first hemispherical image into a fourth facet of the two-dimensional projection of the cubic representation; mapping a second portion of the modified second hemispherical image into the fourth facet of the two-dimensional projection of the cubic representation; and mapping a remainder portion of the overlap portions into the fourth facet of the two-dimensional projection of the cubic representation. ### Claim 3 (depends on 2) The method of claim 2, further comprising arranging the fourth facet of the two-dimensional projection of the cubic representation with the first, the second, and the third facets of the two-dimensional projection of the cubic representation, the arranging defining a second extent of contiguous space within the spherical FOV, the second extent of contiguous space including the first extent of contiguous space within the spherical FOV. ### Claim 4 (depends on 3) The method of claim 3, further comprising mapping a third portion of the modified first hemispherical image into a fifth facet of the two-dimensional projection of the cubic representation; mapping a third portion of the modified second hemispherical image into a sixth facet of the two-dimensional projection of the cubic representation; and arranging the fifth and the sixth facet of the two-dimensional projection of the cubic representation with the second facet of the two-dimensional projection of the cubic representation such that the second, the fifth, and the sixth facets are defining a third extent of contiguous space within the spherical FOV. ### Claim 5 (depends on 2) The method of claim 2, further comprising rotating the fourth facet of the two-dimensional projection of the cubic representation; and arranging the rotated fourth facet of the two-dimensional projection of the cubic representation adjacent the second facet of the two-dimensional projection of the cubic representation, a boundary between the rotated fourth facet and the second facet comprising a non-contiguous extent of space within the spherical FOV. ### Claim 6 (depends on 5) The method of claim 5, further comprising mapping a third portion of the modified first hemispherical image into a fifth facet of the two-dimensional projection of the cubic representation; and mapping a third portion of the modified second hemispherical image into a sixth facet of the two-dimensional projection of the cubic representation. ### Claim 7 (depends on 6) The method of claim 6, further comprising arranging the fifth facet of the two-dimensional projection of the cubic representation adjacent to the rotated fourth facet of the two-dimensional projection of the cubic representation; and arranging the sixth facet of the two-dimensional projection of the cubic representation adjacent to the rotated fourth facet of the two-dimensional projection of the cubic representation such that the fourth, the fifth and the sixth facets of the two-dimensional projection of the cubic representation comprises a second extent of contiguous space within the spherical FOV. ### Claim 8 A computer readable apparatus comprising a storage medium adapted to store a computer program thereon, the computer program being configured to, when executed, cause a computerized apparatus to: retrieve a first hyper-hemispherical image and a second hyper-hemispherical image, each of the first hyper-hemispherical image and the second hyper-hemispherical image comprising an overlap portion, the overlap portions capturing a same area within a spherical field of view (FOV); modify the first hyper-hemispherical image, the modification comprising removal of the overlap portion of the first hyper-hemispherical image, the modification of the first hyper-hemispherical image resulting in a modified first hemispherical image; modify the second hyper-hemispherical image, the modification comprising removing the overlap portion of the second hyper-hemispherical image, the modification of the second hyper-hemispherical image resulting in a modified second hemispherical image; map a first portion of the modified first hemispherical image into first, second and third facets of a two-dimensional projection of a cubic representation, the first, the second and the third facets of the two-dimensional projection of the cubic representation defining a first extent of contiguous space within the spherical FOV; map a first portion of the modified second hemispherical image into the first, the second and the third facets of the two-dimensional projection of the cubic representation of the spherical FOV; map a first portion of the overlap portions into the first, the second and the third facets of the two-dimensional projection of the cubic representation of the spherical FOV; and encode the two-dimensional projection of the cubic representation of the spherical FOV to generate an encoded image representative of the spherical FOV. ### Claim 9 (depends on 8) The computer readable apparatus of claim 8, wherein the computer program, when executed, is further configured to cause the computerized apparatus to: map a second portion of the modified first hemispherical image into a fourth facet of the two-dimensional projection of the cubic representation; map a second portion of the modified second hemispherical image into the fourth facet of the two-dimensional projection of the cubic representation; and map a remainder portion of the overlap portions into the fourth facet of the two-dimensional projection of the cubic representation. ### Claim 10 (depends on 9) The computer readable apparatus of claim 9, wherein the computer program is further configured to, when executed, cause the computerized apparatus to: arrange the fourth facet of the two-dimensional projection of the cubic representation with the first, the second, and the third facets of the two-dimensional projection of the cubic representation, the arrangement defining a second extent of contiguous space within the spherical FOV, the second extent of contiguous space including the first extent of contiguous space within the spherical FOV. ### Claim 11 (depends on 10) The computer readable apparatus of claim 10, wherein the computer program is further configured to, when executed, cause the computerized apparatus to: map a third portion of the modified first hemispherical image into a fifth facet of the two-dimensional projection of the cubic representation; map a third portion of the modified second hemispherical image into a sixth facet of the two-dimensional projection of the cubic representation; and arrange the fifth and the sixth facet of the two-dimensional projection of the cubic representation with the second facet of the two-dimensional projection of the cubic representation such that the second, the fifth, and the sixth facets are defining a third extent of contiguous space within the spherical FOV. ### Claim 12 (depends on 9) The computer readable apparatus of claim 9, wherein the computer program is further configured to, when executed, cause the computerized apparatus to: rotate the fourth facet of the two-dimensional projection of the cubic representation; and arrange the rotated fourth facet of the two-dimensional projection of the cubic representation adjacent the second facet of the two-dimensional projection of the cubic representation, a boundary between the rotated fourth facet and the second facet comprising a non-contiguous extent of space within the spherical FOV. ### Claim 13 (depends on 12) The computer readable apparatus of claim 12, wherein the computer program is further configured to, when executed, cause the computerized apparatus to: map a third portion of the modified first hemispherical image into a fifth facet of the two-dimensional projection of the cubic representation; and map a third portion of the modified second hemispherical image into a sixth facet of the two-dimensional projection of the cubic representation. ### Claim 14 (depends on 13) The computer readable apparatus of claim 13, wherein the computer program is further configured to, when executed, cause the computerized apparatus to: arrange the fifth facet of the two-dimensional projection of the cubic representation adjacent to the rotated fourth facet of the two-dimensional projection of the cubic representation; and arrange the sixth facet of the two-dimensional projection of the cubic representation adjacent to the rotated fourth facet of the two-dimensional projection of the cubic representation such that the fourth, the fifth and the sixth facets of the two-dimensional projection of the cubic representation comprises a second extent of contiguous space within the spherical FOV. ### Claim 15 An image processing system for mapping spherical images to a two-dimensional projection of a cubic representation of a spherical field of view (FOV), the image processing system configured to: retrieve a first hyper-hemispherical image and a second hyper-hemispherical image from memory, each of the first hyper-hemispherical image and the second hyper-hemispherical image comprising an overlap portion, the overlap portions capturing a same area within a spherical field of view (FOV); modify the first hyper-hemispherical image, the modification comprising removal of the overlap portion of the first hyper-hemispherical image, the modification of the first hyper-hemispherical image resulting in a modified first hemispherical image; modify the second hyper-hemispherical image, the modification comprising removing the overlap portion of the second hyper-hemispherical image, the modification of the second hyper-hemispherical image resulting in a modified second hemispherical image; map a first portion of the modified first hemispherical image into first, second and third facets of the two-dimensional projection of the cubic representation, the first, the second and the third facets of the two-dimensional projection of the cubic representation defining a first extent of contiguous space within the spherical FOV; map a first portion of the modified second hemispherical image into the first, the second and the third facets of the two-dimensional projection of the cubic representation of the spherical FOV; map a first portion of the overlap portions into the first, the second and the third facets of the two-dimensional projection of the cubic representation of the spherical FOV; and encode the two-dimensional projection of the cubic representation of the spherical FOV to generate an encoded image representative of the spherical FOV. ### Claim 16 (depends on 15) The image processing system of claim 15, the image processing system further configured to: map a second portion of the modified first hemispherical image into a fourth facet of the two-dimensional projection of the cubic representation; map a second portion of the modified second hemispherical image into the fourth facet of the two-dimensional projection of the cubic representation; and map a remainder portion of the overlap portions into the fourth facet of the two-dimensional projection of the cubic representation. ### Claim 17 (depends on 16) The image processing system of claim 16, the image processing system further configured to: arrange the fourth facet of the two-dimensional projection of the cubic representation with the first, the second, and the third facets of the two-dimensional projection of the cubic representation, the arrangement defining a second extent of contiguous space within the spherical FOV, the second extent of contiguous space including the first extent of contiguous space within the spherical FOV. ### Claim 18 (depends on 17) The image processing system of claim 17, the image processing system further configured to: map a third portion of the modified first hemispherical image into a fifth facet of the two-dimensional projection of the cubic representation; map a third portion of the modified second hemispherical image into a sixth facet of the two-dimensional projection of the cubic representation; and arrange the fifth and the sixth facet of the two-dimensional projection of the cubic representation with the second facet of the two-dimensional projection of the cubic representation such that the second, the fifth, and the sixth facets are defining a third extent of contiguous space within the spherical FOV. ### Claim 19 (depends on 16) The image processing system of claim 16, the image processing system further configured to: rotate the fourth facet of the two-dimensional projection of the cubic representation; and arrange the rotated fourth facet of the two-dimensional projection of the cubic representation adjacent the second facet of the two-dimensional projection of the cubic representation, a boundary between the rotated fourth facet and the second facet comprising a non-contiguous extent of space within the spherical FOV. ### Claim 20 (depends on 19) The image processing system of claim 19, the image processing system further configured to: map a third portion of the modified first hemispherical image into a fifth facet of the two-dimensional projection of the cubic representation; map a third portion of the modified second hemispherical image into a sixth facet of the two-dimensional projection of the cubic representation; arrange the fifth facet of the two-dimensional projection of the cubic representation adjacent to the rotated fourth facet of the two-dimensional projection of the cubic representation; and arrange the sixth facet of the two-dimensional projection of the cubic representation adjacent to the rotated fourth facet of the two-dimensional projection of the cubic representation such that the fourth, the fifth and the sixth facets of the two-dimensional projection of the cubic representation comprises a second extent of contiguous space within the spherical FOV.