# US 10,375,381 B2 · Omnistereo capture and render of panoramic virtual reality content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,375,381 B2
- Document ID: US10375381B2
- Status: Granted
- Kind: B2
- Application number: 15/843,366
- Filed: 2017-12-15
- Published: 2018-06-07
- Granted: 2019-08-06
- Earliest claimed: 2015-05-27
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Carlos Hernandez Esteban; Robert Anderson; Changchang Wu; Michael Krainin
- Web page: https://patentgrep.com/patent/US10375381B2

## CPC

- H04N 13/344: with head-mounted left-right displays
- G02B 27/017: {Head mounted}
- G02B 27/0172: {characterised by optical features}
- G06T 15/205: {Image-based rendering}
- H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597)
- H04N 13/211: using temporal multiplexing
- H04N 13/221: using the relative movement between cameras and objects
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/296: Synchronisation thereof; Control thereof
- H04N 13/366: using viewer tracking
- G02B 2027/011: {comprising device for correcting geometrical aberrations, distortion}
- G02B 2027/0134: {of stereoscopic type}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods are described include defining, at a computing device, a set of images based on captured images, projecting, at the computing device, a portion of the set of images from a planar perspective image plane onto a spherical image plane by recasting a plurality of viewing rays associated with the portion of the set of images from a plurality of viewpoints arranged around a curved path to a viewpoint, determining, at the computing device, a periphery boundary corresponding to the viewpoint and generating updated images by removing pixels that are outside of the periphery boundary, and providing, for display, the updated images within the bounds of the periphery boundary.

## Claims

### Claim 1

A system comprising:
at least one processor;
memory storing instructions that, when executed by the at least one processor, cause the system to perform operations including:
receiving a set of images based on captured video streams collected from at least one stereo pair of cameras;
calculating optical flow between images from the set of images to generate a plurality of image frames that are not part of the set of images, the calculating of the optical flow including analyzing image intensity fields for selected columns of pixels associated with the set of images;
interleaving the plurality of image frames into the set of images at the respective selected columns of pixels and stitching together a portion of the plurality of image frames and the set of images based at least in part on the optical flow; and
generating, using the portion of the plurality of image frames and the set of images, an omnistereo panorama.

### Claim 2 (depends on 1)

The system of claim 1, wherein the omnistereo panorama is formatted for display in a head mounted display device.

### Claim 3 (depends on 1)

The system of claim 1, wherein the set of images are based on captured video streams collected from 6 to 8 stereo pairs of cameras.

### Claim 4 (depends on 1)

The system of claim 1, further comprising:
obtaining pose information associated with the at least one stereo pair of cameras; and
determining the portion of the plurality of image frames to stitch together with the set of images based on the pose information.

### Claim 5 (depends on 1)

The system of claim 1, wherein the set of images correspond to rendered computer graphics (CG) content.

### Claim 6 (depends on 1)

The system of claim 1, wherein the stitching is performed using a stitching ratio selected based at least in part on a diameter of a circular camera rig housing the at least one stereo pair of cameras used to obtain the captured images.

### Claim 7

A computer program product tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to:
receive a set of images based on captured video streams collected from at least one stereo pair of cameras;
generate a plurality of image frames based at least in part on a calculated optical flow between images from the set of images, the calculated optical flow being based on analyzing image intensity fields for selected columns of pixels associated with the set of images;
interleave the plurality of image frames into the set of images at the respective selected columns of pixels and stitching together a portion of the plurality of image frames and the set of images based at least in part on the optical flow; and
generate, using the portion of the plurality of image frames and the set of images, an omnistereo panorama.

### Claim 8 (depends on 7)

The computer program product of claim 7, wherein the omnistereo panorama is formatted for display in a head mounted display device.

### Claim 9 (depends on 7)

The computer program product of claim 7, wherein the set of images are based on captured video streams collected from 6 stereo pairs of cameras.

### Claim 10 (depends on 7)

The computer program product of claim 7, further comprising:
obtaining pose information associated with the at least one stereo pair of cameras; and
determining the portion of the plurality of image frames to stitch together with the set of images based on the pose information.

### Claim 11 (depends on 7)

The computer program product of claim 7, wherein the set of images correspond to rendered computer graphics (CG) content.

### Claim 12 (depends on 7)

The computer program product of claim 7, wherein the stitching is performed using a stitching ratio selected based at least in part on a diameter of a circular camera rig housing the at least one stereo pair of cameras used to obtain the captured images.

### Claim 13 (depends on 1)

The system of claim 1, wherein the set of images include content captured using a rotatable camera rig, and the operations further comprise:
selecting, with the processor, portions of the content, the portions of the content corresponding to the plurality of image frames captured at a distance of about one meter from an outward edge of a base of the camera rig to about five meters from the outward edge of the base of the camera rig; and
stitching together the portions of the content.

### Claim 14 (depends on 13)

The system of claim 13, wherein the operations further comprise:
providing the omnistereo panorama to a head-mounted display device.

### Claim 15 (depends on 13)

The system of claim 13, wherein the stitching together of the portions of content is based on matching one or more image aspects included in at least two of the plurality of image frames.

### Claim 16 (depends on 13)

The system of claim 13, wherein the stitching of the portions of the content is performed using a stitching ratio selected based at least in part on a diameter of the rotatable camera rig that captured the content.

### Claim 17 (depends on 13)

The system of claim 13, the operations further comprising:
interpolating additional image frames that are not part of the portions of the content, and interleaving the additional image frames into the portions of the content;
stitching together the portions of content and the additional image frames; and
generating an updated stereoscopic panoramic view including the portions of the content and the additional image frames.

### Claim 18 (depends on 17)

The system of claim 17, wherein stitching together the portions of the content and the additional image frames includes:
matching a portion of a first column of pixels in a first image frame to a first portion of a second column of pixels in a second image frame,
matching a second portion of the second column of pixels to a portion of a third column of pixels in a third image frame; and
generating a cohesive scene portion by:
combining the first column of pixels with the first portion of the second column of pixels; and
combining the second portion of the second column of pixels with the third column of pixels.