# US 2020/0349672 A1 · Method and apparatus for processing and distributing live virtual reality content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0349672 A1
- Document ID: US20200349672A1
- Status: Published application
- Kind: A1
- Application number: 16/958,698
- Filed: 2019-01-24
- Published: 2020-11-05
- Earliest claimed: 2018-02-07
- Assignees: Intel Corporation
- Inventors: Wayne COCHRAN; Fai YEUNG; Durga Raj MATHUR; Gilson GONCALVES DE LIMA; Patrick Youngung SHON; John A. HARRISON; Ok Joon KIM; Harleen GILL; Kyle SIEHL; Uma JAYARAM; Sankar JAYARAM; Archie SHARMA; Gockcen CLINGIR; Stanley BARAN; Mayuresh VARERKAR; Barnan DAS; Narayan BISWAL; Nilesh SHAH; Ritesh KALE; Greg WEINSTEIN
- Web page: https://patentgrep.com/patent/US20200349672A1

## CPC

- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 15/10: Geometric effects
- G06T 3/12: Panospheric to cylindrical image transformations
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06T 15/08: Volume rendering
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G11B 27/10: Indexing; Addressing; Timing or synchronising; Measuring tape travel
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras}
- H04N 21/2187: Live feed
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/234309: {by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo (conversion of standards in analog television systems H04N7/01)}
- H04N 21/2402: {Monitoring of the downstream path of the transmission network, e.g. bandwidth available (traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876)}
- H04N 21/242: Synchronisation processes, e.g. processing of PCR [Programme Clock References] {(arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18)}
- H04N 21/8133: {specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video programme}
- H04N 21/816: {involving special video data, e.g 3D video}
- H04N 23/662: {by using camera arrangements where one camera controls another camera to affect the control of camera image capture, e.g. placing the camera in a desirable condition to capture a desired image}
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An apparatus, system, and method are described for providing real-time capture, processing, and distribution of panoramic virtual reality (VR) content. For example, one embodiment of a graphics processor comprises a video interface to receive a first plurality of images from a corresponding first plurality of cameras; an image rectifier to perform a perspective re-projection of at least some of the first plurality of images to a common image plane to generate a rectified first plurality of images; a stitcher to analyze overlapping regions of adjacent images in the rectified first plurality and to identify corresponding pixels in the overlapping regions and to stitch the adjacent images in accordance with the corresponding pixels to generate a panoramic image comprising a stitched combination of the rectified first plurality of images; and a cylindrical projector to project the panoramic image onto a cylindrical surface to generate a final panoramic video image to be used to implement a virtual reality (VR) environment on a VR apparatus.

## Claims

### Claim 1

A graphics processor comprising:
a video interface to receive a first plurality of images from a corresponding first plurality of cameras;
an image rectifier to perform a perspective re-projection of at least some of the first plurality of images to a common image plane to generate a rectified first plurality of images;
a stitcher to analyze overlapping regions of adjacent images in the rectified first plurality and to identify corresponding pixels in the overlapping regions and to stitch the adjacent images in accordance with the corresponding pixels to generate a panoramic image comprising a stitched combination of the rectified first plurality of images; and
a cylindrical projector to project the panoramic image onto a cylindrical surface to generate a final panoramic video image to be used to implement a virtual reality (VR) environment on a VR apparatus.

### Claim 2 (depends on 1)

The graphics processor of claim 1 wherein the stitcher is to analyze the overlapping regions by executing a sequence of Belief Propagation operations.

### Claim 3 (depends on 2)

The graphics processor of claim 2 wherein the sequence of Belief Propagation operations comprise:
construction of an initial data cost volume using pixel data in the overlapping regions;
construction of a data cost pyramid based on the initial data cost volume comprising a series of smaller volumes;
iteration through the series of smaller volumes and the initial volume using Belief Propagation message passing to generate a final set of costs; and
constructing a stitch map from the final set of costs.

### Claim 4 (depends on 3)

The graphics processor of claim 3 wherein the stitcher is to stitch the adjacent images in accordance with the final set of costs.

### Claim 5 (depends on 4)

The graphics processor of claim 4 wherein for each pixel in the overlapping region the stitcher is to use the stitch map to determine which pixels to blend.

### Claim 6 (depends on 5)

The graphics processor of claim 5 wherein the stitcher is to use a convex linear combination of pixels to blend from each image.

### Claim 7 (depends on 1, 6)

The graphics processor of claim 1 or 6 further comprising:
a lookahead buffer to store stitching parameters determined from one or more prior images, the stitcher to use at least a portion of the stitching parameters stored in the lookahead buffer to stitch the adjacent images in the rectified first plurality of images.

### Claim 8 (depends on 1, 7)

The graphics processor of claim 1 or 7 wherein the perspective re-projection comprises a homography transformation.

### Claim 9 (depends on 1, 8)

The graphics processor of claim 1 or 8 wherein the first plurality of images comprise left images to be displayed, following processing, in a left display of the VR apparatus.

### Claim 10 (depends on 9)

The graphics processor of claim 9 wherein the video interface is to receive a right plurality of images from a corresponding second plurality of cameras,
wherein the image rectifier is to perform a perspective re-projection of at least some of the right plurality of images to a common image plane,
wherein the stitcher is to analyze overlapping regions of adjacent images in the right plurality and to identify corresponding pixels in the overlapping regions and to stitch the adjacent images in accordance with the corresponding pixels to generate a panoramic right eye image comprising a stitched combination of the right plurality of images; and
wherein the cylindrical projector is to project the panoramic right eye image onto a cylindrical surface to generate a final panoramic right eye image to be combined with the final panoramic left eye image to implement a virtual reality (VR) environment on a VR apparatus.

### Claim 11 (depends on 1)

The graphics processor of claim 1 wherein the image rectifier, stitcher, and cylindrical projector comprise circuitry and/or executable software executed by an execution unit of the graphics processor.

### Claim 12

A method comprising:
receiving a first plurality of images from a corresponding first plurality of cameras;
performing a perspective re-projection of at least some of the first plurality of images to a common image plane to generate a rectified first plurality of images;
analyzing overlapping regions of adjacent images in the rectified first plurality and responsively identifying corresponding pixels in the overlapping regions;
stitching the adjacent images in accordance with the corresponding pixels to generate a panoramic image comprising a stitched combination of the rectified first plurality of rectified images; and
projecting the panoramic image onto a cylindrical surface to generate a final panoramic video image to be used to implement a virtual reality (VR) environment on a VR apparatus.

### Claim 13 (depends on 12)

The method of claim 12 wherein the overlapping regions are analyzed by performing a sequence of Belief Propagation operations.

### Claim 14 (depends on 13)

The method of claim 13 wherein the sequence of Belief Propagation operations comprise:
construction of an initial data cost volume using pixel data in the overlapping regions;
construction of a data cost pyramid based on the initial data cost volume comprising a series of smaller volumes;
iteration through the series of smaller volumes and the initial volume using Belief Propagation message passing to generate a final set of costs; and
constructing a stitch map from the final set of costs.

### Claim 15 (depends on 14)

The method of claim 14 wherein stitching comprises stitching the adjacent images in accordance with the final set of costs.

### Claim 16 (depends on 15)

The method of claim 15 wherein for each pixel in the overlapping region the stitch map is used to determine which pixels to blend.

### Claim 17 (depends on 16)

The method of claim 16 wherein a convex linear combination of pixels is selected to blend from each image.

### Claim 18 (depends on 12, 17)

The method of claim 12 or 17 further comprising:
storing stitching parameters determined from one or more prior images, and
stitching the adjacent images in the rectified first plurality of images using at least a portion of the stitching parameters.

### Claim 19 (depends on 12, 18)

The method of claim 12 or 18 wherein the perspective re-projection comprises a homography transformation.

### Claim 20 (depends on 12, 19)

The method of claim 12 or 19 wherein the first plurality of images comprise left images to be displayed, following processing, in a left display of the VR apparatus.

### Claim 21 (depends on 20)

The method of claim 20 wherein the video interface is to receive a right plurality of images from a corresponding second plurality of cameras, the method further comprising:
performing a perspective re-projection of at least some of the right plurality of images to a common image plane to generate rectified right plurality of images;
analyzing overlapping regions of adjacent images in the rectified right plurality and identifying corresponding pixels in the overlapping regions; and
stitching the adjacent images in accordance with the corresponding pixels to generate a panoramic right eye image comprising a stitched combination of the rectified right plurality of images; and
projecting the panoramic right eye image onto a cylindrical surface to generate a final panoramic right eye image to be combined with the final panoramic left eye image to implement a virtual reality (VR) environment on a VR apparatus.

### Claim 22

A machine-readable medium having program code stored thereon which, when executed by a machine, causes the machine to perform the operations of:
receiving a first plurality of images from a corresponding first plurality of cameras;
performing a perspective re-projection of at least some of the first plurality of images to a common image plane to generate a rectified first plurality of images;
analyzing overlapping regions of adjacent images in the rectified first plurality and responsively identifying corresponding pixels in the overlapping regions;
stitching the adjacent images in accordance with the corresponding pixels to generate a panoramic image comprising a stitched combination of the rectified first plurality of rectified images; and
projecting the panoramic image onto a cylindrical surface to generate a final panoramic video image to be used to implement a virtual reality (VR) environment on a VR apparatus.

### Claim 23 (depends on 22)

The machine-readable medium of claim 22 wherein the overlapping regions are analyzed by performing a sequence of Belief Propagation operations.

### Claim 24 (depends on 23)

The machine-readable medium of claim 23 wherein the sequence of Belief Propagation operations comprise:
construction of an initial data cost volume using pixel data in the overlapping regions;
construction of a data cost pyramid based on the initial data cost volume comprising a series of smaller volumes;
iteration through the series of smaller volumes and the initial volume using Belief Propagation message passing to generate a final set of costs; and
constructing a stitch map from the final set of costs.

### Claim 25 (depends on 24)

The machine-readable medium of claim 24 wherein stitching comprises stitching the adjacent images in accordance with the final set of costs.

### Claim at position 26

-31. (canceled)
