# US 10,455,221 B2 · Stereo viewing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,455,221 B2
- Document ID: US10455221B2
- Status: Granted
- Kind: B2
- Application number: 15/302,123
- Filed: 2015-03-19
- Published: 2017-04-27
- Granted: 2019-10-22
- Earliest claimed: 2014-04-07
- Assignees: Nokia Technologies Oy
- Applicants: Nokia Technologies Oy
- Inventors: Kim Gronholm; Andrew Robert Baldwin
- Web page: https://patentgrep.com/patent/US10455221B2

## CPC

- H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 30/34: Stereoscopes providing a stereoscopic pair of separated images corresponding to parallactically displaced views of the same object, e.g. three-dimensional [3D] slide viewers
- G03B 35/08: by simultaneous recording
- G03B 37/04: with cameras or projectors providing touching or overlapping fields of view
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/16: Sound input; Sound output (speech processing G10L)
- H04N 13/156: Mixing image signals
- H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597)
- H04N 13/189: Recording image signals; Reproducing recorded image signals
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/257: Colour aspects
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 13/296: Synchronisation thereof; Control thereof
- H04N 13/324: Colour aspects
- H04N 13/344: with head-mounted left-right displays
- H04N 13/366: using viewer tracking
- H04N 13/378: for tracking rotational head movements around an axis perpendicular to the screen
- H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
- H04N 19/124: Quantisation
- H04N 19/162: User input
- H04N 19/167: Position within a video image, e.g. region of interest [ROI]
- H04N 19/17: the unit being an image region, e.g. an object
- H04N 19/597: specially adapted for multi-view video sequence encoding
- G02B 2027/0134: {of stereoscopic type}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G03B 35/18: by simultaneous viewing
- H04N 2213/008: Aspects relating to glasses for viewing stereoscopic images
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The invention relates to creating and viewing stereo images, for example stereo video images, also called 3D video. At least three camera sources with overlapping fields of view are used to capture a scene so that an area of the scene is covered by at least three cameras. At the viewer, a camera pair is chosen from the multiple cameras to create a stereo camera pair that best matches the location of the eyes of the user if they were located at the place of the camera sources. That is, a camera pair is chosen so that the disparity created by the camera sources resembles the disparity that the user's eyes would have at that location. If the user tilts his head, or the view orientation is otherwise altered, a new pair can be formed, for example by switching the other camera. The viewer device then forms the images of the video frames for the left and right eyes by picking the best sources for each area of each image for realistic stereo disparity.

## Claims

### Claim 1

A method, comprising:
decoding both active and passive source video signals that have been differently encoded according to a viewing angle of a user with a visible region that was captured by one or more active image sources being encoded in higher quality than one or more regions not currently visible that were captured by one or more passive image sources,
determining head orientation of a user to obtain a first head orientation,
selecting a first image source, a second image source and a third image source, from three or more image sources, based on said first head orientation, said first and second image source forming a stereo image source,
rendering a first target image for one eye of the user using decoded video signals from said first image source and a second target image for another eye of the user using decoded video signals from said second image source,
determining head orientation of said user to obtain a second head orientation,
selecting said second image source and the third image source based on said second head orientation, said second and third image source forming a stereo image source,
rendering a third target image for one eye of the user using decoded video signals from said second image source and a fourth target image for another eye of the user using decoded video signals from said third image source,
stitching at least a part of an image from the third image source with at least one of the first target image and the second target image, and
stitching at least a part of an image from the third image source with at least one of the third target image and the fourth target image.

### Claim 2

An apparatus comprising at least one processor, and memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least:
decode both active and passive source video signals that have been differently encoded according to a viewing angle of a user with a visible region that was captured by one or more active image sources being encoded in higher quality than one or more regions not currently visible that were captured by one or more passive image sources,
determine head orientation of a user to obtain a first head orientation,
select a first image source, a second image source and a third image source, from three or more image sources, based on said first head orientation, said first and second image source forming a stereo image source,
render a first target image for one eye of the user using decoded video signals from said first image source and a second target image for another eye of the user using decoded video signals from said second image source,
determine head orientation of said user to obtain a second head orientation,
select said second image source and the third image source based on said second head orientation, said second and third image source forming a stereo image source,
render a third target image for one eye of the user using decoded video signals from said second image source and a fourth target image for another eye of the user using decoded video signals from said third image source,
stitch at least a part of an image from the third image source with at least one of the first target image and the second target image, and
stitch at least a part of an image from the third image source with at least one of the third target image and the fourth target image.

### Claim 3 (depends on 2)

An apparatus according to claim 2, comprising computer program code to cause the apparatus to:
form color values of a first region of pixels corresponding to a first region of a scene using said first image source, said color values of said first region of pixels being used for forming said first target image for displaying to a first eye,
form color values of a second region of pixels corresponding to said first region of a scene using said second image source, said color values of said second region of pixels being used for forming said second target image for displaying to a second eye,
form color values of a third region of pixels corresponding to said first region of a scene using said third image source, said color values of said third region of pixels being used for forming said third target image for displaying to said first eye,
form color values of a fourth region of pixels corresponding to said first region of a scene using said second image source, said color values of said fourth region of pixels being used for forming said fourth target image for displaying to said second eye.

### Claim 4 (depends on 3)

An apparatus according to claim 3, comprising computer program code to cause the apparatus to:
determine locations of a first and a second virtual eye corresponding to said eyes of the user in a coordinate system using said head orientations,
select said image sources based on said locations of said virtual eyes with respect to image source locations in said coordinate system.

### Claim 5 (depends on 4)

An apparatus according to claim 4, comprising computer program code to cause the apparatus to:
determine an image source for a pixel of an image for a first eye of the user to be a close image source that satisfies a closeness criterion to a virtual eye corresponding to said first eye, where said close image source captures the scene portion corresponding to the pixel, and
determine an image source for a pixel of an image for said first eye of the user to be another source than the close image source to said virtual eye corresponding to said first eye, where the close image source does not capture the scene portion corresponding to the pixel.

### Claim 6 (depends on 2)

An apparatus according to claim 2, comprising computer program code to cause the apparatus to:
render regions of an image for an eye so that the regions correspond to image sources, wherein the regions are rendered in order of closeness of the image sources to a virtual eye corresponding to said eye in image source coordinate system.

### Claim 7 (depends on 6)

An apparatus according to claim 6, comprising computer program code to cause the apparatus to:
combine the regions of said image together at their edges by using an image stitching technique.

### Claim 8 (depends on 7)

An apparatus according to claim 7, comprising computer program code to cause the apparatus to:
combine said regions of said images by blending the edge areas of the regions.

### Claim 9 (depends on 2)

An apparatus according to claim 2, comprising computer program code to cause the apparatus to:
blend a temporal transition from said image formed using said first image source and said image using said third image source.

### Claim 10 (depends on 9)

An apparatus according to claim 9, comprising computer program code to cause the apparatus to:
adjust the duration of the temporal transition blending by using information on head movement speed.

### Claim 11 (depends on 2)

An apparatus according to claim 2, comprising computer program code to cause the apparatus to:
determine audio information for the left ear and audio information for the right ear using said head orientation information to modify audio information from two or more audio sources using a head-related transfer function.

### Claim 12 (depends on 2)

An apparatus according to claim 2, comprising computer program code to cause the apparatus to:
determine source orientation information for said image sources, and
use said source orientation information together with said head orientation information for selecting said image sources.

### Claim 13 (depends on 11)

An apparatus according to claim 11, comprising computer program code to cause the apparatus to:
determine source orientation information for said audio sources, and
use said source orientation information together with said head orientation information for modifying audio information from said audio sources.

### Claim 14 (depends on 2)

An apparatus according to claim 2, wherein said image sources are real-world cameras.

### Claim 15 (depends on 2)

An apparatus according to claim 2, wherein said image sources are formed as an output of a computer device using virtual cameras for producing rendered synthetic images.

### Claim 16 (depends on 2)

An apparatus according to claim 2, wherein said head orientation is related to physical orientation of the user's head.

### Claim 17 (depends on 2)

An apparatus according to claim 2, wherein said head orientation is related to virtual orientation of the user's view direction as controlled by a computer program or by a computer input device.

### Claim 18 (depends on 2)

An apparatus according to claim 2, wherein said stereo images are used to form a stereo video sequence.

### Claim 19

An apparatus comprising at least one processor, and memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least:
decode both active and passive source video signals that have been differently encoded according to a viewing angle of a user with a visible region that was captured by one or more active image sources being encoded in higher quality than one or more regions not currently visible that were captured by one or more passive image sources,
determine head orientations of a user for forming a stereo video sequence of a scene;
select a first image source, a second image source and a third image source, from three or more image sources, based on said head orientations;
render said stereo video sequence by rendering an image sequence for the left eye of the user using decoded video signals from said first image source and said second image source and an image sequence for the right eye of the user using decoded video signals from said second image source and said third image source, wherein said second image source is used to render different areas of said scene for the left and right eyes of the user in each stereo frame of said video sequence, wherein the stereo video sequence is rendered by rendering a first target image for the left eye of the user using decoded video signals from said first image source and a second target image for the right eye of the user using decoded video signals from said second image source and rendering a third target image for the left eye of the user using decoded video signals from said second image source and a fourth target image for the right eye of the user using decoded video signals from said third image source;
stitch at least a part of an image from the third image source with at least one of the first target image and the second target image, and
stitch at least a part of an image from the third image source with at least one of the third target image and the fourth target image.

### Claim 20

A computer program product embodied on a non-transitory computer readable medium, comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to:
decode both active and passive source video signals that have been differently encoded according to a viewing angle of a user with a visible region that was captured by one or more active image sources being encoded in higher quality than one or more regions not currently visible that were captured by one or more passive image sources,
determine head orientation of a user to obtain a first head orientation,
select a first image source, a second image source and a third image source based on said first head orientation, said first and second image source forming a stereo image source,
render a first target image for one eye of the user using decoded video signals from said first image source and a second target image for another eye of the user using decoded video signals from said second image source,
determine head orientation of said user to obtain a second head orientation,
select said second image source and the third image source based on said second head orientation, said second and third image source forming a stereo image source,
render a third target image for one eye of the user using decoded video signals from said second image source and a fourth target image for another eye of the user using decoded video signals from said third image source,
stitch at least a part of an image from the third image source with at least one of the first target image and the second target image, and
stitch at least a part of an image from the third image source with at least one of the third target image and the fourth target image.