US 11,599,968 B2Granted

Apparatus, a method and a computer program for volumetric video

AssigneeNokia Technologies Oy

InventorSebastian Schwarz

Life of this patent

Estimating the term…

Loading patent…

Abstract

Embodiments for volumetric video encoding and decoding relating to one or more three-dimensional objects are disclosed. In encoding, after mapping from 3D space to 2D plane (802) a point in the 2D plane is examined (805) to determine which points of the 3D object are mapped to the same point to obtain a set of candidate points. Candidate points belonging to a same surface can be used to determine a center of mass for the surface (807). A depth value of the centre of mass is mapped to a 2D projection depth plane (808). A colour value for the centre of mass is interpolated from colour values of points of the set of surface points which are nearest neighbours of the center of mass (810), and used as the colour of the surface in the texture plane (812). Corresponding embodiments for decoding are provided.

Claims19

Dependent claims

A method comprising:

receiving coded texture data representing a texture image of a three-dimensional object in volumetric video data, wherein the coded texture data comprises an interpolated colour value for a same surface of the three-dimensional object, wherein the interpolated colour value comprises a color value interpolated for a determined centre of mass for the same surface;
receiving coded depth data representing a depth image of the three-dimensional object in the volumetric video data;
selecting a first pixel from one of the texture image or the depth image;
selecting a second pixel from one of the texture image or the depth image;
comparing coded depth data of the first pixel with coded depth data of the second pixel to determine whether the first pixel and the second pixel belong to the same surface of the three-dimensional object, wherein the coded depth data of the first pixel comprises a first depth value from the depth image, wherein the coded depth data of the second pixel comprises a second depth value from the depth image;
determining the first pixel and the second pixel belong to the same surface of the three-dimensional object in response to the coded depth data of the first pixel and the coded depth data of the second pixel being the same or almost the same;
upsampling coordinates of the first pixel to obtain a first point in a three-dimensional space;
upsampling coordinates of the second pixel to obtain a second point in the three-dimensional space;
and
reconstructing pixels between the first point and the second point in the three-dimensional space, comprising:
interpolating texture values of the first pixel and the second pixel in the texture image based, at least partially, on the interpolated colour value for the same surface of the three-dimensional object;
and
interpolating depth values of the first pixel and the second pixel in the depth image.

10 more elements

An apparatus comprising:

at least one processor and
at least one non-transitory memory,
said at least one memory stored with code thereon, which when executed by said at least one processor, causes the apparatus to perform at least:
receive coded texture data representing a texture image of a three-dimensional object in volumetric video data, wherein the coded texture data comprises an interpolated colour value for a same surface of the three-dimensional object, wherein the interpolated colour value comprises
a color value interpolated for a determined centre of mass for the same surface;
receive coded depth data representing a depth image of the three-dimensional object in the volumetric video data;
select a first pixel from one of the texture image or the depth image;
select a second pixel from one of the texture image or the depth image;
compare coded depth data of the first pixel with coded depth data of the second pixel to determine whether the first pixel and the second pixel belong to the same surface of the three-dimensional object, wherein the coded depth data of the first pixel comprises a first depth value from the depth image, wherein the coded depth data of the second pixel comprises a second depth value from the depth image;
determine the first pixel and the second pixel belong to the same surface of the three-dimensional object in response to the coded depth data of the first pixel and the coded depth data of the second pixel being the same or almost the same;
upsample coordinates of the first pixel to obtain a first point in a three-dimensional space;
upsample coordinates of the second pixel to obtain a second point in the three-dimensional space;
and
reconstruct pixels between the first point and the second point in the three-dimensional space, comprising:
interpolating texture values of the first pixel and the second pixel in the texture image based, at least partially, on the interpolated colour value for the same surface of the three-dimensional object;
and
interpolating depth values of the first pixel and the second pixel in the depth image.

14 more elements

A non-transitory computer-readable medium comprising program instructions stored thereon which, when executed with at least one processor, cause the at least one processor to:

receive coded texture data representing a texture image of a three-dimensional object in volumetric video data, wherein the coded texture data comprises an interpolated colour value for a same surface of the three-dimensional object, wherein the interpolated colour value comprises a color value interpolated for a determined centre of mass for the same surface;
receive coded depth data representing a depth image of the three-dimensional object in the volumetric video data;
cause selection of a first pixel from one of the texture image or the depth image;
cause selection of a second pixel from one of the texture image or the depth image;
compare coded depth data of the first pixel with coded depth data of the second pixel to determine whether the first pixel and the second pixel belong to the same surface of the three-dimensional object, wherein the coded depth data of the first pixel comprises a first depth value from the depth image, wherein the coded depth data of the second pixel comprises a second depth value from the depth image;
determine the first pixel and the second pixel belong to the same surface of the three-dimensional object in response to the coded depth data of the first pixel and the coded depth data of the second pixel being the same or almost the same;
cause upsampling of coordinates of the first pixel to obtain a first point in a three-dimensional space;
cause upsampling of coordinates of the second pixel to obtain a second point in the three-dimensional space;
and
cause reconstruction of pixels between the first point and the second point in the three-dimensional space, comprising:
cause interpolating of texture values of the first pixel and the second pixel in the texture image based, at least partially, on the interpolated colour value for the same surface of the three-dimensional object;
and
cause interpolating of depth values of the first pixel and the second pixel in the depth image.

10 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/754,715Filed 9 Oct 2018
Priority: FI 2017589311 Oct 2017
Published: US 2020/0302571 A124 Sep 2020
Granted: 7 Mar 2023
Term ends: Not estimatedThe source records don’t support an estimate.

Classification15 codes

G06T 3/06
Geometric image transformations in the plane of the image: topological mapping of higher dimensional structures onto lower dimensional surfaces
G06T 3/4007
Scaling of whole images or parts thereof, e.g. expanding or contracting: based on interpolation, e.g. bilinear interpolation
G06T 7/11
Segmentation; Edge detection: region-based segmentation
G06T 7/70
Image analysis: determining position or orientation of objects or cameras