Spatially adaptive video compression for multiple streams of color and depth

A computer program product comprising a nontransitive storage medium, the computer program product including code that, when executed by processing circuitry of a server computing device configured to compress video images of objects, causes the processing circuitry to perform a method, the method comprising:

• receiving (i) depth image data representing a plurality of depth images of an object, each of the plurality of depth images being a depth image of the object captured from a respective viewpoint, and (ii) color image data representing a plurality of color images, each of the plurality of color images (a) being a color image of the object captured from a respective viewpoint and (b) having a plurality of macroblocks, each of the macroblocks corresponding to a respective region of the color image;
• for a color image of the plurality of color images:
• identifying a depth image of the plurality of depth images associated with the color image;
• for each pixel of the depth image:
• determining a position of a surface of the object based on a depth value of the depth image associated with that pixel;
• generating a value of a combination parameter corresponding to the determined position, the value of the combination parameter being based on a surface normal of the depth image at the position and the viewpoints from which the plurality of color images were captured;
• generating a value of a delta quantization parameter based on the value of the combination parameter;
• and
• determining a macroblock of the plurality of macroblocks of the color image corresponding to the position, the macroblock being associated with the value of the delta quantization parameter for the pixel;
• and
• performing a compression operation on the color image, a compression ratio of each of the plurality of macroblocks of the color image being based on the value of the delta quantization parameter with which the macroblock is associated.

An electronic apparatus configured to compress video images of objects, the electronic apparatus comprising:

• memory;
• and
• controlling circuitry coupled to the memory, the controlling circuitry being configured to:
• receive (i) depth image data representing a plurality of depth images of an object, each of the plurality of depth images being a depth image of the object captured from a respective viewpoint, and (ii) color image data representing a plurality of color images, each of the plurality of color images (a) being a color image of the object captured from a respective viewpoint and (b) having a plurality of macroblocks, each of the macroblocks corresponding to a respective region of the color image;
• for a color image of the plurality of color images:
• identify a depth image of the plurality of depth images associated with the color image;
• for each pixel of the depth image:
• determine a position of a surface of the object based on a depth value of the depth image associated with that pixel;
• generate a value of a combination parameter corresponding to the determined position, the value of the combination parameter being based on a surface normal of the depth image at the position and the viewpoints from which the plurality of color images were captured;
• generate a value of a delta quantization parameter based on the value of the combination parameter;
• and
• determine a macroblock of the plurality of macroblocks of the color image corresponding to the position, the macroblock being associated with the value of the delta quantization parameter for the pixel;
• and
• perform a compression operation on the color image, a compression ratio of each of the plurality of macroblocks of the color image being based on the value of the delta quantization parameter with which the macroblock is associated.