Methods and processors for rendering a 3D object using multi-camera image inputs

A method rendering a three-dimensional (3D) object, the method executable by a processor, the method comprising:

• acquiring multi-camera image input including first image frames of the 3D object generated by a first camera and second image frames of the 3D object generated by a second camera, the first camera being different from the second camera;
• acquiring an initial 3D Gaussian Splatting (3DGS) model having a plurality of initial parameters, the plurality of initial parameters including an initial frame-wise Gaussian Splatting (GS) parameter and an initial camera-wise GS parameter;
• generating an adjusted 3DGS model by adjusting, based on the multi-camera image input, at least one of:
• (iii) the initial frame-wise GS parameter to compensate for misalignment between the first image frames and the second image frames;
• and
• (iv) the initial camera-wise GS parameter to compensate for cross-camera variation between the first camera and the second camera;
• wherein the adjusting the initial camera-wise GS parameter is executed in accordance with:
• SH
• d
• ′
• =
• β
• d
• ⁢
• SH
• +
• γ
• d
• ,
• SH
• r
• ′
• =
• β
• r
• ⁢
• SH
• +
• γ
• r
• ,
• wherein β is a scaling factor, γ is a bias factor, SH is the initial camera-wise GS SH parameter, SHd′ is a 0th band of an adjusted camera-wise GS SH parameter, SHr′ is a higher band of an adjusted camera-wise GS SH parameter;
• generating, by the adjusted 3DGS model, a 3DGS output based on the multi-camera image input;
• rendering a two-dimensional (2D) image of the 3D object using the 3DGS output.

A processor for rendering a three-dimensional (3D) object, the processing being configured to:

• acquire multi-camera image input including first image frames of the 3D object generated by a first camera and second image frames of the 3D object generated by a second camera, the first camera being different from the second camera;
• acquire an initial 3D Gaussian Splatting (3DGS) model having a plurality of initial parameters, the plurality of initial parameters including an initial frame-wise Gaussian Splatting (GS) parameter and an initial camera-wise GS parameter;
• generate an adjusted 3DGS model by adjusting, based on the multi-camera image input, at least one of:
• (v) the initial frame-wise GS parameter to compensate for misalignment between the first image frames and the second image frames;
• and
• (vi) the initial camera-wise GS parameter to compensate for cross-camera variation between the first camera and the second camera;
• wherein to adjust the initial frame-wise GS parameter is executed by the processor in accordance with:
• μ
• ′
• =
• M
• R
• ⁢
• μ
• +
• M
• T
• wherein μ is an initial frame-wise GS position parameter, μ′ is an adjusted frame-wise GS position parameter, MR is a rotation matrix, and MT is a translation matrix;
• generate, using the adjusted 3DGS model, a 3DGS output based on the multi-camera image input;
• and
• render a two-dimensional (2D) image of the 3D object using the 3DGS output.