Method and apparatus with multi-modal feature fusion

An apparatus comprising:

• one or more processors comprising processing circuitry;
• memory comprising one or more storage media storing instructions that, when executed by the one or more processors individually or collectively, cause the apparatus to:
• generate three-dimensional (3D) feature information based on a depth image;
• generate two-dimensional (2D) feature information based on a color image;
• fuse the 3D feature information and the 2D feature information using an attention mechanism to generate fused feature information;
• and
• perform at least one of an estimation of a six-dimensional (6D) pose of an object, an estimation of a size of the object, a reconstruction of a shape of the object, or a segmenting of the object based on the fused feature information,
• wherein the attention mechanism includes a self-attention mechanism and cross-attention mechanism,
• wherein the fused feature information is generated through a fusing of the 3D feature information of at least one scale and the 2D feature information of at least one scale, and
• wherein the generation of the fused feature information comprises, for the 3D feature information of one scale and the 2D feature information of one scale, generation of fused feature information of a current scale by performing a feature fusion on 3D feature information of the current scale and 2D feature information of the current scale based on the attention mechanism, the 3D feature information of the current scale being determined based on fused feature information of a previous scale and 3D feature information of the previous scale, and the 2D feature information of the current scale being determined based on 2D feature information of the previous scale.