System and method for AI segmentation-based registration for multi-frame processing

An electronic device comprising:

• at least one imaging sensor configured to capture multiple image frames of a scene;
• and
• at least one processing device configured to:
• obtain a reference frame from among the multiple image frames;
• generate a segmentation mask using the reference frame, the segmentation mask containing information for separation of foreground and background in the scene;
• apply the segmentation mask to each of the multiple image frames to generate multiple foreground image frames and multiple background image frames;
• perform multi-frame registration on each of the multiple foreground image frames to the foreground in the reference frame to generate registered foreground image frames by determining a first homography matrix for each of the multiple foreground image frames based on each tile center within the reference frame and matching features in each of the multiple foreground image frames;
• perform multi-frame registration on each of the multiple background image frames to the background in the reference frame to generate registered background image frames by determining a second homography matrix for each of the multiple background image frames based on each tile center within the reference frame and matching features in each of the multiple background image frames;
• and
• combine the registered foreground image frames and the registered background image frames to generate a combined registered multi-frame image of the scene;
• wherein the at least one processing device is configured to
• combine the registered foreground image frames and the registered background image frames by calculating first and second displacement matrices for the registered foreground image frames and the registered background image frames, respectively;
• and
• use the first and second displacement matrices to either (i) resize the segmentation mask or (ii) separately warp the registered foreground image frames and the registered background image frames.

A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:

• obtain a reference frame from among multiple image frames of a scene;
• generate a segmentation mask using the reference frame, the segmentation mask containing information for separation of foreground and background in the scene;
• apply the segmentation mask to each of the multiple image frames to generate multiple foreground image frames and multiple background image frames;
• perform multi-frame registration on each of the multiple foreground image frames to the foreground in the reference frame to generate registered foreground image frames by determining a first homography matrix for each of the multiple foreground image frames based on each tile center within the reference frame and matching features in each of the multiple foreground image frames;
• perform multi-frame registration on each of the multiple background image frames to the background in the reference frame to generate registered background image frames by determining a second homography matrix for each of the multiple background image frames based on each tile center within the reference frame and matching features in each of the multiple background image frames;
• and
• combine the registered foreground image frames and the registered background image frames to generate a combined registered multi-frame image of the scene;
• wherein the instructions when executed cause the at least one processor to
• combine the registered foreground image frames and the registered background image frames by calculating first and second displacement matrices for the registered foreground image frames and the registered background image frames, respectively;
• and
• use the first and second displacement matrices to either (i) resize the segmentation mask or (ii) separately warp the registered foreground image frames and the registered background image frames.