Bidirectional optical flow estimation method and apparatus

A bidirectional optical flow estimation device comprising:

• a processor;
• and
• a memory,
• wherein the memory stores an application program executable by the processor to cause the processor to:
• acquire a target image pair of which optical flow is to be estimated,
• construct an image pyramid for each target image in the target image pair, and
• perform bidirectional optical flow estimation using a pre-trained optical flow estimation model based on the image pyramid for each target image to obtain bidirectional optical flow between the target images of the target image pair,
• wherein an optical flow estimation module in the optical flow estimation model is recursively called to perform the bidirectional optical flow estimation sequentially based on images of respective layers in the image pyramid for each target image according to a preset order,
• wherein forward warping towards middle processing is performed on an image of a corresponding layer of the image pyramid before each call of the optical flow estimation module, and
• wherein an image of an intermediate frame obtained by the forward warping towards middle processing is inputted into the optical flow estimation module,
• wherein the memory stores an application program configured for the performing of bidirectional optical flow estimation that, when executed by the processor, causes the processor to:
• perform feature extraction on the image of the intermediate frame inputted to the optical flow estimation module using a convolutional neural network (CNN) feature extractor, to obtain a CNN feature of the image of the intermediate frame;
• determine a corresponding cost volume based on the CNN feature of the image of the intermediate frame;
• perform channel stacking using the CNN feature of the image of the intermediate frame, the corresponding cost volume, bidirectional optical flow outputted from a previous optical flow estimation, and a CNN feature of the bidirectional optical flow outputted from the previous optical flow estimation;
• input a channel stacking result into an optical flow estimation network to perform optical flow estimation;
• perform up-sampling on bidirectional optical flow obtained by the optical flow estimation and a CNN feature of the bidirectional optical flow respectively;
• and
• output an up-sampling result.

A non-transitory computer-readable storage medium, storing computer-readable instructions for performing a bidirectional optical flow estimation method comprising:

• acquiring a target image pair of which optical flow is to be estimated;
• constructing an image pyramid for each target image in the target image pair;
• and
• performing bidirectional optical flow estimation using a pre-trained optical flow estimation model based on the image pyramid for each target image, to obtain bidirectional optical flow between the target images of the target image pair,
• wherein an optical flow estimation module in the optical flow estimation model is recursively called to perform the bidirectional optical flow estimation sequentially based on images of respective layers in the image pyramid for each target image according to a preset order,
• wherein forward warping towards middle processing is performed on an image of a corresponding layer of the image pyramid before each call of the optical flow estimation module, and
• wherein an image of an intermediate frame obtained by the forward warping towards middle processing is inputted into the optical flow estimation module,
• wherein the performing of the bidirectional optical flow estimation by each call of the optical flow estimation module comprises:
• performing feature extraction on the image of the intermediate frame inputted to the optical flow estimation module using a convolutional neural network (CNN) feature extractor, to obtain a CNN feature of the image of the intermediate frame;
• determining a corresponding cost volume based on the CNN feature of the image of the intermediate frame;
• performing channel stacking using the CNN feature of the image of the intermediate frame, the corresponding cost volume, bidirectional optical flow outputted from a previous optical flow estimation, and a CNN feature of the bidirectional optical flow outputted from the previous optical flow estimation;
• inputting a channel stacking result into an optical flow estimation network to perform optical flow estimation;
• performing up-sampling on bidirectional optical flow obtained by the optical flow estimation and a CNN feature of the bidirectional optical flow respectively;
• and
• outputting an up-sampling result.