Method for performing digital image stabilization

A method for performing digital image stabilization on a sequence of video frames captured by an image capturing device, the image capturing device comprising:

• a motion sensor configured to output first and second motion signal indicating motion of the image capturing device along first and second sensing axes of the motion sensor, respectively, an optical image stabilization (OIS) system configured to compensate for motion of the image capturing device by controlling a position of a movable element of the OIS system based on the first and second motion signals, and a position sensor configured to output a first and second position signal indicating an instantaneous position of the movable element along a first and second sensing axis of the position sensor, respectively, the method comprising:
• obtaining a sequence of motion vectors, each motion vector including first and second motion values sampled from the first and second motion signals, respectively, while capturing the sequence of video frames, such that each motion vector is associated with a respective video frame;
• obtaining a sequence of position vectors, each position vector including first and second position values sampled from the first and second position signals, respectively, while capturing the sequence of video frames, such that each position vector is associated with a respective video frame;
• determining, for each respective video frame, a residual vector to obtain a sequence of residual vectors for the sequence of video frames, wherein each residual vector is determined based on the motion vector and the position vector associated with the respective video frame such that each residual vector includes a first residual motion value based on the first motion and position values of the motion and position vectors, and a second residual motion value based on the second motion and position values of the motion and position vectors, and indicates a residual motion of the image capturing device upon capturing the respective video frame, not compensated for by the OIS system;
• and
• performing digital image stabilization on the sequence of video frames, comprising:
• detecting an image feature in a reference video frame of the sequence of video frames;
• tracking the image feature in each further video frame of the sequence of video frames;
• determining frame motion data based on a displacement for the image feature in each further video frame in the sequence of video frames;
• and
• generating a stabilized sequence of video frames based on the frame motion data;
• wherein first and second dimensions of a tracking window for tracking the image feature in the further video frames are based on the first and second residual motion values, respectively, of the residual vectors of the sequence of residual vectors.

An image capturing device comprising:

• a motion sensor configured to output a motion signal indicating motion of the image capturing device;
• an optical image stabilization (OIS) system configured to compensate for motion of the image capturing device by controlling a position of a movable element of the OIS system based on the motion signal;
• a position sensor configured to output a position signal indicating a position of the movable element;
• and
• a processing device configured to perform a method for digital image stabilization on a sequence of video frames captured by the image capturing device, the method comprising:
• obtaining a sequence of motion values sampled from the motion signal while capturing the sequence of video frames;
• obtaining a sequence of position values sampled from the position signal while capturing the sequence of video frames, wherein each position value corresponds to a respective motion value of the sequence of motion values, such that the sequences of motion values and position values define a sequence of pairs of motion and position values;
• wherein a sampling rate of the sequences of motion and position values exceed a frame rate of the sequence of video frames, such that each respective video frame is associated with a respective subset of pairs of motion and position values, each pair of motion and position values being associated with a respective subset of pixel rows of the respective video frame, and the method further comprising:
• determining, for each respective video frame, a subset of residual motion values, wherein each residual motion value of the respective subset of residual motion values associated with a respective video frame is determined based on the motion value and the position value of the pair of motion and position values associated with the respective subset of pixel rows of the respective video frame, such that the residual motion value is associated with the respective subset of pixel rows and indicates a residual motion of the image capturing device upon capturing the respective subset of pixel rows of the respective video frame, not compensated for by the OIS system;
• and
• performing digital image stabilization on the sequence of video frames, comprising:
• detecting an image feature in a reference video frame of the sequence of video frames;
• tracking the image feature in each further video frame of the sequence of video frames;
• determining frame motion data based on a displacement for the image feature in each further video frame in the sequence of video frames;
• and
• generating a stabilized sequence of video frames based on the frame motion data;
• wherein a size of a tracking window for tracking the image feature in a given further video frame is based on a representative residual motion value of the subset of residual motion values associated with the given further video frame.