US 5,557,684 A · System for encoding image data into multiple layers representing regions of coherent motion and associated motion parameters

Abstract

A system stores images as a series of layers by determining (i) the boundaries of regions of coherent motion over the entire image, or frame, sequence; and (ii) associated motion parameters, or coefficients of motion equations, that describe the transformations of the regions from frame to frame. The system first estimates motion locally, by determining the movements within small neighborhoods of pixels from one image frame i to the next image frame i+1, to develop an optical flow, or dense motion, model of the image. Next, the system estimates the motion using affine or other low order, smooth transformations within a set of regions which the system has previously identified as having coherent motion, i.e., identified by analyzing the motions in the frames i-1 and i. It groups, or clusters, similar motion models and iteratively produces an updated set of models for the image. The system then uses the local motion estimates to associate individual pixels in the image with the motion model that most closely resembles the pixel's movement, to update the regions of coherent motion. Using these updated regions, the system iteratively updates its motion models and, as appropriate, further updates the coherent motion regions, and so forth. The system then does the same analysis for the remaining frames. The system next segments the image into regions of coherent motion and defines associated layers in terms of (i) pixel intensity values, (ii) associated motion model parameters, and (iii) order in "depth" within the image.

Claims29

Dependent claims

A computer system for encoding digital image data relating to a sequence of image frames to represent the sequence of image frames as digital data organized as a plurality of ordered layers, the layers including information to be used to reproduce and display the same or an associated sequence of image frames, the system including:

A. a local motion estimator, for producing local motion estimates for pixel neighborhoods in sequential frames and producing from these estimates associated dense motion models, each of the pixel neighborhoods including a selected pixel and the pixels adjacent to the selected pixel;
B. a motion segmentation processor, for determining, jointly, from the dense motion models
i. regions of coherent motion over the entire sequence of frames, and
ii. for each region of coherent motion a motion model representing the transformation of the region from frame to frame for the sequence of frames;
C. a layer extraction processor, for forming for each of the regions of coherent motion a layer by associating with the region data that represents the associated image data, the layer including
i. data representing pixel intensity information relating to the sequence of frames for each of the pixels included in the region,
ii. data representing an ordering of the layer relative to other layers by depth in the image, the layer extraction processor determining the depth of a layer by determining if pixels in the layer are occluded in any of the frames in the sequence;
iii. data representing motion information including parameters relating to the motion model associated with the region by the motion segmentation processor, and
D. means for outputting the digital data comprising the layers to reproduce the same or an associated sequence of frames.

6 more elements

A method of determining regions of coherent motion and associated motions in an image sequence which consists of a plurality of frames and using data associated with the regions of coherent motion to reproduce the same or an associated image sequence, the method comprising the steps of:

A. estimating local motion of image pixels from frame to frame by determining motion estimates for selected pixels and pixel neighborhoods of the selected pixels;
B. using the local motion estimates to determine iteratively regions of coherent motion in sequential frames and updating these regions of coherent motion for the plurality of frames;
and
C. for each region of coherent motion determining related motion parameters associated with frame to frame motion, based on the local motion estimates, and updating the motion parameters for updated regions of coherent motion;
D. segmenting the image into regions of coherent motion by assigning each of the pixels to one or more regions of coherent motion associated with the motion models that most closely represent the local motions of the pixels;
E. outputting data representing the image sequence as a plurality of regions of coherent motion to reproduce a display of the same or an associated image sequence.

3 more elements

Dependencies are inferred from the claim text.

System for encoding image data into multiple layers representing regions of coherent motion and associated motion parameters

Life of this patent

System for encoding image data into multiple layers representing regions of coherent motion and associated motion parameters

Life of this patent

Abstract

Claims29