# US 7,436,437 B2 · Method, device and computer program product for image stabilization using color matching > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 7,436,437 B2 - Document ID: US7436437B2 - Status: Granted - Kind: B2 - Application number: 10/669,818 - Filed: 2003-09-24 - Published: 2004-04-01 - Granted: 2008-10-14 - Earliest claimed: 2001-09-07 - Assignees: Intergraph Software Technologies Company - Inventors: Susan Heath Calvin Fletcher; Gene Arthur Grindstaff; Therman Ward McKay, III - Web page: https://patentgrep.com/patent/US7436437B2 ## CPC - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/33: using feature-based methods - G06V 10/20: Image preprocessing - H04N 7/181: {for receiving images from a plurality of remote sources} - H04N 9/646: {for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters} - H04N 23/68: for stable pick-up of the scene, e.g. compensating for camera body vibrations - H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory} - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method for image stabilization of at least two digital images is disclosed. In such an embodiment, a first digital image having a plurality of pixels and at least a second image having a plurality of pixels are provided. Each pixel has an associated address for display and is representative of a color. Either a user of the system provides a color match range or the system provides a predetermined color matched range. A pixel is selected within the first digital image. Preferably the pixel is one that represents an item within the image that is either blurry due to movement of the item or appears jittery due to camera movement. Due to the vast number of available colors a pixel selected in a first image can be matched to a pixel in a second image within a range. The range allows for compensation due to lighting changes. Once the pixel in the first image is selected it can be compared to all pixels within the second image. Each pixel within the second image that is within the color match range is saved and the closest pixel color to the pixel from the first image is selected. The address of the pixels in the second image are then readdressed such that the address of the pixel located in the second image that has the closest color to the pixel in the first image now has the same display address as that of the pixel in the first image. The repositioned second digital image is then stored to memory. ## Claims ### Claim 1 A method of processing a motion picture having first and second frames, each frame having a plurality of pixels, the method comprising: selecting a first pixel in the first frame having a given color; locating a second pixel in the second frame having the given color; aligning the first frame and the second frame based upon the locations of the first and second pixels. ### Claim 2 (depends on 1) The method as defined by claim 1 wherein the first and second frames are aligned in memory. ### Claim 3 (depends on 1) The method as defined by claim 1 wherein aligning comprises aligning the first frame and second frame about the first and second pixels. ### Claim 4 (depends on 1) The method as defined by claim 1 further comprising determining the color of the first pixel, wherein the color is determined to be the given color. ### Claim 5 (depends on 1) The method as defined by claim 1 wherein locating composes determining if the color of the second pixel is within a given color tolerance. ### Claim 6 (depends on 1) The method as defined by claim 1 wherein the first pixel has a first address, further wherein aligning includes rearranging the second pixel to have an address that equals the first address. ### Claim 7 (depends on 1) The method as defined by claim 1 wherein the first frame and the second frame are linearly averaged to provide a third frame. ### Claim 8 (depends on 7) The method as defined by claim 7, wherein the third frame has a higher resolution than the first or second frame. ### Claim 9 (depends on 7) The method as defined by claim 7, wherein the third frame is displayed on a display device. ### Claim 10 (depends on 7) The method according to claim 7 wherein the first and second frames are aligned to a sub-pixel level. ### Claim 11 (depends on 1) The method according to claim 1 further comprising: selecting a region of the second frame for locating a second pixel having the given color within that region. ### Claim 12 (depends on 11) The method according to claim 11, further comprising: if a second pixel is not found within the second frame having the given color, aligning the first frame and the second frame based upon a predetermined formula. ### Claim 13 (depends on 12) The method according to claim 12 wherein the predetermined formula is based upon past alignments of frames. ### Claim 14 A method of processing a motion picture having a plurality of frames, each frame having a plurality of pixels, the method comprising: selecting an initial frame and a final frame, at least one frame being between the initial and final frames; selecting an initial pixel in the initial frame and a final pixel in the final frame, the initial pixel having an initial color, the final pixel having a final color; determining an interpolated pixel for each frame between the initial and final frame based on the initial color and the final color; and aligning the initial frame, final frame, and the at least one frame between the initial and final frames, the frames being aligned based upon the locations of the initial pixel, final pixel, and each interpolated pixel. ### Claim 15 (depends on 14) The method as defined by claim 14 wherein the interpolation is based upon the initial color, the final color, and the total number of frames between the initial and final frames. ### Claim 16 (depends on 14) The method as defined by claim 14 further comprising locating the interpolated pixel in a given frame between the initial and final frames. ### Claim 17 (depends on 16) The method as defined by claim 16 wherein each interpolated pixel has an interpolated color, further wherein the interpolated pixel in the given frame is located based upon the interpolated color of the interpolated pixel. ### Claim 18 (depends on 17) The method as defined by claim 17 wherein locating comprises determining if the interpolated color is within a given color tolerance. ### Claim 19 (depends on 14) The method as defined by claim 14 wherein the initial frame, final frame, and the at least one frame are aligned in memory. ### Claim 20 (depends on 14) The method as defined by claim 14 wherein aligning comprises setting the address of the initial pixel, final pixel and interpolated pixel to the same value. ### Claim 21 (depends on 14) The method according to claim 14 further comprising displaying at least the initial frame, final frame, and the at least one frame between the initial and final frames on a display. ### Claim 22 A method for image stabilization of a digital sequence of images, wherein each image has a plurality of pixels and wherein each pixel has an associated address for display and is representative of a color, the method comprising: selecting a pixel in the beginning image and a pixel in the ending image; interpolating between the colors associated with the selected pixels from the beginning image and the ending image; in the step of interpolating determining a color value for each image between the beginning image and the ending image; locating within each image a pixel that is within the range of the determined color; and readdressing each image so that the located pixels are aligned at the same display address. ### Claim 23 A method for image stabilization of a sequence of temporally displayed digital video images, wherein each image contains a number of pixels and each pixel has an associated display position, the method comprising: receiving a signal representative of a color of a pixel in a first image; identifying a pixel in a second image having the same color within a predetermined range of colors wherein the range may include a single color; and readdressing the second image so that the pixel in the second image resides at the same address as the pixel in the first image. ### Claim 24 A method for image stabilization of a sequence of temporally displayed digital video images, wherein each image contains a number of pixels and each pixel has an associated display position, the method comprising: interpolating between a color of a pixel in a first image and a color of a pixel in a third image to determine a searchable color; identifying a pixel having a color in a second image within a range of the searchable color; and repositioning the images such that the pixels from the first second and third images are aligned. ### Claim 25 A system for image stabilization of a video sequence having a plurality of video frames, the system comprising: a repositioning module for readdressing a second video image so that a pixel of the second video image having a color within a range of colors of a color of a pixel in the first image is readdressed to the address of the pixel in the first image. ### Claim 26 (depends on 25) The system according to claim 25, wherein all of the pixels in the second video image are readdressed. ### Claim 27 (depends on 26) The system according to claim 26, wherein the pixels are readdressed in relation to other pixels in the video frame. ### Claim 28 A computer program product, on a tangible storage medium, for use with a computer system, the computer program product containing computer code for processing a motion picture having first and second frames, each frame having a plurality of pixels, the computer code comprising: computer code for selecting a first pixel in the first frame having a given color; computer code for locating a second pixel in the second frame having the given color; and computer code for aligning the first frame and the second frame based upon the locations of the first and second pixels. ### Claim 29 (depends on 28) The computer program product according to claim 28 wherein the first and second frames are aligned in memory. ### Claim 30 (depends on 28) The computer program product as defined by claim 28 wherein the computer code for aligning comprises aligning the first frame and second frame about the first and second pixels. ### Claim 31 (depends on 28) The computer program product as defined by claim 28 further comprising computer code for determining the color of the first pixel, wherein the color is determined to be the given color. ### Claim 32 (depends on 28) The computer program product as defined by claim 28 wherein the computer code for locating includes computer code for determining if the color of the second pixel is within a given color tolerance. ### Claim 33 (depends on 28) The computer program product as defined by claim 28 wherein the first pixel has a first address, further wherein the computer code for aligning includes computer code for rearranging the second pixel to have an address that equals the first address. ### Claim 34 (depends on 28) The computer program product as defined by claim 28 further including computer code to linearly average the first frame and the second frame to provide a third frame. ### Claim 35 (depends on 34) The computer program product as defined by claim 34, wherein the third frame has a higher resolution than the first or second frame. ### Claim 36 (depends on 35) The computer program product as defined by claim 35, further comprising computer code for displaying the third frame on a display device. ### Claim 37 (depends on 28) The computer program product according to claim 28 further comprising: computer code for selecting a region of the second frame for locating a second pixel having the given color within that region. ### Claim 38 (depends on 37) The computer program product according to claim 37, further comprising: computer code for aligning the first frame and the second frame based upon a predetermined formula if a second pixel is not found within the second frame having the given color. ### Claim 39 (depends on 28) A computer program product according to claim 28, wherein the tangible storage medium is removable. ### Claim 40 A computer program product, on a tangible storage medium, for use on a computer system, the computer program product having computer readable code thereon for processing a motion picture having a plurality of frames, each frame having a plurality of pixels, the computer code comprising: computer code for selecting an initial frame and a final frame, at least one frame being between the initial and final frames; computer code for selecting an initial pixel in the initial frame and a final pixel in the final frame, the initial pixel having an initial color, the final pixel having a final color; computer code for determining an interpolated pixel for each frame between the initial and final frame based on the initial color and the final color; and computer code for aligning the initial frame, final frame, and the at least one frame between the initial and final frames, the frames being aligned based upon the locations of the initial pixel, final pixel, and each interpolated pixel. ### Claim 41 (depends on 40) The computer program product as defined by claim 40 wherein the interpolation is based upon the initial color, the final color, and the total number of frames between the initial and final frames. ### Claim 42 (depends on 40) The computer program product as defined by claim 40 further comprising computer code for locating the interpolated pixel in a given frame between the initial and final frames. ### Claim 43 (depends on 42) The computer program product as defined by claim 42 wherein each interpolated pixel has an interpolated color, further wherein the interpolated pixel in the given frame is located based upon the interpolated color of the interpolated pixel. ### Claim 44 (depends on 43) The computer program product as defined by claim 43 wherein the computer code for locating includes computer code for determining if the interpolated color is within a given color tolerance. ### Claim 45 (depends on 40) The computer program product as defined by claim 40 including computer code for aligning the initial frame, final frame, and the at least one frame in memory. ### Claim 46 (depends on 40) The computer program product as defined by claim 40 wherein the computer code for aligning includes computer code for setting the address of the initial pixel, final pixel and interpolated pixel to the same value. ### Claim 47 (depends on 40) The computer program product according to claim 40 further comprising computer code for displaying at least the initial frame, final frame, and the at least one frame between the initial and final frames on a display. ### Claim 48 (depends on 40) A computer program product according to claim 40, wherein the tangible storage medium is removable. ### Claim 49 A computer program product, on a tangible storage medium, for use with a computer system, the computer program product including computer readable code for image stabilization of a digital sequence of images, wherein each image has a plurality of pixels and wherein each pixel has an associated address for display and is representative of a color, the computer readable code comprising: computer code for selecting a pixel in the beginning image and a pixel in the ending image; computer code for interpolating between the colors associated with the selected pixels from the beginning image and the ending image; the computer code for interpolating including computer code for determining a color value for each image between the beginning image and the ending image; computer code for locating within each image a pixel that is within the range of the determined color; and computer code for readdressing each image so that the located pixels are aligned at the same display address. ### Claim 50 (depends on 49) A computer program product according to claim 49, wherein the tangible storage medium is removable. ### Claim 51 A computer program product, on a tangible storage medium for use with a computer system, the computer program product including computer readable code for image stabilization of a sequence of temporally displayed digital video images, wherein each image contains a number of pixels and each pixel has an associated display position, the computer readable code comprising: computer code for receiving a signal representative of a color of a pixel in a first image; computer code for identifying a pixel in a second image having the same color within a predetermined range of colors wherein the range may include a single color; and computer code for readdressing the second image so that the pixel in the second image resides at the same address as the pixel in the first image. ### Claim 52 (depends on 51) A computer program product according to claim 51, wherein the tangible storage medium is removable. ### Claim 53 A computer program product, on a tangible storage medium, for use with a computer system, the computer program product including computer readable code for image stabilization of a sequence of temporally displayed digital video images, wherein each image contains a number of pixels and each pixel has an associated display position, the computer readable code comprising: computer code for interpolating between a color of a pixel in a first image and a color of a pixel in a third image to determine a searchable color; computer code for identifying a pixel having a color in a second image within a range of the searchable color; and computer code for repositioning the images such that the pixels from the first second and third images are aligned. ### Claim 54 (depends on 53) A computer program product according to claim 53, wherein the tangible storage medium is removable. ### Claim 55 A computer program product, on a tangible storage medium for use with a computer system, the computer program product including computer readable code for image stabilization of a video sequence having a plurality of video frames, the computer readable code comprising: computer code for readdressing a second video image so that a pixel of the second video image having a color within a range of colors of a color of a pixel in the first image is readdressed to the address of the pixel in the first image. ### Claim 56 (depends on 55) The computer program product according to claim 55, wherein in the computer code for readdressing, all of the pixels in the second video image are readdressed. ### Claim 57 (depends on 56) The computer program product according to claim 56, wherein in the computer code for readdressing, the pixels are readdressed in relation to other pixels in the video frame. ### Claim 58 (depends on 55) A computer program product according to claim 55, wherein the tangible storage medium is removable.