# US 2017/0257572 A1 · Methods and systems for camera drift correction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2017/0257572 A1 - Document ID: US20170257572A1 - Status: Published application - Kind: A1 - Application number: 15/058,660 - Filed: 2016-03-02 - Published: 2017-09-07 - Earliest claimed: 2016-03-02 - Assignees: Xerox Corporation - Applicants: Xerox Corporation - Inventors: Hao Wu; Howard Mizes; Wencheng Wu; Graham Pennington; Rakesh Kulkarni - Web page: https://patentgrep.com/patent/US20170257572A1 ## CPC - G06T 5/73: Deblurring; Sharpening - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/174: involving the use of two or more images - G06T 7/215: Motion-based segmentation - G06T 7/33: using feature-based methods - G06V 10/147: Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19) - G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes} - G06V 20/54: of traffic, e.g. cars on the road, trains or boats - H04N 23/6811: {based on the image signal} - H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory} - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method, system, and apparatus for video frame alignment comprises collecting video data comprising at least two video frames; extracting a line profile along at least one line profile in each of the at least two video frames; selecting one of the at least two video frames as a reference video frame; segmenting each of the at least one line profiles into a plurality of segmented line profile group segments; aligning the plurality of segmented line profiles with the segmented line profiles in the reference video frame; translating each of the at least two video frames for each of the plurality of corresponding segmented line profile alignments; and removing a camera shift from the at least two video frames according to the translation and alignment of the plurality of segmented line profiles with the plurality of segmented line profile in the reference video frame. ## Claims ### Claim 1 A method for video frame alignment comprising: collecting video data comprising at least two video frames; extracting a line profile along at least one line path in each of said at least two video frames; selecting one of said at least two video frames as a reference video frame; aligning at least one of said at least one line profiles with said line profile in said reference video frame; translating each of said at least two video frames according to said corresponding line profile alignments; and removing a camera shift from said at least two video frames according to said translation and alignment of said at least one of said at least one line profiles with said line profile in said reference video frame. ### Claim 2 (depends on 1) The method of claim 1 further comprising: stacking said at least one line profiles in each of said at least two frames forming at least one spatiotemporal image. ### Claim 3 (depends on 1) The method of claim 1 further comprising: segmenting each of said at least one line profiles into at least one line profile group segment. ### Claim 4 (depends on 1) The method of claim 1 wherein said at least one line profile further comprises a sequence of image pixel values sampled from a beginning pixel of said line path to an end pixel of said line path. ### Claim 5 (depends on 2) The method of claim 2 wherein selecting one of said at least two video frames as a reference video frame further comprises: identifying one of said at least two video frames containing a plurality of scene features shared with said at least one other of said at least two video frames; and selecting said identified one of said at least two video frames as a reference video frame according to said stacked spatiotemporal image. ### Claim 6 (depends on 1) The method of claim 1 wherein said line path comprise at least one of: a horizontal line; a vertical line; a diagonal line; a curved line; a line along an expected drift direction in said video data; and a line along a path with a stable background scene content in said at least two video frames. ### Claim 7 (depends on 1) The method of claim 1 wherein aligning at least one of said at least one line profiles with said line profile in said reference video frame further comprises: finding a shift value that maximizes a cross-correlation of said at least one line profile with said line profile in said reference video frame; finding a shift value that maximizes a covariance of said at least one line profile with said line profiles in said reference video frame; and outputting said shift value for each of said at least one line profiles. ### Claim 8 (depends on 6) The method of claim 6 wherein removing a camera shift from said at least two video frames further comprises: collecting all said shift values for each of said at least one line profiles; calculating an overall camera shift according to said shift values for each of said at least one line profiles; and adjusting said collected video data comprising at least two video frames to remove said overall camera shift in said collected video data. ### Claim 9 A system for video frame alignment comprising: a video acquisition module configured to collect image data comprising at least two image frames; a processor; and a computer-usable medium embodying computer code, said computer-usable medium being coupled to said processor, said computer code comprising non-transitory instruction media executable by said processor configured for: extracting a line profile along at least one line path in each of said at least two video frames; selecting one of said at least two video frames as a reference video frame; aligning at least one of said at least one line profiles with said line profile in said reference video frame; translating each of said at least two video frames according to said corresponding line profile alignments; and removing a camera shift from said at least two video frames according to said translation and alignment of said at least one of said at least one line profiles with said line profile in said reference video frame. ### Claim 10 (depends on 9) The system of claim 9 wherein said computer code comprising non-transitory instruction media executable by said processor is further configured for: stacking said at least one line profiles in each of said at least two frames forming at least one spatiotemporal image. ### Claim 11 (depends on 9) The system of claim 9 wherein said computer code comprising non-transitory instruction media executable by said processor is further configured for: segmenting each of said at least one line profiles into at least one line profile group segment. ### Claim 12 (depends on 9) The system of claim 9 wherein said at least one line profile further comprises a sequence of image pixel values sampled from a beginning pixel of said line path to an end pixel of said line path. ### Claim 13 (depends on 9) The system of claim 9 wherein selecting one of said at least two video frames as a reference video frame further comprises: identifying one of said at least two video frames containing a plurality of scene features shared with said at least one other of said at least two video frames; and selecting said identified one of said at least two video frames as a reference video frame according to said stacked spatiotemporal image. ### Claim 14 (depends on 9) The system of claim 9 wherein said line path comprise at least one of: a horizontal line; a vertical line; a diagonal line; a curved line; a line along an expected drift direction in said video data; and a line along a path with a stable background scene content in said at least two video frames. ### Claim 15 (depends on 9) The system of claim 9 wherein aligning at least one of said at least one line profiles with said line profile in said reference video frame further comprises: finding a shift value that maximizes a cross-correlation of said at least one line profile with said line profile in said reference video frame; finding a shift value that maximizes a covariance of said at least one line profile with said line profiles in said reference video frame; and outputting said shift value for each of said at least one line profiles. ### Claim 16 (depends on 15) The system of claim 15 wherein removing a camera shift from said at least two video frames further comprises: collecting all said shift values for each of said at least one line profiles; calculating an overall camera shift according to said shift values for each of said at least one line profiles; and adjusting said collected video data comprising at least two video frames to remove said overall camera shift in said collected video data. ### Claim 17 An apparatus for video frame alignment comprising: a video acquisition module for collecting video data comprising at least two video frames; a processor; a computer-usable medium embodying computer code, said computer-usable medium being coupled to said processor, said computer code comprising non-transitory instruction media executable by said processor; a line profile extraction module which extracts a line profile along at least one line path in each of said at least two video frames; a reference frame selection module which selects one of said at least two video frames as a reference video frame; a registration module which aligns at least one of said at least one line profiles with said line profile in said reference video frame; and a correction module for translating each of said at least two video frames according to said corresponding line profile alignments to thereby remove a camera shift from said at least two video frames according to said translation and alignment of said at least one of said at least one line profiles with said line profile in said reference video frame. ### Claim 18 (depends on 17) The apparatus of claim 17 further comprising: a segmenting module which segments each of said at least one line profiles into a plurality of segmented line profiles. ### Claim 19 (depends on 17) The apparatus of claim 17 further comprising: a stacking module which stacks said at least one line profiles in each of said at least two frames forming at least one spatiotemporal image. ### Claim 20 (depends on 17) The apparatus of claim 17 wherein said line path comprise at least one of: a horizontal line; a vertical line; a diagonal line; a curved line; a line along an expected drift direction in said video data; and a line along a path with a stable background scene content in said at least two video frames.