# US 10,121,262 B2 · Method, system and apparatus for determining alignment data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,121,262 B2 - Document ID: US10121262B2 - Status: Granted - Kind: B2 - Application number: 15/385,388 - Filed: 2016-12-20 - Published: 2018-06-21 - Granted: 2018-11-06 - Earliest claimed: 2016-12-20 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Dmitri Katchalov - Web page: https://patentgrep.com/patent/US10121262B2 ## CPC - G06T 7/37: using transform domain methods - G06T 7/337: {involving reference images or patches} - G06T 2207/10024: Color image - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT] - G06T 2210/12: Bounding box - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method of determining alignment data for a target frame and a reference frame. Patch data is received for each of a plurality of reference patches of the reference frame. The patch data comprises a reference direction of a feature in an associated reference patch and a reference signal corresponding to a projection of reference patch image data in a direction substantially perpendicular to the reference direction. A shift between at least one of said reference patches and a corresponding target patch in the target frame is determined using the reference signal associated with the reference patch and a target signal corresponding to a projection of the target patch image data in the direction substantially perpendicular to the reference direction associated with said reference patch. The alignment data for the target frame is determined using the determined shift for the reference patch. ## Claims ### Claim 1 A method of aligning data for a target frame and a reference frame, the method comprising: receiving patch data for each of a plurality of reference patches of the reference frame, the patch data comprising a reference direction of a feature in an associated reference patch and a reference signal corresponding to a projection of reference patch image data in a direction substantially perpendicular to the reference direction; determining a shift between at least one of said reference patches and a corresponding target patch in the target frame using the reference signal associated with said reference patch and a target signal corresponding to a projection of the target patch image data in the direction substantially perpendicular to the reference direction associated with said reference patch; determining the alignment data for the target frame using the determined shift for said reference patch; and transforming image data of the target frame using the alignment data. ### Claim 2 (depends on 1) The method according to claim 1, wherein the alignment data is used to generate camera configuration. ### Claim 3 (depends on 2) The method according to claim 2, wherein the image data of the target frame is transformed based on the generated camera configuration. ### Claim 4 (depends on 1) The method according to claim 1, further comprising: correlating a reference patch with a cropped version of the reference patch to form a correlation image; transforming the correlation image to a polar coordinate space; and determining a peak in the transformed correlation image. ### Claim 5 (depends on 1) The method according to claim 1, further comprising summing pixel values in the direction perpendicular to the reference direction. ### Claim 6 (depends on 1) The method according to claim 1, wherein the projection is determined while reading pixels for the patch. ### Claim 7 (depends on 1) The method according to claim 1, wherein the reference patch is selected by: selecting a plurality of the reference patches from the reference frame; and determining a reference direction and a quality measure for each of the selected candidate patches. ### Claim 8 (depends on 1) The method according to claim 1, further comprising selecting the reference patches using quality measures, coordinates of the reference patches and corresponding reference directions. ### Claim 9 (depends on 1) The method according to claim 1, wherein the alignment data is a transform. ### Claim 10 (depends on 1) The method according to claim 1, wherein the alignment data is used to generate camera calibration parameters. ### Claim 11 An apparatus for aligning data for a target frame and a reference frame, the apparatus comprising: means for receiving patch data for each of a plurality of reference patches of the reference frame, the patch data comprising a reference direction of a feature in an associated reference patch and a reference signal corresponding to a projection of reference patch image data in a direction substantially perpendicular to the reference direction; means for determining a shift between at least one of said reference patches and a corresponding target patch in the target frame using the reference signal associated with said reference patch and a target signal corresponding to a projection of the target patch image data in the direction substantially perpendicular to the reference direction associated with said reference patch; means for determining the alignment data for the target frame using the determined shift for said reference patch; and means for transforming the target frame using the alignment data. ### Claim 12 A system for aligning data for a target frame and a reference frame, the system comprising: a memory for storing data and a computer program; a processor coupled to the memory for executing the computer program, the computer program comprising instructions for: receiving patch data for each of a plurality of reference patches of the reference frame, the patch data comprising a reference direction of a feature in an associated reference patch and a reference signal corresponding to a projection of reference patch image data in a direction substantially perpendicular to the reference direction; determining a shift between at least one of said reference patches and a corresponding target patch in the target frame using the reference signal associated with said reference patch and a target signal corresponding to a projection of the target patch image data in the direction substantially perpendicular to the reference direction associated with said reference patch; determining the alignment data for the target frame using the determined shift for said reference patch; and transforming the target frame using the alignment data. ### Claim 13 A non-transitory computer readable medium having stored thereon a computer program for aligning data for a target frame and a reference frame, the program comprising: code for receiving patch data for each of a plurality of reference patches of the reference frame, the patch data comprising a reference direction of a feature in an associated reference patch and a reference signal corresponding to a projection of reference patch image data in a direction substantially perpendicular to the reference direction; code for determining a shift between at least one of said reference patches and a corresponding target patch in the target frame using the reference signal associated with said reference patch and a target signal corresponding to a projection of the target patch image data in the direction substantially perpendicular to the reference direction associated with said reference patch; code for determining the alignment data for the target frame using the determined shift for said reference patch; and code for transforming the target frame using the alignment data. ### Claim 14 A method of determining a camera configuration, the method comprising: receiving patch data for each of a plurality of reference patches of a reference frame, the patch data comprising a reference direction of a feature in an associated reference patch and a reference signal corresponding to a projection of reference patch image data in a direction substantially perpendicular to the reference direction; determining a shift between at least one of said reference patches and a corresponding target patch in a target frame using the reference signal associated with said reference patch and a target signal corresponding to a projection of the target patch image data in the direction substantially perpendicular to the reference direction associated with said reference patch; determining the alignment data for the target frame using the determined shift for said reference patch; and determining the camera configuration using the alignment data. ### Claim 15 (depends on 14) The method according to claim 14, further comprising summing pixel values in the direction perpendicular to the reference direction. ### Claim 16 (depends on 14) The method according to claim 14, wherein the projection is determined while reading pixels for the patch. ### Claim 17 (depends on 14) The method according to claim 14, wherein the reference patch is selected by: selecting a plurality of the reference patches from the reference frame; and determining a reference direction and a quality measure for each of the selected candidate patches. ### Claim 18 (depends on 14) The method according to claim 14, further comprising selecting the reference patches using quality measures, coordinates of the reference patches and corresponding reference directions. ### Claim 19 A system for generating a free viewpoint video, the system comprising: a memory for storing data and a computer program; a processor coupled to the memory for executing the computer program, the computer program comprising instructions for: receiving patch data for each of a plurality of reference patches of a reference frame, the patch data comprising a reference direction of a feature in an associated reference patch and a reference signal corresponding to a projection of reference patch image data in a direction substantially perpendicular to the reference direction; determining a shift between at least one of said reference patches and a corresponding target patch in a target frame using the reference signal associated with said reference patch and a target signal corresponding to a projection of the target patch image data in the direction substantially perpendicular to the reference direction associated with said reference patch; determining the alignment data for the target frame using the determined shift for said reference patch to determine a camera calibration parameter; and generating the free viewpoint video using the determined camera calibration parameter.