# US 2013/0279751 A1 · Keypoint unwarping > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2013/0279751 A1 - Document ID: US20130279751A1 - Status: Published application - Kind: A1 - Application number: 13/869,639 - Filed: 2013-04-24 - Published: 2013-10-24 - Earliest claimed: 2012-04-24 - Assignees: STMicroelectronics S.r.I. - Applicants: Stmicroelectronics S.R.I. - Inventors: Arcangelo Ranieri Bruna; Danilo Pietro Pau - Web page: https://patentgrep.com/patent/US20130279751A1 ## CPC - G06V 10/24: Aligning, centring, orientation detection or correction of the image - G06F 18/211: Selection of the most significant subset of features - G06T 3/18: Image warping, e.g. rearranging pixels individually - G06T 7/33: using feature-based methods - G06T 7/337: {involving reference images or patches} - G06T 7/344: {involving models} - G06T 7/75: {involving models} - G06V 30/18143: {Extracting features based on salient regional features, e.g. scale invariant feature transform [SIFT] keypoints} - H04N 19/57: Motion estimation characterised by a search window with variable size or shape - G06V 10/247: {by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Apparatus and methods to unwarp at least portions of distorted, electronically-captured images are described. Keypoints, instead of an entire image, may be unwarped and used in various machine-vision algorithms, such as object recognition, image matching, and 3D reconstruction algorithms. When using unwarped keypoints, the machine-vision algorithms may perform reliably irrespective of distortions that may be introduced by one or more image capture systems. ## Claims ### Claim 1 An image processing system having at least one processor adapted to unwarp keypoints, the system comprising: a keypoint extractor configured to receive image data representative of a first image and to identify a plurality of keypoints within the image data and generate first keypoint data; a keypoint transformer configured to transform the first keypoint data to second keypoint data based upon an image deformation model; and a descriptor configured to produce, from the image data that has not been transformed based upon the image deformation model, descriptor data for at least some of the plurality of keypoints. ### Claim 2 (depends on 1) The image processing system of claim 1, wherein at least one of the keypoint extractor, the keypoint transformer, or the descriptor comprises a field-programmable gate array. ### Claim 3 (depends on 1) The image processing system of claim 1, further comprising a multiplexor configured to combine at least the descriptor data and second keypoint data into a data stream. ### Claim 4 (depends on 1) The image processing system of claim 1, wherein the first keypoint data comprises first spatial coordinates of the plurality of keypoints and the second keypoint data comprises second spatial coordinates that are transformations of the first spatial coordinates according to the image deformation model. ### Claim 5 (depends on 4) The image processing system of claim 4, further comprising: a descriptor compressor configured to compress the descriptor data; and a keypoint data compressor configured to compress the second keypoint data, wherein the descriptor compressor and keypoint data compressor provide respective compressed data to the multiplexor. ### Claim 6 (depends on 4) The image processing system of claim 4, wherein the second keypoint data further comprises image rotation and/or image magnification information. ### Claim 7 (depends on 1) The image processing system of claim 1, further comprising an image processor configured to execute a machine-vision algorithm using at least the second keypoint data and the descriptor data. ### Claim 8 (depends on 1) The image processing system of claim 1, further comprising a feature matcher configured to compare features of the first image with features of a second image based at least upon the second keypoint data and the descriptor data. ### Claim 9 (depends on 1) The image processing system of claim 1, further comprising an image capture device disposed in a smart phone, mobile phone, or personal digital assistant. ### Claim 10 (depends on 1) The image processing system of claim 1, wherein the image deformation model is representative of an image distortion introduced into the image by an image-capture device or is representative of an operation to remove image distortion introduced into the image by an image-capture device. ### Claim 11 (depends on 1) The image processing system of claim 1, wherein information about the image deformation model is received with the image data. ### Claim 12 (depends on 1) The image processing system of claim 1, further comprising image distortion detection apparatus configured to enable or disable the keypoint transformer based upon a type of image distortion detected. ### Claim 13 An image processing method for electronically unwarping keypoints by at least one processor, the method comprising: receiving, by a keypoint extractor, image data representative of a first image; identifying, by a keypoint extractor, a plurality of keypoints within the image data; generating, by the keypoint extractor, first keypoint data corresponding to the plurality of identified keypoints; transforming, by a keypoint transformer, the first keypoint data to second keypoint data based upon an image deformation model; and producing, by a descriptor, from the image data that has not been transformed based upon the image deformation model, descriptor data for at least some of the plurality of keypoints. ### Claim 14 (depends on 13) The image processing method of claim 13, wherein at least one of the keypoint extractor, the keypoint transformer, or the descriptor comprises a field-programmable gate array. ### Claim 15 (depends on 13) The image processing method of claim 13, further comprising multiplexing, by a multiplexor, at least the descriptor data and second keypoint data into a data stream. ### Claim 16 (depends on 13) The image processing method of claim 13, wherein transforming the first keypoint data comprises transforming spatial coordinates of the first keypoint data and not transforming spatial coordinates of the full image. ### Claim 17 (depends on 13) The image processing method of claim 13, further comprising compressing the second keypoint data and descriptor data. ### Claim 18 (depends on 13) The image processing method of claim 13, further comprising comparing features of the first image with features of a second image based at least upon the second keypoint data and the descriptor data. ### Claim 19 An image processing system having at least one processor adapted to unwarp keypoints, the system comprising: a matching model generator configured to receive image deformation data associated with a distorted image and to identify a matching model to a feature matcher; and the feature matcher configured to receive keypoint data and descriptor data obtained from a distorted image, the matching model, and feature data associated with at least one comparison image, wherein the feature matcher unwarps keypoints in the keypoint data in accordance with the matching model. ### Claim 20 (depends on 19) The image processing system of claim 19, wherein at least one of the matching model generator or the feature matcher comprises a field-programmable gate array. ### Claim 21 (depends on 19) The image processing system of claim 19, wherein the image deformation data is representative of an image distortion introduced into the image by an image-capture device or is representative of an operation to remove image distortion introduced into the image by an image-capture device. ### Claim 22 (depends on 19) The image processing system of claim 19, further comprising a demultiplexor that is configured to parse the keypoint data, the descriptor data, and the image deformation data from received data. ### Claim 23 (depends on 19) The image processing system of claim 19, wherein the identified matching model is selected or generated based upon the received image deformation data. ### Claim 24 (depends on 19) The image processing system of claim 19, further comprising image distortion detection apparatus configured to enable or disable the matching model generator based upon a type of image distortion detected in the received image deformation data and a type of image distortion detected in the feature data. ### Claim 25 An image processing method for electronically unwarping keypoints by at least one processor, the method comprising: receiving, by a matching model generator, image deformation data associated with a distorted image identifying, by the matching model generator, a matching model to a feature matcher; receiving, by the feature matcher, keypoint data obtained from a distorted image; and unwarping, by the feature matcher, the keypoint data in accordance with the identified matching model. ### Claim 26 (depends on 25) The image processing method of claim 25, wherein at least one of the matching model generator or the feature matcher comprises a field-programmable gate array. ### Claim 27 (depends on 25) The image processing method of claim 25, further comprising receiving, by the feature matcher, descriptor data associated with the keypoint data. ### Claim 28 (depends on 25) The image processing method of claim 25, wherein the identifying comprises selecting or generating the matching model based upon the received image deformation data. ### Claim 29 (depends on 25) The image processing method of claim 25, further comprising: receiving, by the feature matcher, feature data associated with at least one image; detecting a first type of image distortion in the received image deformation data and a second type of image distortion in the feature data; and making a determination to enable or disable the matching model generator based upon the first and second types of image distortion detected.