# US 2021/0241022 A1 · Feature extraction with keypoint resampling and fusion (krf) > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241022 A1 - Document ID: US20210241022A1 - Status: Published application - Kind: A1 - Application number: 17/162,246 - Filed: 2021-01-29 - Published: 2021-08-05 - Earliest claimed: 2020-01-30 - Assignees: Mediatek Inc. - Applicants: Mediatek Inc. - Inventors: Chia-Da LEE; Wai Mun WONG; Pei-Kuei TSUNG - Web page: https://patentgrep.com/patent/US20210241022A1 ## CPC - G06N 3/08: Learning methods - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method of feature extraction from an image can include receiving the image including pixels, generating confidence values corresponding to positions of the pixels in the image by an artificial intelligence (AI) based feature extractor, selecting a first position among the positions of the pixels in the image, a first confidence value among the generated confidence values at the first position being higher than a first threshold, and generating a final set of keypoint-descriptor pairs based on the confidence values corresponding to positions of the pixels in the image. The final set of keypoint-descriptor pairs includes at least two keypoint-descriptor pairs corresponding to the first position among the positions of the pixels in the image. ## Claims ### Claim 1 A method of feature extraction from an image, comprising: receiving the image including pixels; generating confidence values corresponding to positions of the pixels in the image by an artificial intelligence (AI) based feature extractor; selecting a first position among the positions of the pixels in the image, a first confidence value among the generated confidence values at the first position being higher than a first threshold; and generating a final set of keypoint-descriptor pairs based on the confidence values corresponding to positions of the pixels in the image, the final set of keypoint-descriptor pairs including at least two keypoint-descriptor pairs corresponding to the first position among the positions of the pixels in the image. ### Claim 2 (depends on 1) The method of claim 1, further comprising: selecting a first set of positions from the positions of the pixels in the image to be a first set of keypoints based on the generated confidence values, each of the first set of the positions corresponding to one of the generated confidence values that is above a second threshold; and selecting a second set of positions from the positions of the pixels in the image to be a second set of keypoints based on the generated confidence values, each of the second set of positions corresponding to one of the generated confidence values that is above the first threshold, the first set of positions including the second set of positions, wherein the selecting the second set of positions from the positions of the pixels in the image includes the selecting of the first position among the positions of the pixels in the image, and the final set of keypoint-descriptors includes a first set of keypoint-descriptor pairs corresponding to members of the first set of the keypoints and a second set of keypoint-descriptor pairs corresponding to members of the second set of the keypoints. ### Claim 3 (depends on 1) The method of claim 1, further comprising: selecting a first set of positions from the positions of the pixels in the image to be a first set of keypoints based on the generated confidence values, each of the first set of the positions corresponding to one of the generated confidence values that is above a second threshold; and selecting a second set of positions from the first set of positions in the image to be a second set of keypoints based on the generated confidence values, each of the second set of positions corresponding to one of the generated confidence values that is above the first threshold, wherein the selecting the second set of positions from the first set of positions in the image includes the selecting of the first position among the positions of the pixels in the image, and the final set of keypoint-descriptors includes a first set of keypoint-descriptor pairs corresponding to members of the first set of the keypoints and a second set of keypoint-descriptor pairs corresponding to members of the second set of the keypoints. ### Claim 4 (depends on 1) The method of claim 1, further comprising: determining whether the first confidence value among the generated confidence values at the first position is higher than the first threshold; and in response to the first confidence value among the generated confidence values at the first position being higher than the first threshold, selecting a first keypoint descriptor from the first position in a first descriptor map including keypoint descriptors corresponding to the positions of the pixels in the image, and selecting a second keypoint descriptor from the first position in a second descriptor map including keypoint descriptors corresponding to the positions of the pixels in the image, wherein the at least two keypoint-descriptor pairs include two keypoint-descriptor pairs corresponding to the first keypoint descriptor and the second keypoint descriptor. ### Claim 5 (depends on 1) The method of claim 1, further comprising: generating a first descriptor map by the AI based feature extractor; and generating a second descriptor map, wherein the at least two keypoint-descriptor pairs include two keypoint-descriptor pairs corresponding to a first keypoint descriptor in the first descriptor map and a second keypoint descriptor in the second descriptor map. ### Claim 6 (depends on 5) The method of claim 5, wherein the first keypoint descriptor in the first descriptor map is a first binary vector, and the second keypoint descriptor in the second descriptor map is a second binary vector complementary to the first binary vector, where 1 or 0 in the first binary vector is switched to 0 and 1, respectively, in the second binary vector. ### Claim 7 (depends on 1) The method of claim 1, further comprising: resizing the image to generate a resized image; determining keypoint-descriptor pairs corresponding to the resized image; and including the keypoint-descriptor pairs corresponding to the resized image in the final set of keypoint-descriptor pairs. ### Claim 8 An apparatus of feature extraction from an image, comprising circuitry configured to: receive the image including pixels; generate confidence values corresponding to positions of the pixels in the image by an artificial intelligence (AI) based feature extractor; select a first position among the positions of the pixels in the image, a first confidence value among the generated confidence values at the first position being higher than a first threshold; and generate a final set of keypoint-descriptor pairs based on the confidence values corresponding to positions of the pixels in the image, the final set of keypoint-descriptor pairs including at least two keypoint-descriptor pairs corresponding to the first position among the positions of the pixels in the image. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the circuitry is further configured to: select a first set of positions from the positions of the pixels in the image to be a first set of keypoints based on the generated confidence values, each of the first set of the positions corresponding to one of the generated confidence values that is above a second threshold; and select a second set of positions from the positions of the pixels in the image to be a second set of keypoints based on the generated confidence values, each of the second set of positions corresponding to one of the generated confidence values that is above the first threshold, the first set of positions including the second set of positions, wherein the second set of positions includes the first position among the positions of the pixels in the image, and the final set of keypoint-descriptors includes a first set of keypoint-descriptor pairs corresponding to members of the first set of the keypoints and a second set of keypoint-descriptor pairs corresponding to members of the second set of the keypoints. ### Claim 10 (depends on 8) The apparatus of claim 8, wherein the circuitry is further configured to: select a first set of positions from the positions of the pixels in the image to be a first set of keypoints based on the generated confidence values, each of the first set of the positions corresponding to one of the generated confidence values that is above a second threshold; and select a second set of positions from the first set of positions in the image to be a second set of keypoints based on the generated confidence values, each of the second set of positions corresponding to one of the generated confidence values that is above the first threshold, wherein the second set of positions includes the first position among the positions of the pixels in the image, and the final set of keypoint-descriptors includes a first set of keypoint-descriptor pairs corresponding to members of the first set of the keypoints and a second set of keypoint-descriptor pairs corresponding to members of the second set of the keypoints. ### Claim 11 (depends on 8) The apparatus of claim 8, wherein the circuitry is further configured to: determine whether the first confidence value among the generated confidence values at the first position is higher than the first threshold; and in response to the first confidence value among the generated confidence values at the first position being higher than the first threshold, select a first keypoint descriptor from the first position in a first descriptor map including keypoint descriptors corresponding to the positions of the pixels in the image, and select a second keypoint descriptor from the first position in a second descriptor map including keypoint descriptors corresponding to the positions of the pixels in the image, wherein the at least two keypoint-descriptor pairs include two keypoint-descriptor pairs corresponding to the first keypoint descriptor and the second keypoint descriptor. ### Claim 12 (depends on 8) The apparatus of claim 8, wherein the circuitry is further configured to: generate a first descriptor map by the AI based feature extractor; and generate a second descriptor map, wherein the at least two keypoint-descriptor pairs include two keypoint-descriptor pairs corresponding to a first keypoint descriptor in the first descriptor map and a second keypoint descriptor in the second descriptor map. ### Claim 13 (depends on 12) The apparatus of claim 12, wherein the first keypoint descriptor in the first descriptor map is a first binary vector, and the second keypoint descriptor in the second descriptor map is a second binary vector complementary to the first binary vector, where 1 or 0 in the first binary vector is switched to 0 and 1, respectively, in the second binary vector. ### Claim 14 (depends on 8) The apparatus of claim 8, wherein the circuitry is further configured to: resize the image to generate a resized image; determine keypoint-descriptor pairs corresponding to the resized image; and include the keypoint-descriptor pairs corresponding to the resized image in the final set of keypoint-descriptor pairs. ### Claim 15 A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method of feature extraction from an image, the method comprising: receiving the image including pixels; generating confidence values corresponding to positions of the pixels in the image by an artificial intelligence (AI) based feature extractor; selecting a first position among the positions of the pixels in the image, a first confidence value among the generated confidence values at the first position being higher than a first threshold; and generating a final set of keypoint-descriptor pairs based on the confidence values corresponding to positions of the pixels in the image, the final set of keypoint-descriptor pairs including at least two keypoint-descriptor pairs corresponding to the first position among the positions of the pixels in the image. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the method further comprises: selecting a first set of positions from the positions of the pixels in the image to be a first set of keypoints based on the generated confidence values, each of the first set of the positions corresponding to one of the generated confidence values that is above a second threshold; and selecting a second set of positions from the positions of the pixels in the image to be a second set of keypoints based on the generated confidence values, each of the second set of positions corresponding to one of the generated confidence values that is above the first threshold, the first set of positions including the second set of positions, wherein the selecting the second set of positions from the positions of the pixels in the image includes the selecting of the first position among the positions of the pixels in the image, and the final set of keypoint-descriptors includes a first set of keypoint-descriptor pairs corresponding to members of the first set of the keypoints and a second set of keypoint-descriptor pairs corresponding to members of the second set of the keypoints. ### Claim 17 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the method further comprises: selecting a first set of positions from the positions of the pixels in the image to be a first set of keypoints based on the generated confidence values, each of the first set of the positions corresponding to one of the generated confidence values that is above a second threshold; and selecting a second set of positions from the first set of positions in the image to be a second set of keypoints based on the generated confidence values, each of the second set of positions corresponding to one of the generated confidence values that is above the first threshold, wherein the selecting the second set of positions from the first set of positions in the image includes the selecting of the first position among the positions of the pixels in the image, and the final set of keypoint-descriptors includes a first set of keypoint-descriptor pairs corresponding to members of the first set of the keypoints and a second set of keypoint-descriptor pairs corresponding to members of the second set of the keypoints. ### Claim 18 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the method further comprises: determining whether the first confidence value among the generated confidence values at the first position is higher than the first threshold; and in response to the first confidence value among the generated confidence values at the first position being higher than the first threshold, selecting a first keypoint descriptor from the first position in a first descriptor map including keypoint descriptors corresponding to the positions of the pixels in the image, and selecting a second keypoint descriptor from the first position in a second descriptor map including keypoint descriptors corresponding to the positions of the pixels in the image, wherein the at least two keypoint-descriptor pairs include two keypoint-descriptor pairs corresponding to the first keypoint descriptor and the second keypoint descriptor. ### Claim 19 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the method further comprises: generating a first descriptor map by the AI based feature extractor; and generating a second descriptor map, wherein the at least two keypoint-descriptor pairs include two keypoint-descriptor pairs corresponding to a first keypoint descriptor in the first descriptor map and a second keypoint descriptor in the second descriptor map. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein the first keypoint descriptor in the first descriptor map is a first binary vector, and the second keypoint descriptor in the second descriptor map is a second binary vector complementary to the first binary vector, where 1 or 0 in the first binary vector is switched to 0 and 1, respectively, in the second binary vector.