# US 2021/0241114 A1 · Fully convolutional interest point detection and description via homographic adaptation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241114 A1 - Document ID: US20210241114A1 - Status: Published application - Kind: A1 - Application number: 17/179,226 - Filed: 2021-02-18 - Published: 2021-08-05 - Earliest claimed: 2017-11-14 - Assignees: Magic Leap, Inc. - Applicants: Magic Leap, Inc. - Inventors: Andrew Rabinovich; Daniel DeTone; Tomasz Jan Malisiewicz - Web page: https://patentgrep.com/patent/US20210241114A1 ## CPC - G06N 3/08: Learning methods - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06F 18/24143: {Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/048: Activation functions - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)} - G06V 10/242: {by image rotation, e.g. by 90 degrees} - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]} - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/20: in augmented reality scenes - Y02T 10/40: Engine management systems - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION ## Abstract Systems, devices, and methods for training a neural network and performing image interest point detection and description using the neural network. The neural network may include an interest point detector subnetwork and a descriptor subnetwork. An optical device may include at least one camera for capturing a first image and a second image. A first set of interest points and a first descriptor may be calculated using the neural network based on the first image, and a second set of interest points and a second descriptor may be calculated using the neural network based on the second image. A homography between the first image and the second image may be determined based on the first and second sets of interest points and the first and second descriptors. The optical device may adjust virtual image light being projected onto an eyepiece based on the homography. ## Claims ### Claim 1 (canceled) ### Claim 2 A computer-implemented method comprising: obtaining a first image; obtaining a second image; calculating, by a neural network receiving the first image as input, a first set of calculated interest points and a first calculated descriptor; calculating, by the neural network receiving the second image as input, a second set of calculated interest points and a second calculated descriptor; and determining a homography between the first image and the second image based on the first and second sets of calculated interest points and the first and second calculated descriptors; wherein the neural network includes: an interest point detector subnetwork configured to calculate the first set of calculated interest points and the second set of calculated interest points; and a descriptor subnetwork configured to calculate the first calculated descriptor and the second calculated descriptor. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, further comprising: determining point correspondences between the first and second sets of calculated interest points, wherein the homography is determined based on the point correspondences. ### Claim 4 (depends on 2) The computer-implemented method of claim 2, wherein the first image has dimensions of H×W, and wherein the second image has dimensions of H×W. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the first set of calculated interest points is represented by a first two-dimensional (2D) map having dimensions of H×W, and wherein the second set of calculated interest points is represented by a second 2D map having dimensions of H×W. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein the first calculated descriptor has dimensions of H×W×D, and wherein the second calculated descriptor has dimensions of H×W×D. ### Claim 7 (depends on 2) The computer-implemented method of claim 2, wherein the interest point detector subnetwork is configured to calculate the first set of calculated interest points concurrently with the descriptor subnetwork calculating the first calculated descriptor. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein the interest point detector subnetwork is configured to calculate the second set of calculated interest points concurrently with the descriptor subnetwork calculating the second calculated descriptor. ### Claim 9 A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: obtaining a first image; obtaining a second image; calculating, by a neural network receiving the first image as input, a first set of calculated interest points and a first calculated descriptor; calculating, by the neural network receiving the second image as input, a second set of calculated interest points and a second calculated descriptor; and determining a homography between the first image and the second image based on the first and second sets of calculated interest points and the first and second calculated descriptors; wherein the neural network includes: an interest point detector subnetwork configured to calculate the first set of calculated interest points and the second set of calculated interest points; and a descriptor subnetwork configured to calculate the first calculated descriptor and the second calculated descriptor. ### Claim 10 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein the operations further comprise: determining point correspondences between the first and second sets of calculated interest points, wherein the homography is determined based on the point correspondences. ### Claim 11 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein the first image has dimensions of H×W, and wherein the second image has dimensions of H×W. ### Claim 12 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the first set of calculated interest points is represented by a first two-dimensional (2D) map having dimensions of H×W, and wherein the second set of calculated interest points is represented by a second 2D map having dimensions of H×W. ### Claim 13 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein the first calculated descriptor has dimensions of H×W×D, and wherein the second calculated descriptor has dimensions of H×W×D. ### Claim 14 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein the interest point detector subnetwork is configured to calculate the first set of calculated interest points concurrently with the descriptor subnetwork calculating the first calculated descriptor. ### Claim 15 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the interest point detector subnetwork is configured to calculate the second set of calculated interest points concurrently with the descriptor subnetwork calculating the second calculated descriptor. ### Claim 16 A system comprising: one or more processors; and a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: obtaining a first image; obtaining a second image; calculating, by a neural network receiving the first image as input, a first set of calculated interest points and a first calculated descriptor; calculating, by the neural network receiving the second image as input, a second set of calculated interest points and a second calculated descriptor; and determining a homography between the first image and the second image based on the first and second sets of calculated interest points and the first and second calculated descriptors; wherein the neural network includes: an interest point detector subnetwork configured to calculate the first set of calculated interest points and the second set of calculated interest points; and a descriptor subnetwork configured to calculate the first calculated descriptor and the second calculated descriptor. ### Claim 17 (depends on 16) The system of claim 16, wherein the operations further comprise: determining point correspondences between the first and second sets of calculated interest points, wherein the homography is determined based on the point correspondences. ### Claim 18 (depends on 16) The system of claim 16, wherein the first image has dimensions of H×W, and wherein the second image has dimensions of H×W. ### Claim 19 (depends on 18) The system of claim 18, wherein the first set of calculated interest points is represented by a first two-dimensional (2D) map having dimensions of H×W, and wherein the second set of calculated interest points is represented by a second 2D map having dimensions of H×W. ### Claim 20 (depends on 19) The system of claim 19, wherein the first calculated descriptor has dimensions of H×W×D, and wherein the second calculated descriptor has dimensions of H×W×D. ### Claim 21 (depends on 16) The system of claim 16, wherein the interest point detector subnetwork is configured to calculate the first set of calculated interest points concurrently with the descriptor subnetwork calculating the first calculated descriptor.