# US 11,443,439 B1 · Auto simultaneous referencing air-to-air background oriented schlieren > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,439 B1 - Document ID: US11443439B1 - Status: Granted - Kind: B1 - Application number: 16/820,197 - Filed: 2020-03-16 - Granted: 2022-09-13 - Earliest claimed: 2019-03-14 - Assignees: U.S.A. as represented by the Administrator of the National Aeronautics and Space Administration - Applicants: U.S.A. as Represented by the Administrator of NASA - Inventors: Daniel W Banks; James T Heineck - Web page: https://patentgrep.com/patent/US11443439B1 ## CPC - G06T 7/248: {involving reference images or patches} - G06T 7/269: using gradient-based methods - G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration - G06T 2207/10032: Satellite or aerial image; Remote sensing - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An air-to-air background-oriented schlieren system in which reference frames are acquired concurrently with the image frames, recording a target aircraft from a sensor aircraft flying in formation, while concurrently recording reference frames of underlying terrain to provide a visually textured background as a reference. This auto-referencing method improves the original AirBOS method by allowing a much more flexible and useful measurement, reducing the flight planning and piloting burden, and broadening the possible camera choices to acquire imaging of visible density changes in air that cause a refractive index change by an airborne vehicle. ## Claims ### Claim 1 A method for discerning density changes in air caused by an airborne vehicle, the method comprising the steps of: flying a target airborne vehicle at a predetermined airspeed and altitude over a background; flying a sensor airborne vehicle in formation with said target airborne vehicle at an airspeed and altitude such that said target airborne vehicle remains fixed in a field-of-view of a target sensor and such that refractive wake features of said target airborne vehicle are included within said field-of-view of said target sensor; capturing a series of image pairs from said sensor airborne vehicle, each image pair including a reference image of said background taken in advance of said target airborne vehicle such that said reference image is undisturbed by said target airborne vehicle, and a target image taken with said target airborne vehicle over said background; for each image pair, time shifting said reference image to adjust for a time delay relative to said target image; and processing the series of image pairs by analyzing comparative displacement of features in said background between said time-shifted reference images and said target images as a result of said target airborne vehicle's refractive wake features. ### Claim 2 (depends on 1) The method for discerning density changes in air according to claim 1, wherein said processing step comprises a substep of mathematically comparing, for at least one of said image pairs, said target image and said time-shifted reference image and calculating displacements of background detail caused by said refractive wake features. ### Claim 3 (depends on 2) The method for visualizing density changes in air according to claim 2, wherein said processing step comprises the substeps of acquiring a sequence of data relevant to a flight condition, compiling a data array of displacement data from said sequence of data, mathematically averaging at each node the displacement data in the data array; mathematically determining vector of displacement at each node in any arbitrary direction, and mathematically determining at each node an absolute value of displacement. ### Claim 4 (depends on 3) The method for discerning density changes in air according to claim 3, wherein said processing step further comprises a substep of rendering an averaged image of visible density changes in the air. ### Claim 5 (depends on 1) The method for discerning density changes in air according to claim 1, wherein said processing step further comprises a substep of rendering a composite image of visible density changes in the air. ### Claim 6 (depends on 1) The method for discerning density changes in air according to claim 1, wherein said step of capturing a series of image pairs from said sensor airborne vehicle further comprises capturing said series of images when said sensor airborne vehicle is flying above said target airborne vehicle. ### Claim 7 (depends on 1) The method for discerning density changes in air according to claim 1, wherein said step of capturing a series of image pairs from said sensor airborne vehicle further comprises capturing said series of images when said sensor airborne vehicle is flying below said target airborne vehicle. ### Claim 8 (depends on 1) The method for discerning density changes in air according to claim 1, wherein said step of capturing a series of image pairs from said sensor airborne vehicle further comprises capturing said series of images when said sensor airborne vehicle is flying at an oblique angle to said target airborne vehicle relative to said background. ### Claim 9 (depends on 1) The method for discerning density changes in air according to claim 1, wherein said step of capturing a series of image pairs from said sensor airborne vehicle further comprises capturing said series of images when said sensor airborne vehicle is flying at a co-altitude to said target airborne vehicle relative to said background. ### Claim 10 (depends on 1) The method for discerning density changes in air according to claim 1, wherein said step of capturing a series of image pairs from said sensor airborne vehicle further comprises using multiple cameras to capture each image pair. ### Claim 11 (depends on 1) The method for discerning density changes in air according to claim 1, wherein said step of capturing a series of image pairs from said sensor airborne vehicle further comprises, for each image pair, a single camera capturing both said target image and said reference image, wherein said camera captures said target image and said reference image by partitioning each camera frame into a first segment for said target image and a second segment for said reference image. ### Claim 12 (depends on 1) The method for discerning density changes in air according to claim 1, wherein said step of capturing a series of image pairs from said sensor airborne vehicle further comprises, for each image pair, capturing said target image from a first camera operating as said target sensor and capturing said reference image from a second camera. ### Claim 13 (depends on 1) The method of claim 1, wherein the target airborne vehicle is supersonic and the sensor airborne vehicle is supersonic. ### Claim 14 A system for imaging density changes in air caused by a moving target, comprising: a target airborne vehicle flying at a predetermined airspeed and altitude over a background; a sensor airborne vehicle comprising a target sensor and flying in formation with said target airborne vehicle at an airspeed and altitude such that said target airborne vehicle remains fixed in a field-of-view of said target sensor and such that refractive wake features of said target airborne vehicle are included within said field of view of said target sensor; and a computer system, wherein said target sensor is configured to capture a series of image airs from said sensor airborne vehicle, each image pair including a reference image of said background taken in advance of said target airborne vehicle such that said reference image is undisturbed by said target airborne vehicle, and a target image taken with said target airborne vehicle over said background, and wherein said computer system is configured to: for each image pair, time shift said reference image to adjust for a time delay relative to said target image; and process the series of image pairs by analyzing comparative displacement of features in said background between said time-shifted reference images and said target images as a result of said target airborne vehicle's refractive wake features. ### Claim 15 (depends on 14) The system of claim 14, wherein said computer system is further configured to, when processing the series of image pairs, mathematically compare the target images and the reference images and calculate displacements of background detail caused by refractive wake features. ### Claim 16 (depends on 14) The system of claim 14, wherein said computer system is further configured to analyze comparative displacement of features in a plurality of target images relative to their paired reference images. ### Claim 17 (depends on 16) The system of claim 16, wherein said computer system is further configured to average the displacement features. ### Claim 18 (depends on 17) The system of claim 17, wherein said computer system is further configured to render a composite image of visible density changes in the air. ### Claim 19 (depends on 18) The system of claim 18, wherein said target sensor comprises a single camera that captures both the target image and the reference image for each image pair, wherein said camera captures said target image and said reference image by partitioning each camera frame into a first segment for said target image and a second segment for said reference image. ### Claim 20 (depends on 19) The system of claim 19, wherein said computer system is further configured to analyze comparative displacement of features by applying one or combination of a cross-correlation algorithm, direct correlation algorithm, and optical flow algorithm. ### Claim 21 (depends on 14) The system of claim 14, wherein said computer system is further configured to analyze comparative displacement of features by analyzing displacement of features in a target image relative to a plurality of corresponding reference frames. ### Claim 22 (depends on 14) The system of claim 14, wherein the target airborne vehicle is supersonic and the sensor airborne vehicle is supersonic. ### Claim 23 (depends on 14) The system of claim 14, wherein said target sensor comprises multiple cameras.