# US 9,019,795 B2 · Method of object tracking using sonar imaging > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,019,795 B2 - Document ID: US9019795B2 - Status: Granted - Kind: B2 - Application number: 13/603,723 - Filed: 2012-09-05 - Published: 2014-03-06 - Granted: 2015-04-28 - Earliest claimed: 2012-09-05 - Assignees: Codaoctopus Group - Applicants: Martyn Sloss - Inventors: Martyn Sloss - Web page: https://patentgrep.com/patent/US9019795B2 ## CPC - G01S 15/89: for mapping or imaging - G01S 7/6245: {Stereoscopic displays; Three-dimensional displays; Pseudo-three dimensional displays} - G01S 15/66: Sonar tracking systems - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES ## Abstract An object is measured to record the relative surface coordinates. Then, a portion of the object “the front side” immersed in a fluid is imaged by directing a sonar pulse at the object and recording sonar signals reflected from the object with a sonar imaging array. Then, the recorded relative surface coordinates are iteratively fit to coordinates calculated from the sonar image. Thereafter, the coordinates of the surface of the “backside” of the object that is not observable in the sonar image are known, and a computer generated image of the backside is stitched to sonar image so that the object can be viewed from a plurality of viewpoints separated from the sonar imaging array. ## Claims ### Claim 1 A method of tracking an object in a fluid, comprising: a) recording a first set of three dimensional points representing of a first portion of the surface of the object, the first set of three dimensional points recorded in computer readable media; b) directing sound waves from a source towards at least a second portion of the surface of the object, wherein the object is immersed in a fluid, the sound waves directed at a time t1; then c) receiving sound waves reflected from the second portion of the surface with a two dimensional sonar receiving array, wherein the sonar receiving array communicates signals measuring the reflected sound waves to a computer readable recording device, and wherein the first and second portions of the surface have at least one overlap region; then d) constructing a second set of three dimensional points representing the surface of the second portion of the surface of the object from the signals recorded on the computer readable recording device; then e) iteratively matching the first set of points to the second set of points; then f) displaying a three dimensional sonar image of the second portion of the surface combined with a three dimensional image of a third portion of the surface of the object, the three dimensional image of the third portion of the surface calculated the first set of points, wherein the third portion of the surface is included in the first portion of the surface and not part of the second portion of the surface. ### Claim 2 (depends on 1) The method of claim 1, further comprising; g) displaying a background sonar image of a background behind the image displayed in step f), wherein the background sonar image is recorded before step c). ### Claim 3 (depends on 2) The method of claim 2, wherein step g) comprises displaying sequential images on an image display device, the sequential images comprising images of the mode and background seen from a plurality of points of view different from the position of the sonar receiving array. ### Claim 4 (depends on 2) The method of claim 2, wherein step g) comprises displaying an image of the object and background on a 3D imaging device, wherein two separate images constructed from two different points of view are presented to two eyes of a human observer. ### Claim 5 (depends on 1) The method of claim 1, further comprising rotating and translating the object to bring the object into contact with at least one background object, wherein the fit between the object and the at least one background object satisfies a criterion. ### Claim 6 A method of tracking an object in a fluid, comprising: a) recording relative coordinates of at least a first portion of the surface of the object and creating a model of the first portion of the surface; then b) constructing a first set of three dimensional points representing the first portion of the surface of the model; c) directing sound waves from a source towards at least a second portion of the surface of the object, wherein the object is immersed in a fluid, the sound waves directed at a time t1; then d) receiving sound waves reflected from the second portion of the surface with a two dimensional sonar receiving array, wherein the sonar receiving array communicates signals measuring the reflected sound waves to a computer readable recording device, and wherein the first and second portions of the surface have at least one overlap region; then f) constructing a second set of three dimensional points representing the surface of the second portion of the surface of the object from the recorded signals of the computer readable recording device; then g) estimating the orientation of the model to the object; h) choosing a third set of points to be the points of the first set of points which would be seen by a sonar image of the model estimated be in the orientation of step g); and i) iteratively matching the second set of points with the third set of points; then j) displaying a three dimensional sonar image of the second portion of the surface combined with an image of a third portion of the surface of the object calculated from the model, wherein the third portion of the surface is included in the first portion of the surface and not part of the second portion of the surface. ### Claim 7 (depends on 6) The method of claim 6, further comprising; k) repeating steps c)-j) at time t2, l) estimating rotational and translational velocities of the object from the differences in orientation and position measured at times t1 and t2; m) estimating the position of the object at time t3 from the position measured at time earlier than t3 and the estimated rotational and translational velocities of the object; o) repeating steps c)-j) at time t3, where step g) uses the results of step m). ### Claim 8 (depends on 6) The method of claim 6, wherein step g) uses the vertical line through the known center of mass of the object and a measured height of the center of mass of the object with respect to the source and/or the two dimensional sonar recording array. ### Claim 9 (depends on 6) The method of claim 6, further comprising; k) displaying a background sonar image of a background behind the image displayed in step j), wherein the background sonar image is recorded before step c). ### Claim 10 (depends on 9) The method of claim 9, wherein step k) comprises displaying sequential images on an image display device, the sequential images comprising images of the model and background seen from a plurality of points of view different from the position of the sonar receiving array. ### Claim 11 (depends on 9) The method of claim 9, wherein step k) comprises displaying an image of the object and background on a 3D imaging device, wherein two separate images constructed from two different points of view are presented to two eyes of a human observer. ### Claim 12 (depends on 6) The method of claim 6, further comprising rotating and translating the object to bring the object into contact with at least one background object, wherein the fit between the object and the at least one background object satisfies a criterion. ### Claim 13 A method of tracking an object in a fluid, comprising: a) recording a first set of three dimensional points representing of a first portion of the surface of the object, the first set of three dimensional points recorded in computer readable media; b) directing sound waves from a source towards a background surface, wherein the background surface is immersed in a fluid; then c) receiving sound waves reflected from the background surface with a two dimensional sonar receiving array, wherein the sonar receiving array communicates signals measuring the reflected sound waves to a computer readable recording device; then d) constructing a model of the background surface; e) moving the object into a position between the background surface and the source and/or the sonar receiving array; and wherein the first and second portions of the surface have at least one overlap region f) directing sound waves from a source towards at least a second portion of the surface of the object, wherein the object is immersed in a fluid, and wherein the sound waves are directed at a time t1; then g) receiving sound waves reflected from the second portion of the surface with a two dimensional sonar receiving array, wherein the sonar receiving array communicates signals measuring the reflected sound waves to a computer readable recording device, and wherein the first and second portions of the surface have at least one overlap region; then h) constructing a second set of three dimensional points representing the surface of the second portion of the surface of the object from the recorded signals; then i) iteratively matching the first set of points to the second set of points; then j) translating and rotating the object to fit the object into the background surface, wherein the fit satisfies a criterion.