# US 2021/0341922 A1 · Optical object tracking on focal plane with dynamic focal length

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0341922 A1
- Document ID: US20210341922A1
- Status: Published application
- Kind: A1
- Application number: 17/001,309
- Filed: 2020-08-24
- Published: 2021-11-04
- Earliest claimed: 2020-05-04
- Assignees: Honeywell International Inc.
- Applicants: Honeywell International Inc.
- Inventors: Shouvik Das; Nikhil Gupta
- Web page: https://patentgrep.com/patent/US20210341922A1

## CPC

- G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target}
- G01S 3/7864: {T.V. type tracking systems}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/73: using feature-based methods
- B64D 47/08: Arrangements of cameras
- B64U 2101/30: for imaging, photography or videography
- G06T 2207/10032: Satellite or aerial image; Remote sensing
- G06T 2207/30244: Camera pose
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B64D: EQUIPMENT FOR FITTING IN OR TO AIRCRAFT; FLIGHT SUITS; PARACHUTES; ARRANGEMENT OR MOUNTING OF POWER PLANTS OR PROPULSION TRANSMISSIONS IN AIRCRAFT
- B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR

## Abstract

A method of tracking an image feature is described. The method comprises acquiring an image with a camera, and determining, using processing circuitry, a bounding area in the image, the bounding area surrounding a feature in the image. The method further comprises determining, using processing circuitry, a rotation axis and a rotation angle based on a first focal length of the camera and a position of the bounding area relative to a center of the image. The method further comprises determining, using processing circuitry, at least one of a pan angle, a roll angle, and a tilt angle for the camera at which the bounding area is centered in the image. The method further comprises adjusting, using a gimbal, an orientation of the camera based on the at least one of the pan angle, the roll angle, and the tilt angle.

## Claims

### Claim 1

A method of tracking an image feature, the method comprising:
acquiring an image with a camera;
determining, using processing circuitry, a bounding area in the image, the bounding area surrounding a feature in the image;
determining, using processing circuitry, a rotation axis and a rotation angle based on a first focal length of the camera and a position of the bounding area relative to a center of the image;
determining, using processing circuitry, at least one of a pan angle, a roll angle, and a tilt angle for the camera at which the bounding area is centered in the image; and
adjusting, using a gimbal, an orientation of the camera based on the at least one of the pan angle, the roll angle, and the tilt angle.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the rotation axis and the rotation angle further comprises determining a position of a centroid of the bounding area in the image relative to the center of the image.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining at least one of the pan angle, the roll angle, and the tilt angle for the camera at which the bounding area is centered in the image further comprises:
determining a quaternion in an actuation frame based on the rotation axis and the rotation angle;
converting the quaternion to earth-frame coordinates; and
determining the at least one of the pan angle, the roll angle, and the tilt angle for the camera at which the bounding area is centered in the image based on the quaternion in the earth-frame coordinates.

### Claim 4 (depends on 1)

The method of claim 1, wherein adjusting the orientation of the camera based the at least one of the pan angle, the roll angle, and the tilt angle results in a centroid of the bounding area being substantially co-located with the center of the image.

### Claim 5 (depends on 1)

The method of claim 1, wherein adjusting the orientation of the camera based on the at least one of the pan angle, the roll angle, and the tilt angle comprises servoing the gimbal while maintaining a servoing control-loop gain.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
acquiring a second image with the camera having a second focal length different from the first focal length;
determining, using processing circuitry, a bounding area in the second image, the bounding area surrounding the feature in the image;
determining, using processing circuitry, a second rotation axis and a second rotation angle based on the second focal length and a position of the bounding area relative to a center of the second image;
determining, using processing circuitry, at least one of a pan angle, a roll angle, and a tilt angle for the camera at which the bounding area is centered in the second image; and
adjusting, using the gimbal, an orientation of the camera based on the at least one of the pan angle, the roll angle, and the tilt angle.

### Claim 7 (depends on 2)

The method of claim 2, further comprising:
determining, using processing circuitry, a second bounding area in the image, the second bounding area surrounding a second feature in the image,
wherein determining a position of a centroid of the bounding area in the image relative to the center of the image comprises determining a position of a global centroid of a global bounding area based on the centroids of the first and second bounding areas.

### Claim 8

A system comprising:
a camera; and
processing circuitry configured to:
receive, from the camera, an image including at least one feature;
determine a bounding area in the image surrounding the at least one feature;
determine a rotation axis and a rotation angle based on a first focal length of the camera and a position of the bounding area relative to a center of the image; and
determine at least one of a pan angle, a roll angle, and a tilt angle for the camera at which the bounding area is centered in the image; and
adjust, using a gimbal, an orientation of the camera about the rotation axis and the rotation angle based on the at least one of the pan angle, the roll angle, and the tilt angle.

### Claim 9 (depends on 8)

The system of claim 8, wherein the processing circuitry is further configured to determine the rotation axis and the rotation angle via determining a position of a centroid of the bounding area in the image relative to the center of the image.

### Claim 10 (depends on 9)

The system of claim 9, wherein the processing circuitry is further configured to:
determine a quaternion in an actuation frame based on the rotation axis and the rotation angle;
convert the quaternion to earth-frame coordinates; and
determine at least one of the pan angle, the roll angle, and the tilt angle for the camera at which the bounding area is centered in the image based on the quaternion in the earth-frame coordinates.

### Claim 11 (depends on 8)

The system of claim 8, wherein the camera is attached to the gimbal, and wherein the gimbal is attached to a vehicle.

### Claim 12 (depends on 11)

The system of claim 11, wherein the vehicle comprises an unmanned aerial vehicle.

### Claim 13 (depends on 10)

The system of claim 10, wherein adjustment of the orientation of the camera via the gimbal results in the centroid of the bounding area being substantially co-located with the center of the image.

### Claim 14 (depends on 8)

The system of claim 8, wherein adjustment the orientation of the camera comprises servoing the gimbal while maintaining a servoing control-loop gain.

### Claim 15 (depends on 8)

The system of claim 8, wherein the processing is further configured to:
acquire a second image with the camera having a second focal length different from the first focal length;
determine a bounding area in the second image, the bounding area surrounding the feature in the image;
determine a second rotation axis and a second rotation angle based on the second focal length and a position of the bounding area relative to a center of the second image;
determine at least one of a pan angle, a roll angle, and a tilt angle for the camera at which the bounding area is centered in the second image; and
adjust, using the gimbal, an orientation of the camera based on the at least one of the pan angle, the roll angle, and the tilt angle.

### Claim 16 (depends on 8)

The system of claim 8, wherein the processing circuitry is further configured to:
automatically determine at least one of a pan angle, a roll angle, and a tilt angle for the camera at which the bounding area is centered in the image after a first lens of the camera has been replaced by a second lens having a second focal length different from a first focal length of the first lens.

### Claim 17 (depends on 8)

The system of claim 8, wherein the processing circuitry is further configured to:
acquire a second image with a second camera replacing the camera;
determine a bounding area in the second image, the bounding area surrounding the feature in the image;
determine a second rotation axis and a second rotation angle based on a focal length of the second camera and a position of the bounding area relative to a center of the second image;
determine at least one of a pan angle, a roll angle, and a tilt angle for the second camera at which the bounding area is centered in the second image; and
adjust, using the gimbal, an orientation of the second camera based on the at least one of the pan angle, the roll angle, and the tilt angle.

### Claim 18 (depends on 11)

The system of claim 11, wherein the processing circuitry is further configured to:
determine a second bounding area in the image, the second bounding area surrounding a second feature in the image,
wherein determining a position of a centroid of the bounding area in the image relative to the center of the image comprises determining a position of a global centroid of a global bounding area based on the centroids of the first and second bounding areas.

### Claim 19

A method of servoing a camera, the method comprising:
determining a bounding area, the bounding area including a feature in a sequence of images of a camera;
determining a rotation axis and a rotation angle based on a focal length of the camera and a position of the bounding area relative to a center of one or more images of the sequence of images;
determining at least one of a pan angle, a roll angle, and a tilt angle such that the bounding area is centered in the one or more images based on the rotation axis and the rotation angle and without basing the at least one of the pan angle, the roll angle, and the tilt angle on the a between the feature and the camera; and
servoing the camera based on the at least one of the pan angle, the roll angle, and the tilt angle such that the bounding area is centered in the one or more images.

### Claim 20 (depends on 19)

The method of claim 19, wherein servoing the camera occurs without changing a servoing control-loop gain.
