# US 2020/0404245 A1 · Mapping and tracking system with features in three-dimensional space

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0404245 A1
- Document ID: US20200404245A1
- Status: Published application
- Kind: A1
- Application number: 17/010,727
- Filed: 2020-09-02
- Published: 2020-12-24
- Earliest claimed: 2011-08-04
- Assignees: Trx Systems, Inc.
- Applicants: Trx Systems, Inc.
- Inventors: John George Karvounis
- Web page: https://patentgrep.com/patent/US20200404245A1

## CPC

- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- G06F 18/21355: {nonlinear criteria, e.g. embedding a manifold in a Euclidean space}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/277: involving stochastic approaches, e.g. using Kalman filters
- G06T 7/85: {Stereo camera calibration}
- G06V 20/64: Three-dimensional [3D] objects
- H04N 5/2224: {related to virtual studio applications}
- G06T 2200/28: involving image processing hardware
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20088: Trinocular vision calculations; trifocal tensor
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 2201/11: Technique with transformation invariance effect
- H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence)
- H04N 13/246: Calibration of cameras
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 2013/0092: {Image segmentation from stereoscopic image signals}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

LK-SURF, Robust Kalman Filter, HAR-SLAM, and Landmark Promotion SLAM methods are disclosed. LK-SURF is an image processing technique that combines Lucas-Kanade feature tracking with Speeded-Up Robust Features to perform spatial and temporal tracking using stereo images to produce 3D features can be tracked and identified. The Robust Kalman Filter is an extension of the Kalman Filter algorithm that improves the ability to remove erroneous observations using Principal Component Analysis and the X84 outlier rejection rule. Hierarchical Active Ripple SLAM is a new SLAM architecture that breaks the traditional state space of SLAM into a chain of smaller state spaces, allowing multiple tracked objects, multiple sensors, and multiple updates to occur in linear time with linear storage with respect to the number of tracked objects, landmarks, and estimated object locations. In Landmark Promotion SLAM, only reliable mapped landmarks are promoted through various layers of SLAM to generate larger maps.

## Claims

### Claim 1

A method for managing objects in a multi-dimensional space when the objects are no longer visible in a collection of images captured from the multi-dimensional space over time, the method comprising:
identifying one or more objects visible in a first image among the collection of images;
adding the identified objects to a first list of tracked identified objects;
tracking the identified objects from the first list of objects in one or more subsequent images among the collection of images, the one or more subsequent images being captured after capturing the first image;
determining whether the tracked identified objects are absent in the one or more subsequent images; and
removing, based on the determination, the tracked identified objects from the first list.

### Claim 2 (depends on 1)

The method of claim 1 further comprising:
filtering outlier objects in the first image to create a second list comprising the outlier objects; and
tracking the outlier objects in the second list independently of the identified objects in the first list.

### Claim 3 (depends on 1)

The method of claim 1, wherein the first list represents a state space of the multi-dimensional space, the state space comprising object identifiers, a state vector, and a covariance matrix.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
determining whether an object is visible in the one or more subsequent images;
determining whether the object is absent in the first image; and
adding the object to the first list.

### Claim 5 (depends on 1)

The method of claim 1 further comprising correcting locations of tracked identified objects.

### Claim 6 (depends on 5)

The method of claim 5, wherein correcting locations of tracked identified objects comprise processing the tracked identified objects through an extended Kalman filter for simultaneous localization and mapping.

### Claim 7 (depends on 1)

The method of claim 1, wherein tracking the identified objects from the first list comprises determining whether a tracked identified object from the first list is an unreliable object and removing the tracked identified object from the first list based on the determination.

### Claim 8 (depends on 1)

The method of claim 1 further comprising saving the removed object in a second list.

### Claim 9 (depends on 1)

The method of claim 1 further comprising:
mapping a high-dimensional feature into a one-dimensional space, wherein the high-dimensional feature is associated with one of the tracked identified objects captured from a high-dimensional observation; and
applying an outlier rejection rule in the one-dimensional space to remove the mapped feature.

### Claim 10 (depends on 9)

The method of claim 9, wherein mapping the high-dimensional feature into a one-dimensional space comprises applying a weighting metric to the high-dimensional observation, the weighting metric being derived from an observation covariance and an expectation covariance.

### Claim 11 (depends on 10)

The method of claim 10 further comprising:
determining an observation covariance associated with the high-dimensional observation;
predicting an observation and covariance, the predicted observation being related to the high-dimensional observation, the predicted covariance being associated with the predicted observation;
extracting the high-dimensional feature from the high-dimensional observation;
determining a first covariance for the high-dimensional feature, the determination being based on the observation covariance;
determining a second covariance for the high-dimensional feature, the determination being based on the predicted covariance; and
utilizing a weighting metric in the outlier rejection rule, the weighting metric being derived from the first covariance and the second covariance.

### Claim 12 (depends on 11)

The method of claim 11 further comprising determining first and second state matrices, the first state matrix being associated with the high-dimensional observation and the second state matrix being associated with the predicted observation, and wherein the first and second matrices are Jacobian matrices.

### Claim 13 (depends on 9)

The method of claim 9, wherein the outlier rule comprises using a median of a mapped feature and a median absolute deviation of the mapped feature to determine whether the mapped feature disagrees with a general affine transformation of the observation.

### Claim 14

A system managing objects in a multi-dimensional space when the objects are no longer visible in a collection of images captured from the multi-dimensional space over time, the system comprising:
a processor; and
a memory communicatively coupled to the processor when the system is operational, the memory bearing processor instructions that, when executed on the processor, cause the system to at least:
identify one or more objects visible in a first image among the collection of images;
add the identified objects to a first list of tracked identified objects;
track the identified objects from the first list of objects in one or more subsequent images among the collection of images, the one or more subsequent images being captured after capturing the first image;
determine whether the tracked identified objects are absent in the one or more subsequent images; and
remove, based on the determination, the tracked identified objects from the first list.

### Claim 15 (depends on 14)

The system of claim 14, wherein the memory bears processor instructions that, when executed on the processor, further cause the system to at least:
filter outlier objects in the first image to create a second list comprising the outlier objects; and
track the outlier objects in the second list independently of the identified objects in the first list.

### Claim 16 (depends on 14)

The system of claim 14, wherein the first list represents a state space of the multi-dimensional space, the state space comprising object identifiers, a state vector, and a covariance matrix.

### Claim 17 (depends on 14)

The system of claim 14, wherein the memory bears processor instructions that, when executed on the processor, further cause the system to at least:
determine whether an object is visible in the one or more subsequent images;
determine whether the object is absent in the first image; and
add the object to the first list.

### Claim 18 (depends on 14)

The system of claim 14, wherein the memory bears processor instructions that, when executed on the processor, further cause the system to at least:
correct locations of tracked identified objects; and
process the tracked identified objects through an extended Kalman filter for simultaneous localization and mapping.

### Claim 19 (depends on 14)

The system of claim 14, wherein the instructions that cause the system to track the identified objects from the first list comprise: determine whether a tracked identified object from the first list is an unreliable object and remove the tracked identified object from the first list based on the determination.

### Claim 20

A method for filtering outlier features from captured high-dimensional observations, the method comprising:
mapping a high-dimensional feature into a one-dimensional space;
determining an observation covariance associated with the high-dimensional observation;
predicting an observation and covariance, the predicted observation being related to the high-dimensional observation, the predicted covariance being associated with the predicted observation;
extracting the high-dimensional feature from the high-dimensional observation;
determining a first covariance for the high-dimensional feature, the determination being based on the observation covariance;
determining a second covariance for the high-dimensional feature, the determination being based on the predicted covariance; and
applying an outlier rejection rule in the one-dimensional space to remove the mapped feature, the outlier rejection rule utilizing a weighting metric derived from the first covariance and the second covariance.
