# US 2020/0005469 A1 · Event-based feature tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0005469 A1
- Document ID: US20200005469A1
- Status: Published application
- Kind: A1
- Application number: 16/486,066
- Filed: 2018-02-14
- Published: 2020-01-02
- Earliest claimed: 2017-02-14
- Assignees: The Trustees of the University of Pennsylvania
- Applicants: The Trustees of the University of Pennsylvania
- Inventors: Konstantinos Daniilidis; Alex Zihao Zhu; Nikolay Asenov Atanasov
- Web page: https://patentgrep.com/patent/US20200005469A1

## CPC

- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/277: involving stochastic approaches, e.g. using Kalman filters
- G06V 10/24: Aligning, centring, orientation detection or correction of the image
- G06V 10/40: Extraction of image or video features
- G06V 10/757: {Matching configurations of points or features}
- H04N 5/147: {Scene change detection}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/80: Camera processing pipelines; Components thereof
- H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio
- G06T 2207/20164: Salient point detection; Corner detection
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Methods, systems and computer readable media for implementing a soft data association modeled with probabilities. The association probabilities are computed in an intertwined EM scheme with the optical flow computation that maximizes the expectation (marginalization) over all associations. In addition, longer tracks can be enabled by computing the affme deformation with respect to the initial point and use the resulting residual as a measure of persistence. The computed optical flow enables a varying temporal integration different for every feature and sized inversely proportional to the length of the flow. The results can be seen in egomotion and very fast vehicle sequences.

## Claims

### Claim 1

A method for tracking features in image data, the method comprising:
detecting, over a period of time, a plurality of asynchronous events from an asynchronous event-based sensor;
initializing one or more features in the image data by integrating events from an initial temporal window into an initial image and detecting the one or more features in the initial image;
tracking the one or more features after the initial temporal window, including assigning events, corresponding to temporal windows after the initial temporal window, to the one or more features by determining, for each event and each point within the feature, a probability that the event is associated with the point; and
identifying, based on the tracking, motion of one or more physical objects.

### Claim 2 (depends on 1)

The method of claim 1, wherein tracking the one or more features comprises estimating an optical flow for each feature describing motion of the feature in an image plane over a spatiotemporal window.

### Claim 3 (depends on 2)

The method of claim 2, wherein estimating an optical flow comprises solving a constraint based on grouping events generated from the same point over the spatiotemporal window and a corresponding set of events using an Expectation Maximization algorithm.

### Claim 4 (depends on 1)

The method of claim 1, wherein initializing the one or more features comprises, for each feature, determining an optical flow for the feature, reconstructing a set of point projections for the feature using the optical flow, and refining a position of the feature using an Expectation Maximization Iterative Closest Point (EM-ICP) algorithm.

### Claim 5 (depends on 4)

The method of claim 4, wherein initializing the one or more features comprises integrating the events from the initial temporal window with zero flow and selecting, using a corner detector, a plurality of spatial windows containing at least a threshold number of variations in event density for tracking the features.

### Claim 6 (depends on 1)

The method of claim 1, wherein tracking the features comprises asynchronously selecting a size for each temporal window after the initial temporal window.

### Claim 7 (depends on 1)

The method of claim 1, wherein receiving the asynchronous events comprises receiving, over the period of time, pixel data from a camera configured to respond only to changes detected in a view of the camera.

### Claim 8 (depends on 1)

The method of claim 1, wherein tracking the one or more features comprises executing an extended Kalman filter with a structureless measurement model to include inertial measurement data from an inertial measurement unit of the asynchronous event-based sensor.

### Claim 9

A system for tracking features, the system comprising:
at least one processor; and
an event-based feature tracker implemented on the at least one processor and configured for:
detecting, over a period of time, a plurality of asynchronous events from an asynchronous event-based sensor;
initializing one or more features in the image data by integrating events from an initial temporal window into an initial image and detecting the one or more features in the initial image;
tracking the one or more features after the initial temporal window, including assigning events, corresponding to temporal windows after the initial temporal window, to the one or more points within the features by determining, for each event and each point, a probability that the event is associated with the point; and
identifying, based on the tracking, motion of one or more physical objects.

### Claim 10 (depends on 9)

The system of claim 9, wherein tracking the one or more features comprises estimating an optical flow for each feature describing a motion of the feature in an image plane over a spatiotemporal window.

### Claim 11 (depends on 10)

The system of claim 10, wherein estimating an optical flow comprises solving a constraint based on grouping events generated from the same point over the spatiotemporal window and a corresponding set of events using an Expectation Maximization algorithm.

### Claim 12 (depends on 9)

The system of claim 9, wherein initializing the one or more features comprises, for each feature, determining an optical flow for the feature, reconstructing a set of point projections for the feature using the optical flow, and refining a position of the feature using an Expectation Maximization Iterative Closest Point (EM-ICP) algorithm.

### Claim 13 (depends on 12)

The system of claim 12, wherein initializing the one or more features comprises integrating the events from the initial temporal window with zero flow and selecting, using a corner detector, a plurality of spatial windows containing at least a threshold number of variations in event density for tracking the features.

### Claim 14 (depends on 9)

The system of claim 9, wherein tracking the one or more features comprises asynchronously selecting a size for each later temporal window.

### Claim 15 (depends on 9)

The system of claim 9, wherein receiving the asynchronous events comprises receiving, over the period of time, pixel data from a camera configured to respond only to changes detected in a view of the camera.

### Claim 16 (depends on 9)

The system of claim 9, wherein tracking the one or more features comprises executing an extended Kalman filter with a structureless measurement model to include inertial measurement data from an inertial measurement unit of the asynchronous event-based sensor.

### Claim 17

A non-transitory computer readable medium storing executable instructions that when executed by at least one processor of a computer control the computer to perform operations comprising:
detecting, over a period of time, a plurality of asynchronous events from an asynchronous event-based sensor;
initializing one or more features in the image data by integrating events from an initial temporal window into an initial image and detecting the one or more features in the initial image;
tracking the one or more features after the initial temporal window, including assigning events, corresponding to temporal windows after the initial temporal window, to the one or more points within the features by determining, for each event and each point, a probability that the event is associated with the point; and
identifying, based on the tracking, motion of one or more physical objects.

### Claim 18 (depends on 17)

The non-transitory computer readable medium of claim 17, wherein tracking the one or more features comprises estimating an optical flow for each feature describing a motion of the feature in an image plane over a spatiotemporal window.

### Claim 19 (depends on 18)

The non-transitory computer readable medium of claim 18, wherein estimating an optical flow comprises solving a sparsity constraint over the spatiotemporal window and a corresponding set of events using an Expectation Maximization algorithm.

### Claim 20 (depends on 17)

The non-transitory computer readable medium of claim 17, wherein initializing the one or more features comprises, for each feature, determining an optical flow for the feature, reconstructing a set of point projections for the feature using the optical flow, and refining a position of the feature using an Expectation Maximization Iterative Closest Point (EM-ICP) algorithm.
