US 11,379,996 B2Granted

Deformable object tracking

AssigneeApple Inc.

InventorsPeter Kaufmann, Daniel Kurz, Brian Amberg, Yanghai Tsin

Life of this patent

Estimating the term…

Loading patent…

Abstract

Various implementations disclosed herein include devices, systems, and methods that use event camera data to track deformable objects such as faces, hands, and other body parts. One exemplary implementation involves receiving a stream of pixel events output by an event camera. The device tracks the deformable object using this data. Various implementations do so by generating a dynamic representation of the object and modifying the dynamic representation of the object in response to obtaining additional pixel events output by the event camera. In some implementations, generating the dynamic representation of the object involves identifying features disposed on the deformable surface of the object using the stream of pixel events. The features are determined by identifying patterns of pixel events. As new event stream data is received, the patterns of pixel events are recognized in the new data and used to modify the dynamic representation of the object.

Claims29

Dependent claims

A system comprising:

an event camera comprising a two-dimensional (2D) array of pixel sensors;
non-transitory computer-readable storage medium;
and
one or more processors communicatively coupled to the non-transitory computer-readable storage medium and the event camera, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations, the operations comprising:
receiving a stream of pixel events output by the event camera, the event camera comprising a plurality of pixel sensors positioned to receive light from a scene disposed within a field of view of the event camera, each respective pixel event generated in response to a respective pixel sensor detecting a change in light intensity that exceeds a comparator threshold;
identifying features disposed on a deformable surface of the object using the stream of pixel events;
and
generating a dynamic representation of the object, the dynamic representation comprising the features;
modifying the dynamic representation of the object in response to obtaining additional pixel events output by the event camera, wherein the dynamic representation of the object is a 3D representation having 3D portions that are modified to track deformation of a surface of the object varying over time;
and
outputting the dynamic representation of the object for further processing.

8 more elements

A method for deformable object tracking, the method comprising:

at a device with one or more processors and a non-transitory computer-readable storage medium:
receiving a stream of pixel events output by an event camera, the event camera comprising a plurality of pixel sensors positioned to receive light from a scene disposed within a field of view of the event camera, each respective pixel event generated in response to a respective pixel sensor detecting a change in light intensity that exceeds a comparator threshold;
generating a dynamic representation of an object in the scene using the stream of pixel events, the object having a deformable surface that varies over time;
and
modifying the dynamic representation of the object in response to obtaining additional pixel events output by the event camera, wherein the dynamic representation of the object is a 3D representation having 3D portions that are modified to track deformation of a surface of the object varying over time.

2 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/761,582Filed 13 Nov 2018 · us provisional application
Priority: 14 Nov 2017
Published: US 2020/0273180 A127 Aug 2020
Granted: 5 Jul 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification11 codes

G06T 7/246
Analysis of motion: using feature-based methods, e.g. the tracking of corners or segments
G06T 7/248
Using feature-based methods, e.g. the tracking of corners or segments: involving reference images or patches
G06T 7/251
Using feature-based methods, e.g. the tracking of corners or segments: involving models
G06T 7/292
Analysis of motion: multi-camera tracking