Cmos-assisted inside-out dynamic vision sensor tracking for low power mobile platforms

A method, comprising:

• receiving, at a hybrid feature handler of an apparatus, an output of a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data;
• receiving, at the hybrid feature handler, an output of a complementary metal-oxide-semiconductor (CMOS) image sensor, the CMOS image sensor configured to output frames of image data;
• determining, by the hybrid feature handler, based on one or more of the output of the DVS sensor or the output of the CMOS image sensor, tracked features;
• providing the tracked features to a visual-inertial simultaneous location and mapping (SLAM) pipeline performing inside-out device tracking;
• and
• switching off the CMOS image sensor, by a sensor scheduler, based on a current value of one or more CMOS control factors.

A non-transitory computer-readable medium comprising program code, which when executed by a processor, causes the apparatus to:

• receive, at a hybrid feature handler of an apparatus, an output of a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data;
• receive, at the hybrid feature handler, an output of a complementary metal-oxide-semiconductor (CMOS) image sensor, the CMOS image sensor configured to output frames of image data;
• determine, by the hybrid feature handler, based on one or more of the output of the DVS sensor or the output of the CMOS image sensor, tracked features;
• provide the tracked features to a visual-inertial simultaneous location and mapping (SLAM) pipeline performing inside-out device tracking;
• and
• switch off the CMOS image sensor, by a sensor scheduler, based on a current value of one or more CMOS control factors.

An untethered apparatus for performing inside-out device tracking based on visual-inertial simultaneous location and mapping (SLAM) comprising:

• a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data from within a first field of view;
• an inertial measurement unit (IMU) sensor configured to collect IMU data associated with motion of the apparatus at a predetermined interval;
• a processor;
• and
• a memory containing instructions, which when executed by the processor, cause the apparatus to:
• accumulate DVS sensor output over a sliding time window, the sliding time window including the predetermined interval,
• apply a motion correction to the accumulated DVS sensor output, the motion correction based on the IMU data collected over the predetermined interval,
• generate an event-frame histogram of DVS sensor events based on the motion correction, and
• provide the event-frame histogram of the DVS sensor events and the IMU data to a visual inertial SLAM pipeline.