US 10,368,057 B1Granted

Synchronizing data streams

AssigneeAmazon Technologies, Inc.

InventorsSajesh Kumar Saran, Brad Nathaniel Arnold, Jack G Song, Tyler Ogren Vrooman, Wei Chen, Todd Nelson Schoepflin

Life of this patent

Estimating the term…

Loading patent…

Abstract

This disclosure describes techniques for synchronizing independent data streams. In some instances, a computing device couples to multiple independent sensors, such as cameras, and applies accurate timestamp information to the individual frames of sensor data from the independent sensors. After aligning these data streams by applying these accurate timestamps, the computing device may, in some instances, encode and transmit these timestamped data streams to one or more entities for further processing. In one example, a first camera (e.g., a depth camera configured to generate a depth map) may capture images of an environment, as may a second camera (e.g., an Red-Green-Blue (RGB) camera configured to generate color images). The resulting images may be temporally aligned with one another via the timestamping, and the resulting aligned images from both the depth sensor and the RGB camera may be used to create a three-dimensional (3D) model of the environment.

Claims18

Dependent claims

A system comprising:

a first camera configured to generate depth data of an environment based on acquired depth images;
a second camera configured to acquire color images of the environment;
a computing device coupled to the first camera and the second camera, the computing device comprising:
a pulse-width-modulation (PWM) unit coupled to the first camera and the second camera and configured to generate a pulse and an interrupt at a first time;
one or more processors to receive the interrupt;
memory, coupled to the one or more processors;
a first driver stored in the memory and executable on the one or more processors to receive a depth image acquired by the first camera, the first camera configured to acquire the first depth image in response to receiving the pulse from the PWM unit;
a second driver stored in the memory and executable on the one or more processors to receive a color image acquired by the second camera, the second camera acquiring the color image in response to receiving the pulse from the PWM unit;
a first timestamp queue for storing, in the memory, timestamps for association with depth images acquired by the first camera;
a second timestamp queue for storing, in the memory, timestamps for association with color images acquired by the second camera;
an interrupt service stored in the memory and executable on the one or more processors to:
receive a call from the one or more processors in response to the one or more processors receiving the interrupt;
store, in response to receiving the call:
(i) a first timestamp corresponding to the first time in the first timestamp queue, and (ii) a second timestamp corresponding to the first time in the second timestamp queue;
an application stored in the memory and executable on the one or more processors to:
receive the depth image from the first driver;
read the first timestamp from the first timestamp queue;
add metadata that is based on the first timestamp to the depth image;
receive the color image from the second driver;
read the second timestamp from the second timestamp queue;
and
add metadata that based on the second timestamp to the color image.

19 more elements

A method comprising:

generating, at a first time, a pulse by a trigger unit configured to generate periodic pluses, the trigger unit coupled to a first camera and to a second camera;
receiving, by the first camera, the pulse generated at the first time;
receiving, by the second camera, the pulse generated at the first time;
acquiring first image data by the first camera at least partly in response to receiving the pulse;
acquiring second image data by the second camera at least partly in response to receiving the pulse;
storing, by one or more processors of a computing device, the first image data in a first location;
storing, by the one or more processors, the second image data in a second location;
storing, in a first timestamp queue and by the one or more processors, a first timestamp that is based at least in part on the first time;
storing, in a second timestamp queue and by the one or more processors, a second timestamp that is based at least in part on the first time;
receiving the first image data from the first location;
reading the first timestamp from the first timestamp queue;
associating the first timestamp with the first image data;
receiving the second image data from the second location;
reading the second timestamp from the second timestamp queue;
and
associating the second timestamp with the second image data.

13 more elements

A system comprising:

a first sensor;
a second sensor;
a trigger unit coupled to the first sensor and the second sensor and configured to generate a pulse;
one or more processors;
and
one or more computer-readable media storing computer-executable instructions that, when executed on the one or more processors, cause the one or more processors to perform acts comprising:
acquiring, at a first time, first data using the first sensor at least partly in response to the first sensor receiving the pulse;
acquiring, at or approximately at the first time, second data using the second sensor at least partly in response to the second sensor receiving the pulse;
storing the first data in a first location;
storing the second data in a second location;
storing, in a first timestamp queue, a first timestamp specifying the first time;
storing, in a second timestamp queue, a second timestamp specifying the first time or approximately the first time;
receiving the first data from the first location;
reading the first timestamp from the first timestamp queue;
associating the first timestamp with the first data;
receiving the second data from the second location;
reading the second timestamp from the second timestamp queue;
and
associating the second timestamp with the second data.

16 more elements

Dependencies are inferred from the claim text.