# US 10,771,669 B1 · Time synchronization of sensor data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,771,669 B1 - Document ID: US10771669B1 - Status: Granted - Kind: B1 - Application number: 16/126,030 - Filed: 2018-09-10 - Granted: 2020-09-08 - Earliest claimed: 2017-09-26 - Assignees: Apple Inc. - Applicants: Koussalya Balasubramanian; Paul A. Baker; Harold C. Ockerse; Brent E. Tweddle - Inventors: Koussalya Balasubramanian; Paul A. Baker; Harold C. Ockerse; Brent E. Tweddle - Web page: https://patentgrep.com/patent/US10771669B1 ## CPC - H04N 23/66: Remote control of cameras or camera parts, e.g. by remote control devices - G01S 7/003: {Transmission of data between radar, sonar or lidar systems and remote stations} - G01S 7/497: Means for monitoring or calibrating - G01S 17/87: Combinations of systems using electromagnetic waves other than radio waves - G01S 17/89: for mapping or imaging - H04N 23/73: by influencing the exposure time - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G01S 13/865: {Combination of radar systems with lidar systems} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES ## Abstract Sensor data produced in a multi-sensor environment is time synchronized, such as using a master time. A component that derives the master time transmits a signal to a first sensor of the multi-sensor environment, which signal indicates the master time. The component subsequently receives a signal from the first sensor, which signal indicates a first time determined by the first sensor based on the master time and a latency associated with the first sensor. The first time indicates a time of an event associated with the first sensor data. A determination is then made that the first sensor data corresponds to second sensor data produced using a second sensor of the multi-sensor environment based on the first time and a second time indicating a time of an event associated with the second sensor data. ## Claims ### Claim 1 A method for time synchronization of sensor data in a multi-sensor environment using a master time, the method comprising: transmitting, from a component that derives the master time, a first signal to a first sensor of the multi-sensor environment and to a second sensor of the multi-sensor environment, the first signal including a master time stamp; receiving, by the component, a second signal from the first sensor, the second signal including first sensor data produced using the first sensor, the second signal including a first time stamp determined by the first sensor based on the master time and a latency associated with the first sensor, the first time stamp indicating a time of an event associated with the first sensor data; receiving, by the component, a third signal from the second sensor, the third signal including second sensor data produced using the second sensor, the third signal including a second time stamp determined by the second sensor based on the master time stamp and a latency associated with the second sensor, the second time stamp indicating a time of an event associated with the second sensor data; and determining that the first sensor data corresponds to the second sensor data based on the first time stamp and the second time stamp. ### Claim 2 (depends on 1) The method of claim 1, wherein the first signal includes a trigger for causing the first sensor to produce the first sensor data. ### Claim 3 (depends on 1) The method of claim 1, further comprising: subsequent to determining that the first sensor data corresponds to the second sensor data, associating the event associated with the first sensor data and the event associated with the second sensor data, wherein determining that the first sensor data corresponds to the second sensor data includes determining that a difference between the first time and the second time is less than a threshold. ### Claim 4 (depends on 1) The method of claim 1, wherein the latency associated with the first sensor changes based on at least one of a condition associated with the multi-sensor environment or a configuration of the first sensor. ### Claim 5 A method for time synchronization of sensor data in a multi-sensor environment using a master time, the method comprising: initializing, by a component that derives the master time, the master time according to a geolocation time obtained using a geolocation component of the multi-sensor environment, the geolocation time indicating a time according to a geolocation system clock external to the multi-sensor environment; transmitting, from the component, a first signal to a first sensor of the multi-sensor environment, the first signal including a master time stamp; receiving, by the component, a second signal from the first sensor, the second signal including first sensor data produced using the first sensor, the second signal including a first time stamp determined by the first sensor based on the master time and a latency associated with the first sensor, the first time stamp indicating a time of an event associated with the first sensor data; and determining that the first sensor data corresponds to second sensor data produced using a second sensor of the multi-sensor environment, the determining based on the first time stamp and a second time stamp indicating a time of an event associated with the second sensor data. ### Claim 6 (depends on 5) The method of claim 5, wherein the first signal includes a trigger for causing the first sensor to produce the first sensor data. ### Claim 7 (depends on 5) The method of claim 5, further comprising: subsequent to determining that the first sensor data corresponds to the second sensor data, associating the event associated with the first sensor data and the event associated with the second sensor data, wherein determining that the first sensor data corresponds to the second sensor data includes determining that a difference between the first time stamp and the second time stamp is less than a threshold. ### Claim 8 (depends on 5) The method of claim 5, wherein the latency associated with the first sensor changes based on at least one of a condition associated with the multi-sensor environment or a configuration of the first sensor. ### Claim 9 A non-transitory computer-readable storage device including program instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations for time synchronization of sensor data in a multi-sensor environment, the operations comprising: transmitting a first signal to a lidar sensor to cause the lidar sensor to produce first sensor data at a first time; transmitting a second signal to a second sensor to cause the second sensor to produce second sensor data at a second time, wherein the second time is determined based on the first time and a phase offset; and associating the first sensor data with the second sensor data based on the first time and the second time. ### Claim 10 (depends on 9) The non-transitory computer-readable storage device of claim 9, the operations further comprising: determining the phase offset based on an angle of the lidar sensor. ### Claim 11 (depends on 9) The non-transitory computer-readable storage device of claim 9, wherein the second sensor is a camera. ### Claim 12 (depends on 11) The non-transitory computer-readable storage device of claim 11, wherein a mid-exposure of the camera is timed according to the phase offset. ### Claim 13 (depends on 9) The non-transitory computer-readable storage device of claim 9, wherein the first signal includes a pulse-per-second signal defining a periodic basis for producing the first sensor data using the lidar sensor and the second signal includes a trigger for causing the second sensor to produce the second sensor data, and the trigger is synchronized to the pulse-per-second signal. ### Claim 14 (depends on 9) The non-transitory computer-readable storage device of claim 9, wherein the phase offset is a first phase offset, the operations further comprising: transmitting a third signal to a third sensor to cause the third sensor to produce third sensor data at a third time, wherein the third time is determined based on the first time and a second phase offset; and associating the first sensor data with the third sensor data based on the first time and the third time. ### Claim 15 A system for time synchronization of sensor data in a multi-sensor environment, comprising: a control component that is configured to: transmit a first signal to a lidar sensor to cause the lidar sensor to produce first sensor data at a first time, transmit a second signal to a second sensor to cause the second sensor to produce second sensor data at a second time, wherein the second time is determined based on the first time and a phase offset, and associate the first sensor data with the second sensor data based on the first time and the second time. ### Claim 16 (depends on 15) The system of claim 15, wherein the control component is further configured to determine the phase offset based on an angle of the lidar sensor. ### Claim 17 (depends on 15) The system of claim 15, wherein the second sensor is a camera. ### Claim 18 (depends on 17) The system of claim 17, wherein a mid-exposure of the camera is timed according to the phase offset. ### Claim 19 (depends on 15) The system of claim 15, wherein the first signal includes a pulse-per-second signal defining a periodic basis for producing the first sensor data using the lidar sensor and the second signal includes a trigger for causing the second sensor to produce the second sensor data, and the trigger is synchronized to the pulse-per-second signal. ### Claim 20 (depends on 15) The system of claim 15, wherein the phase offset is a first phase offset, and the control component is further configured to: transmit a third signal to a third sensor to cause the third sensor to produce third sensor data at a third time, wherein the third time is determined based on the first time and a second phase offset, and associate the first sensor data with the third sensor data based on the first time and the third time. ### Claim 21 A system for time synchronization of sensor data in a multi-sensor environment using a master time, comprising: a component that derives the master time, wherein the component is configured to: transmit a first signal to a first sensor of the multi-sensor environment and to a second sensor of the multi-sensor environment, the first signal including a master time stamp, receive a second signal from the first sensor, the second signal including first sensor data produced using the first sensor, the second signal including a first time stamp determined by the first sensor based on the master time and a latency associated with the first sensor, the first time stamp indicating a time of an event associated with the first sensor data, receive a third signal from the second sensor, the third signal including second sensor data produced using the second sensor, the third signal including a second time stamp determined by the second sensor based on the master time stamp and a latency associated with the second sensor, the second time stamp indicating a time of an event associated with the second sensor data, and determine that the first sensor data corresponds to the second sensor data based on the first time stamp and the second time stamp. ### Claim 22 (depends on 21) The system of claim 21, wherein the first signal includes a trigger for causing the first sensor to produce the first sensor data. ### Claim 23 (depends on 21) The system of claim 21, wherein the component is configured to associate the event associated with the first sensor data and the event associated with the second sensor data subsequent to a determination that the first sensor data corresponds to the second sensor data, wherein the component is configured to determine that the first sensor data corresponds to the second sensor data when a difference between the first time and the second time is less than a threshold. ### Claim 24 (depends on 21) The system of claim 21, wherein the latency associated with the first sensor changes based on at least one of a condition associated with the multi-sensor environment or a configuration of the first sensor. ### Claim 25 (depends on 21) The system of claim 21, wherein the component is further configured to: receive a geolocation time using a geolocation component, the geolocation time indicating a time according to a geolocation system clock external to the multi-sensor environment, and store a difference between the master time and the geolocation time. ### Claim 26 A system for time synchronization of sensor data in a multi-sensor environment using a master time, comprising: a component that derives the master time, wherein the component is configured to: initialize the master time according to a geolocation time obtained using a geolocation component of the multi-sensor environment, the geolocation time indicating a time according to a geolocation system clock external to the multi-sensor environment, transmit a first signal to a first sensor of the multi-sensor environment, the first signal including a master time stamp, receive a second signal from the first sensor, the second signal including first sensor data produced using the first sensor, the second signal including a first time stamp determined by the first sensor based on the master time and a latency associated with the first sensor, the first time stamp indicating a time of an event associated with the first sensor data, and determine that the first sensor data corresponds to second sensor data produced using a second sensor of the multi-sensor environment based on the first time stamp and a second time stamp indicating a time of an event associated with the second sensor data. ### Claim 27 (depends on 26) The system of claim 26, wherein the first signal includes a trigger for causing the first sensor to produce the first sensor data. ### Claim 28 (depends on 26) The system of claim 26, wherein the component is configured to associate the event associated with the first sensor data and the event associated with the second sensor data subsequent to a determination that the first sensor data corresponds to the second sensor data. ### Claim 29 (depends on 28) The system of claim 28, wherein the component is configured to determine that the first sensor data corresponds to the second sensor data when a difference between the first time stamp and the second time stamp is less than a threshold. ### Claim 30 (depends on 26) The system of claim 26, wherein the latency associated with the first sensor changes based on at least one of a condition associated with the multi-sensor environment or a configuration of the first sensor.