# US 11,451,688 B2 · Image scan line timestamping > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,451,688 B2 - Document ID: US11451688B2 - Status: Granted - Kind: B2 - Application number: 16/142,865 - Filed: 2018-09-26 - Published: 2020-03-26 - Granted: 2022-09-20 - Earliest claimed: 2018-09-26 - Assignees: Zoox, Inc. - Applicants: Zoox, Inc. - Inventors: Arthur Benemann; Christopher Allen Johnson; Turhan Karadeniz - Web page: https://patentgrep.com/patent/US11451688B2 ## CPC - G01S 17/931: of land vehicles - H04N 5/04: Synchronising (for television systems using pulse code modulation H04N7/56) - G01S 7/4972: {Alignment of sensor} - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/89: for mapping or imaging - G06T 5/80: Geometric correction - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06T 7/74: {involving reference images or patches} - H04N 23/60: Control of cameras or camera modules - H04N 23/6812: {based on additional sensors, e.g. acceleration sensors} - H04N 23/689: {Motion occurring during a rolling shutter mode} - H04N 25/531: by controlling rolling shutters in CMOS SSIS - G06T 3/14: Transformations for image registration, e.g. adjusting or mapping for alignment of images - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30241: Trajectory - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Techniques are disclosed for adding time data to scan lines of an image frame. In some examples, an image sensor may perform a rolling shutter image capture to produce the scan lines. Data captured by another sensor may be associated with at least a portion of a scan line based at least in part on the time data added to the scan line in some examples. Furthermore, techniques are disclosed for synchronizing data capture by multiple sensors. For example, a rolling shutter image capture performed by an image sensor may be synchronized with a data capture performed by another sensor. ## Claims ### Claim 1 A system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising: receiving, from an image sensor, first scan line data of an image frame associated with a rolling shutter image capture; appending first time data prior to first end of line data of the first scan line data; adjusting, based at least in part on a size of the first time data, a width associated with the first scan line data; receiving, from the image sensor, second scan line data of the image frame associated with the rolling shutter image capture; and appending second time data prior to second end of line data of the second scan line data. ### Claim 2 (depends on 1) The system of claim 1, the operations further comprising: receiving, from a LIDAR sensor, LIDAR data comprising a LIDAR point, the LIDAR data associated with third time data; and associating, based at least in part on the first time data and the third time data, the LIDAR point with at least a portion of the first scan line data. ### Claim 3 (depends on 1) The system of claim 1, wherein: the first scan line data is associated with a first exposure time; and the second scan line data is associated with a second exposure time that is different than the first exposure time. ### Claim 4 (depends on 1) The system of claim 1, the operations further comprising: modifying, based at least in part on the first time data, at least a portion of an image to correct or compensate for a distortion effect caused by motion associated with the rolling shutter image capture, to produce a modified image, wherein the distortion effect comprises one or more of: wobble; skew; spatial aliasing; or temporal aliasing; and controlling, based at least in part on the modified image, movement of an autonomous vehicle. ### Claim 5 (depends on 1) The system of claim 1, further comprising: a precision time protocol (PTP) clock; and a system processor clock that is synchronized with the PTP clock; wherein the operations further comprise: obtaining data from a counter that is slaved to the system processor clock; and determining, based at least in part on the data, the first time data. ### Claim 6 (depends on 1) The system of claim 1, wherein the first time data is appended after the first scan line data and prior to the first end of line data. ### Claim 7 A method comprising: receiving, from an image sensor, first scan line data of an image frame associated with a rolling shutter image capture; appending first time data prior to first end of line data of the first scan line data; adjusting, based at least in part on a size of the first time data, a width associated with the first scan line data; receiving, from the image sensor, second scan line data of the image frame associated with the rolling shutter image capture; and appending second time data prior to second end of line data of the second scan line data. ### Claim 8 (depends on 7) The method of claim 7, further comprising: receiving, from a LIDAR sensor, LIDAR data comprising a LIDAR point, the LIDAR data associated with third time data; and associating, based at least in part on the first time data and the third time data, the LIDAR point with at least a portion of the first scan line data. ### Claim 9 (depends on 7) The method of claim 7, wherein: the first scan line data is associated with a first exposure time; and the second scan line data is associated with a second exposure time that is different than the first exposure time. ### Claim 10 (depends on 7) The method of claim 7, further comprising: modifying, based at least in part on the first time data, at least a portion of an image to correct or compensate for a distortion effect caused by motion associated with the rolling shutter image capture, to produce a modified image, wherein the distortion effect comprises one or more of: wobble; skew; spatial aliasing; or temporal aliasing; and controlling, based at least in part on the modified image, movement of an autonomous vehicle. ### Claim 11 (depends on 7) The method of claim 7, further comprising obtaining data from a counter that is synced to a system processor clock; and determining, based at least in part on the data, the first time data. ### Claim 12 (depends on 7) The method of claim 7, wherein the first scan line data is associated with a first sensor and the second scan line data is associated with a second sensor. ### Claim 13 (depends on 12) The method of claim 12, wherein: the first sensor comprises an image sensor; and the second sensor comprises at least one of: an additional image sensor; a LIDAR sensor; a radar sensor; or a time-of-flight sensor. ### Claim 14 (depends on 12) The method of claim 12, further comprising: modifying, based at least in part on the first time data, at least a portion of an image to correct or compensate for a distortion effect caused by motion associated with the rolling shutter image capture. ### Claim 15 One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising: receiving, from an image sensor, first scan line data of an image frame associated with a rolling shutter image capture; appending first time data prior to first end of line data of the first scan line data; adjusting, based at least in part on a size of the first time data, a width associated with the first scan line data; receiving, from the image sensor, second scan line data of the image frame associated with the rolling shutter image capture; and appending second time data prior to second end of line data of the second scan line data. ### Claim 16 (depends on 15) The one or more non-transitory computer-readable media of claim 15, wherein: receiving, from a LIDAR sensor, LIDAR data comprising a LIDAR point, the LIDAR data associated with third time data; and associating, based at least in part on the first time data and the third time data, the LIDAR point with at least a portion of the first scan line data. ### Claim 17 (depends on 15) The one or more non-transitory computer-readable media of claim 15, wherein: the first scan line data is associated with a first exposure time; and the second scan line data is associated with a second exposure time that is different than the first exposure time. ### Claim 18 (depends on 15) The one or more non-transitory computer-readable media of claim 15, further comprising: modifying, based at least in part on the first time data, at least a portion of an image to correct or compensate for a distortion effect caused by motion associated with the rolling shutter image capture, to produce a modified image, wherein the distortion effect comprises one or more of: wobble; skew; spatial aliasing; or temporal aliasing; and controlling, based at least in part on the modified image, movement of an autonomous vehicle. ### Claim 19 (depends on 15) The one or more non-transitory computer-readable media of claim 15, further comprising: obtaining data from a counter that is synced to a system processor clock; and determining, based at least in part on the data, the first time data. ### Claim 20 (depends on 15) The one or more non-transitory computer-readable media of claim 15, wherein the first scan line data is associated with a first sensor and the second scan line data is associated with a second sensor.