# US 10,902,884 B2 · Methods and apparatus for ordered serial synchronization of multimedia streams upon sensor changes > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,902,884 B2 - Document ID: US10902884B2 - Status: Granted - Kind: B2 - Application number: 16/690,541 - Filed: 2019-11-21 - Published: 2020-06-25 - Granted: 2021-01-26 - Earliest claimed: 2017-05-09 - Assignees: Echo360, Inc. - Applicants: Echo360, Inc. - Inventors: Mark Reynolds; Shaun Procter; Andrew Cramer; Martin Chudik - Web page: https://patentgrep.com/patent/US10902884B2 ## CPC - G11B 27/34: Indicating arrangements {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)} - H04N 5/268: Signal distribution or switching - G11B 27/10: Indexing; Addressing; Timing or synchronising; Measuring tape travel - G11B 27/3036: {Time code signal} - G11B 27/3081: {used signal is a video-frame or a video-field (P.I.P)} - H04N 5/265: Mixing - H04N 5/77: between a recording apparatus and a television camera - H04N 5/772: {the recording apparatus and the television camera being placed in the same enclosure} - H04N 9/8042: {involving data reduction} - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An apparatus includes a processor with first and second input ports and a memory operably coupled to the processor. The processor can detect streams of media samples at the input ports and determine, in response to the detection of the streams of media samples, a capture start time. The processor can also capture a first frame of a first stream of media samples beginning at the capture start time, and a first frame of a second stream of media samples beginning at a first time subsequent to the capture start time. The processor can also calculate a relative offset time based on the capture start time, the first time, and a rate associated with the second stream of media samples, and store, in the memory, an indication of an association between the captured first frame of the second stream of media samples and the relative offset time. ## Claims ### Claim 1 A non-transitory processor-readable medium storing code representing instructions to be executed by a processor, the code comprising code to cause the processor to: detect, at a first input port, a first stream of media samples; detect, at a second input port, a second stream of media samples; determine, in response to the detection of the first stream of media samples and the second stream of media samples, a capture start time associated with the first stream of media samples; capture, beginning at the capture start time, a first frame of the first stream of media samples; capture, beginning at a first time subsequent to the capture start time, a first frame of the second stream of media samples; calculate a relative offset time based on the capture start time, the first time, and a rate associated with the second stream of media samples; store, in the memory, an indication of an association between the captured first frame of the second stream of media samples and the relative offset time; combine the captured first frame of the first stream of media samples and the captured first frame of the second stream of media samples, based on the relative offset time, to form a synchronized media file; and transmit the synchronized media file to a remote compute device. ### Claim 2 (depends on 1) The non-transitory processor-readable medium of claim 1, the code further comprising code to cause the processor to receive an indication of the capture start time from a control server in operable communication with the processor. ### Claim 3 (depends on 1) The non-transitory processor-readable medium of claim 1, the code further comprising code to cause the processor to receive an indication of the capture start time from a local scheduler. ### Claim 4 (depends on 1) The non-transitory processor-readable medium of claim 1, wherein the relative offset time is a first relative offset time, the code further comprising code to cause the processor to: detect, at a third input port of the processor, a third stream of media samples; capture, beginning at a second time subsequent to the first time, a first frame of the third stream of media samples; calculate a second relative offset time based on the capture start time, the second time, and a rate associated with the third stream of media samples; and store, in the memory, an indication of an association between the captured first frame of the third stream of media samples and the second relative offset time. ### Claim 5 (depends on 1) The non-transitory processor-readable medium of claim 1, the code further comprising code to cause the processor to form the synchronized media file by combining a captured first frame of a third stream of media samples with the captured first frame of the first stream of media samples and the captured first frame of the second stream of media samples. ### Claim 6 (depends on 1) The non-transitory processor-readable medium of claim 1, the code further comprising code to cause the processor to: capture a plurality of additional frames of the first stream of media samples; capture a plurality of additional frames of the second stream of media samples; and form the synchronized media file by combining the plurality of additional frames of the first stream of media samples and the plurality of additional frames of the second stream of media samples with the captured first frame of the first stream of media samples and the captured first frame of the second stream of media samples, based on the relative offset time. ### Claim 7 (depends on 1) The non-transitory processor-readable medium of claim 1, the code further comprising code to cause the processor to identify the first time based on a hardware timestamp encoded by the second stream of media samples. ### Claim 8 (depends on 1) The non-transitory processor-readable medium of claim 1, wherein a frame rate of the first stream of media samples is higher than a frame rate of the second stream of media samples. ### Claim 9 (depends on 1) The non-transitory processor-readable medium of claim 1, wherein the first input port has a first port type, and the second input port has a second port type different from the first port type. ### Claim 10 (depends on 1) The non-transitory processor-readable medium of claim 1, wherein: the first stream of media samples includes one of: sequenced audio samples, sequenced video frames, or sequenced desktop frames; and the second stream of media samples includes a different one of: sequenced audio samples, sequenced video frames, or sequenced desktop frames, as compared with the first stream of media samples. ### Claim 11 A non-transitory processor-readable medium storing code representing instructions to be executed by a processor, the code comprising code to cause the processor to: detect, at a first input port of a processor, a rate of a first stream of media samples; detect, at a second input port of the processor, a rate of a second stream of media samples; detect, at a third input port of the processor, a rate of a third stream of media samples; initiate a capture period having a capture start time based on the first stream of media samples; during the capture period: capture, beginning at the capture start time, a first frame of the first stream of media samples, based on the rate of the first stream of media samples being higher than the rate of the second stream of media samples and the rate of the third stream of media samples; capture, beginning at a first time subsequent to the capture start time, a first frame of the second stream of media samples, based on the rate of the second stream of media samples being higher than the rate of the third stream of media samples; capture, beginning at a second time subsequent to the first time, a first frame of the third stream of media samples; calculate a relative offset time of the captured first frame of the second stream of media samples; and calculate a relative offset time of the captured first frame of the third stream of media samples, and combine the captured first frame of the first stream of media samples, the captured first frame of the second stream of media samples, and the captured first frame of the third stream of media samples, based on the relative offset time of the captured first frame of the second stream of media samples and the relative offset time of the captured first frame of the third stream of media samples, to form a synchronized, streamable media file. ### Claim 12 (depends on 11) The non-transitory processor-readable medium of claim 11, wherein the first stream of media samples includes a plurality of sequenced audio samples, the second stream of media samples includes a plurality of sequenced video frames, and the third stream of media samples includes a plurality of sequenced desktop frames. ### Claim 13 (depends on 11) The non-transitory processor-readable medium of claim 11, wherein the synchronized, streamable media file is a first synchronized, streamable media file, the code further comprising code to cause the processor to: detect a fourth stream of media samples received at the processor via a fourth input port; capture the fourth stream of media samples; and generate a second synchronized, streamable media file that includes the fourth stream of media samples. ### Claim 14 (depends on 11) The non-transitory processor-readable medium of claim 11, wherein the code to cause the processor to initiate the capture period includes code to initiate the capture period in response to a communication received from a local scheduler. ### Claim 15 (depends on 11) The non-transitory processor-readable medium of claim 11, wherein the code to cause the processor to initiate the capture period includes code to initiate the capture period in response to a communication received from a control server. ### Claim 16 (depends on 11) The non-transitory processor-readable medium of claim 11, the code further comprising code to cause the processor to store, in a memory: an indication of an association between the captured first frame of the second stream of media samples and the relative offset time of the captured first frame of the second stream of media samples; and an indication of an association between the captured first frame of the third stream of media samples and the relative offset time of the captured first frame of the third stream of media samples. ### Claim 17 (depends on 11) The non-transitory processor-readable medium of claim 11, wherein the first input port has a first port type, and the second input port has a second port type different from the first port type. ### Claim 18 (depends on 11) The non-transitory processor-readable medium of claim 11, wherein the first input port has a first port type, the second input port has a second port type different from the first port type, and the third input port has a third port type different from the first port type and different from the second port type. ### Claim 19 (depends on 11) The non-transitory processor-readable medium of claim 11, wherein: the first stream of media samples includes one of: sequenced audio samples, sequenced video frames, or sequenced desktop frames; and the second stream of media samples includes a different one of: sequenced audio samples, sequenced video frames, or sequenced desktop frames, as compared with the first stream of media samples. ### Claim 20 (depends on 11) The non-transitory processor-readable medium of claim 11, wherein: the second stream of media samples includes one of: sequenced audio samples, sequenced video frames, or sequenced desktop frames; and the third stream of media samples includes a different one of: sequenced audio samples, sequenced video frames, or sequenced desktop frames, as compared with the second stream of media samples.