# US 11,451,931 B1 · Multi device clock synchronization for sensor data fusion > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,451,931 B1 - Document ID: US11451931B1 - Status: Granted - Kind: B1 - Application number: 17/112,376 - Filed: 2020-12-04 - Granted: 2022-09-20 - Earliest claimed: 2018-09-28 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Sriram Hariharan; John D. Blackwell; Jonathan C. Shavit; Jay N. Bruins; Carter B. Peterson; Daniel E. Loffgren; Akifumi Kobashi; Jacob S. Weiss; Todd Wheeler - Web page: https://patentgrep.com/patent/US11451931B1 ## CPC - H04W 4/029: Location-based management or tracking services - G06F 18/25: Fusion techniques - G06F 18/254: {of classification results, e.g. of results related to same input data} - H04W 4/38: for collecting sensor information - H04W 56/001: {Synchronization between nodes} - H04W 64/00: Locating users or terminals {or network equipment} for network management purposes, e.g. mobility management - H04W 76/10: Connection setup - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Immersive audio can be generated and/or updated in real-time as an accessory device presenting the audio is moved with respect to a computing device presenting accompanying video content. This real-time immersive audio is enabled by determining positions of the accessory device with respect to the computing device, based on real-time analysis of sensor information from the accessory device and the computing device. Accurate positions can be determined by synchronizing timestamped sensor data from the multiple devices through the use of a clock of a common wireless communication link (e.g., a Bluetooth connection), which may have lower drift than the global clocks of the devices themselves. Calculated offsets associated with differences between the clock of the wireless communication link and the global clocks of the multiple devices can be used to account for inaccuracies in the global clocks of the multiple devices with respect to one another. ## Claims ### Claim 1 A computing device comprising: one or more data processors; and a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations including: establishing, on the computing device, a wireless communication link between the computing device and an accessory device; transmitting an audio signal using the wireless communication link; receiving sensor data from a sensor module of the accessory device or the computing device, wherein the accessory device is movable with respect to the computing device; and generating position information of the accessory device with respect to the computing device using the sensor data, wherein the position information is usable for generating an updated audio signal corresponding to the position information. ### Claim 2 (depends on 1) The computing device of claim 1, wherein the sensor data is associated with a position of the computing device. ### Claim 3 (depends on 1) The computing device of claim 1, wherein the receiving of the sensor data comprises receiving the sensor data using the wireless communication link. ### Claim 4 (depends on 1) The computing device of claim 1, wherein the operations further comprise: generating the updated audio signal using the position information, wherein the updated audio signal is a spatial audio signal corresponding to the position information; and transmitting the updated audio signal using the wireless communication link. ### Claim 5 (depends on 1) The computing device of claim 1, wherein the operations further comprise transmitting the position information using the wireless communication link, wherein the position information is usable to generate the updated audio signal from the audio signal when received by the accessory device. ### Claim 6 (depends on 5) The computing device of claim 5, wherein the audio signal includes a plurality of sub-signals, and wherein generating the updated audio signal includes interpolating at least one of the plurality of sub-signals. ### Claim 7 (depends on 1) The computing device of claim 1, wherein the operations further comprise displaying video content associated with the audio signal, wherein generating the position information of the accessory device comprises predicting a future orientation of the accessory device with respect to the video content, and wherein generating the updated audio signal comprises generating an immersive audio signal based on the future orientation of the accessory device with respect to the video content. ### Claim 8 (depends on 1) The computing device of claim 1, wherein the sensor module an inertial measurement unit of the accessory device. ### Claim 9 (depends on 1) The computing device of claim 1, wherein the sensor module is an imaging sensor of the computing device. ### Claim 10 (depends on 9) The computing device of claim 9, wherein the imaging sensor is a ranging sensor. ### Claim 11 A computer-implemented method, comprising: establishing, on a computing device, a wireless communication link between the computing device and an accessory device; transmitting an audio signal using the wireless communication link; receiving sensor data from a sensor module of the accessory device or the computing device, wherein the accessory device is movable with respect to the computing device; and generating position information of the accessory device with respect to the computing device using the sensor data associated with the accessory device, wherein the position information is usable for generating an updated audio signal corresponding to the position information. ### Claim 12 (depends on 11) The method of claim 11, wherein the sensor data is associated with a position of the computing device. ### Claim 13 (depends on 11) The method of claim 11, wherein the receiving of the sensor data comprises receiving the sensor data using the wireless communication link. ### Claim 14 (depends on 11) The method of claim 11, further comprising: generating the updated audio signal using the position information, wherein the updated audio signal is a spatial audio signal corresponding to the position information; and transmitting the updated audio signal using the wireless communication link. ### Claim 15 (depends on 11) The method of claim 11, further comprising transmitting the position information using the wireless communication link, wherein the position information is usable to generate the updated audio signal from the audio signal when received by the accessory device. ### Claim 16 (depends on 15) The method of claim 15, wherein the audio signal includes a plurality of sub-signals, and wherein generating the updated audio signal includes interpolating at least one of the plurality of sub-signals. ### Claim 17 (depends on 11) The method of claim 11, further comprising displaying video content associated with the audio signal, wherein generating the position information of the accessory device comprises predicting a future orientation of the accessory device with respect to the video content, and wherein generating the updated audio signal comprises generating an immersive audio signal based on the future orientation of the accessory device with respect to the video content. ### Claim 18 (depends on 11) The method of claim 11, wherein the position information includes a movement prediction of the accessory device. ### Claim 19 (depends on 18) The method of claim 18, wherein the updated audio signal is generated using the movement prediction and corresponds to future position of the accessory device. ### Claim 20 A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to perform operations including: establishing, on a computing device, a wireless communication link between the computing device and an accessory device; transmitting an audio signal using the wireless communication link; receiving sensor data from a sensor module of the accessory device or the computing device, wherein the accessory device is movable with respect to the computing device; and generating position information of the accessory device with respect to the computing device using the sensor data, wherein the position information is usable for generating an updated audio signal corresponding to the position information.