# US 11,445,238 B2 · System and method for synchronizing audio content on a mobile device to a separate visual display system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,445,238 B2 - Document ID: US11445238B2 - Status: Granted - Kind: B2 - Application number: 17/610,089 - Filed: 2020-05-08 - Published: 2022-08-04 - Granted: 2022-09-13 - Earliest claimed: 2019-05-10 - Assignees: Cinewav Pte. Ltd. - Applicants: Cinewav Pte. Ltd. - Inventors: Jason Keng-Kwin Chan; Christian Joel Lee - Web page: https://patentgrep.com/patent/US11445238B2 ## CPC - H04N 21/43076: {of the same content streams on multiple devices, e.g. when family members are watching the same movie on different devices} - H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/41407: {embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop (constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57)} - H04N 21/41415: {involving a public display, viewable by several users in a public space outside their home, e.g. movie theatre, information kiosk} - H04N 21/4307: {Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen} - H04N 21/436: Interfacing a local distribution network, e.g. communicating with another STB {or one or more peripheral devices} inside the home - H04N 21/439: Processing of audio elementary streams - H04N 21/8547: involving timestamps for synchronizing content - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system and method of synchronizing a slave audio content configured for playback on a mobile device to a master audio content or master visual content configured for playing on a separate visual display system are provided for the purpose of enhancing an audience member's audio and visual experience in a public audio-visual show. ## Claims ### Claim 1 A method of synchronizing an audio content configured for playback on a mobile device to a visual content of an audio-visual display configured for playing on a separate visual display system, comprising the steps of: presenting the visual content on a visual output device controlled by the visual display system at a first location and time; obtaining, from the visual display system, a visual display system true time based on a visual display system local time and a visual display system time deviation, generating, by the visual display system, a synchronisation message configured for transmission to a processing server, wherein the synchronisation message includes the visual display system true time, a visual content state and a visual content timecode; transmitting, by the processing server, the synchronisation message at a periodic time interval to the mobile device, determining an audio content timecode that is based on the visual content timecode, a mobile device true time and the visual display system true time; wherein when triggered to play the audio content on the mobile device at a second location and time when the visual content state is in a playing mode, the audio content is played at the audio content timecode that is synchronised with the visual content timecode associated with the current position of the visual content on the visual display system. ### Claim 2 (depends on 1) The method according to claim 1, comprising the step of: determining if the audio content timecode is out of synchronisation beyond a predetermined time interval with the visual content timecode on the visual display system, and; adjusting playback of the audio content on the mobile device to the audio content timecode that is synchronised with the visual content timecode associated with the current position of the visual content on the visual display system, in response to when the audio content timecode is out of synchronisation beyond the predetermined time interval. ### Claim 3 (depends on 1) The method according to claim 1, wherein the audio content timecode is determined based on aggregating the visual content timecode and the time difference between the mobile device true time and the visual display system true time. ### Claim 4 (depends on 1) The method according to claim 1, wherein the synchronisation message is transmitted when there is an adjustment made to the visual content timecode. ### Claim 5 (depends on 1) The method according to claim 1, wherein the synchronisation message is transmitted when there is a change in state to the visual content state. ### Claim 6 (depends on 3) The method according to claim 3, wherein the mobile device true time is determined based on a mobile device local time and a mobile device time deviation, wherein the mobile device time deviation is the time difference between the mobile device local time and a time server current time derived from a time server in communication with the mobile device. ### Claim 7 (depends on 1) The method according to claim 1, wherein the visual display system time deviation is the time difference between the visual display system local time and a time server current time derived from a time server in communication with the visual display system. ### Claim 8 (depends on 2) The method according to claim 2, wherein the predetermined time interval is approximately 200 ms. ### Claim 9 A method of synchronizing a slave audio content configured for playback on a mobile device to a master audio content synchronised with a live visual display, wherein the master audio content is configured for playing on a separate visual display system, comprising the steps of: playing the master audio content on the visual display system at a first location and time; obtaining, from the visual display system, a visual display system true time based on a visual display system local time and a visual display system time deviation; generating, by the visual display system, a synchronisation message configured for transmission to a processing server, wherein the synchronisation message includes the visual display system true time, a master audio content state and a master audio content timecode; transmitting, by the processing server, the synchronisation message at a periodic time interval to the mobile device; determining a slave audio content timecode that is based on the master audio content timecode, a mobile device true time and the visual display system true time, wherein when triggered to play the slave audio content on the mobile device at a second location and time when the master audio content state is in a playing mode, the slave audio content is played at the slave audio content timecode that is synchronised with the master audio content timecode associated with the current position of the master audio content on the visual display system. ### Claim 10 (depends on 9) The method according to claim 9, comprising the step of: determining if the slave audio content timecode is out of synchronisation beyond a predetermined time interval with the master audio content timecode on the visual display system, and; adjusting playback of the slave audio content on the mobile device to the audio content timecode that is synchronised with the master audio content timecode associated with the current position of the master audio content on the visual display system, in response to when the slave audio content timecode is out of synchronisation beyond the predetermined time interval. ### Claim 11 (depends on 9) The method according to claim 9, wherein the slave audio content comprises slave audio cue tracks for initiating a portion of the live visual display, whereby the slave audio cue tracks are transmitted to one or more control systems by one or more computing devices, such that the audio cue tracks are configured for synchronisation with the master audio content. ### Claim 12 (depends on 9) The method according to claim 9, wherein the slave audio content comprises an audio sound track associated with a voice recording that is configured for synchronisation with the master audio content. ### Claim 13 (depends on 9) The method according to claim 9, wherein the slave audio content comprises an audio sound track associated with a music accompaniment that is configured for synchronisation with the master audio content. ### Claim 14 (depends on 9) The method according to claim 9, wherein the master audio content includes an audio cue track, configured for triggering a portion of a live visual display by a controller or manual operator. ### Claim 15 (depends on 9) The method according to claim 9, wherein the slave audio content comprises a slave audio cue track configured for triggering a portion of a live visual display, by a manual operator, that is configured for synchronisation with the master audio content. ### Claim 16 (depends on 9) The method according to claim 9, wherein the synchronisation message is transmitted when there is an adjustment made to the visual content timecode. ### Claim 17 (depends on 9) The method according to claim 9, wherein the synchronisation message is transmitted when there is a change in state to the visual content state. ### Claim 18 (depends on 9) The method according to claim 9, wherein the slave audio content timecode is determined based on aggregating the master audio content timecode and the time difference between the mobile device true time and the visual display system true time, and wherein the mobile device true time is determined based on a mobile device local time and a mobile device time deviation, wherein the mobile device time deviation is the time difference between the mobile device local time and a time server current time derived from a time server in communication with the mobile device. ### Claim 19 (depends on 9) The method according to claim 9, wherein the visual display system true time is determined based on a visual display system local time and a visual display system time deviation, wherein the visual display system time deviation is the time difference between the visual display system local time and a time server current time derived from a time server in communication with the visual display system. ### Claim 20 (depends on 9) The method according to claim 9, wherein the predetermined time interval is approximately 200 ms. ### Claim 21 A method configured for implementing on a mobile device having at least one processor, at least one computer-readable storage medium, and a synchronisation application connected to a network, comprising: transmitting an audio content configured for storing on the mobile device; obtaining a mobile device true time based on a mobile device local time and a mobile device time deviation, wherein the mobile device time deviation is the time difference between the mobile device local time and a time server current time derived from a time server in communication with the mobile device; receiving a synchronisation message at a periodic time interval from a processing server in wireless communication with a visual display system, wherein the synchronisation message includes a visual display system true time, a visual content state, and a visual content timecode; determining an audio content timecode that is based on the visual content timecode, the mobile device true time and the visual display system true time; wherein the visual display system true time is determined based on a visual display system local time and a visual display system time deviation, wherein the visual display system time deviation is the time difference between the visual display system local time and a time server current time derived from a time server in communication with the visual display system; wherein when triggered to play the audio content on the mobile device when the visual content state is in a playing mode, the audio content is played at the audio content timecode that is synchronised with the visual content timecode associated with the current position of the visual content on the visual display system. ### Claim 22 A method of synchronizing a slave audio content configured for playback on a mobile device to a master audio content configured for playing on a separate visual display system, comprising the steps of: obtaining, from the visual display system, a visual display system true time based on a visual display system local time and a visual display system time deviation wherein the visual display system time deviation is the time difference between the visual display system local time and a time server current time derived from a time server in communication with the visual display system; generating, by the visual display system, a synchronisation message configured for transmission to a processing server, wherein the synchronisation message includes the visual display system true time, a master audio content state associated with the operation mode, and a master audio content timecode associated with the current position of the visual display system; transmitting, by the processing server, the synchronisation message at a periodic time interval to the mobile device, determining a slave audio content timecode that is based on the master audio content timecode, a mobile device true time and the visual display system true time; wherein the mobile device true time is based on the mobile device local time and a mobile device time deviation wherein the mobile device time deviation is the time difference between the mobile device local time and a time server current time derived from the time server in communication with the mobile device; wherein when triggered to play the slave audio content on the mobile device when the master audio content state is in a playing mode, the slave audio content is played at the slave audio content timecode that is synchronised with the master audio content timecode associated with the current position of the master audio content on the visual display system. ### Claim 23 A system for synchronizing a slave audio content configured for playback on a mobile device to a master audio content configured for playing on a separate visual display system, comprising: a memory; one or more processors coupled with the memory, wherein the memory includes processor executable code that, when executed by the processor, causes the processor to perform operations including: obtaining, from the visual display system, a visual display system true time; wherein the visual display system true time is determined based on a visual display system local time and a visual display system time deviation, wherein the visual display system time deviation is the time difference between the visual display system local time and a time server current time derived from a time server in communication with the visual display system; generating, by the visual display system, a synchronisation message configured for transmission to a processing server, wherein the synchronisation message includes the visual display system true time, a master audio content state associated with the operation mode, and a master audio content timecode associated with the current position of the visual display system; transmitting, by the processing server, the synchronisation message at a periodic time interval to the mobile device, determining a slave audio content timecode that is based on the master audio content timecode, a mobile device true time and the visual display system true time wherein the mobile device true time is based on the mobile device local time and a mobile device time deviation wherein the mobile device time deviation is the time difference between the mobile device local time and a time server current time derived from the time server in communication with the mobile device; wherein when triggered to play the slave audio content on the mobile device when the master audio content state is in a playing mode, the slave audio content is played at the slave audio content timecode that is synchronised with the master audio content timecode associated with the current position of the master audio content on the visual display system.