# US 11,451,855 B1 · Voice interaction with digital signage using mobile device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,451,855 B1 - Document ID: US11451855B1 - Status: Granted - Kind: B1 - Application number: 17/471,975 - Filed: 2021-09-10 - Granted: 2022-09-20 - Earliest claimed: 2020-09-10 - Applicants: Joseph F. Kirley - Inventors: Joseph F. Kirley - Web page: https://patentgrep.com/patent/US11451855B1 ## CPC - H04N 21/41407: {embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop (constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57)} - H04N 21/422: Input-only peripherals {, i.e. input devices connected to specially adapted client devices}, e.g. global positioning system [GPS] {(input devices also receiving signals from specially adapted client devices H04N21/4104)} - G06F 3/147: using display panels - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - H04N 21/41415: {involving a public display, viewable by several users in a public space outside their home, e.g. movie theatre, information kiosk} - H04N 21/42203: {sound input device, e.g. microphone} - H04N 21/43076: {of the same content streams on multiple devices, e.g. when family members are watching the same movie on different devices} - H04N 21/43637: {involving a wireless protocol, e.g. Bluetooth®, RF or wireless LAN [IEEE 802.11] (arrangements for wireless networking or broadcasting of information in indoor or near-field type systems H04B10/114)} - H04N 21/472: End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content {(end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612)} - H04W 4/80: Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication - G10L 2015/223: {Execution procedure of a spoken command} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract A digital signage system allows a person to use the person's own mobile device as a voice input device to the digital signage system and also can leverage the person's mobile device in other ways such as by accepting other inputs from the mobile device, providing various outputs to the mobile device, or utilizing communication capabilities of the mobile device. ## Claims ### Claim 1 A method for interaction with a digital signage system using a mobile device, the digital signage system including a digital signage display and a signage control processor, the method comprising: establishing one or more wireless communication connections between a signage control application running on the mobile device and the signage control processor; displaying, by the digital signage system, video information on the digital signage display for interaction by a user; transmitting, by the signage control processor to the signage control application running on the mobile device, over at least one of the one or more wireless communication connections, audio information associated with the video information for rendering through an audio output of the mobile device; rendering the audio information by the signage control application running on the mobile device through an audio output of the mobile device in synchronization with the displayed video information; capturing, by the signage control application, user voice signals from a microphone of the mobile device; transmitting, by the signage control application to the signage control processor, voice control information based on the user voice signals; determining, by the digital signage system, a user voice control command based on the voice control information; and controlling operation of the digital signage system based on the user voice control command and the video information displayed on the digital signage display, wherein the user voice control command is synchronized with the video information being displayed on the digital signage display for interaction with said video information. ### Claim 2 (depends on 1) A method according to claim 1, wherein: the video information displayed on the digital signage display is delayed relative to corresponding audio information by a predetermined amount of delay greater than or equal to an amount of delay added to the audio information by the at least one of the one or more wireless communication connections so that the audio information is received by the signage control application at or before the corresponding video information is displayed on the display device; and the signage control application running on the mobile device adds delay to the audio information if needed to render the audio information in synchronization with the delayed video information being displayed on the digital signage display. ### Claim 3 (depends on 1) A method according to claim 1, wherein the one or more communication connections are automatically established when the mobile device running the signage control application is in communicative proximity with the signage control processor. ### Claim 4 (depends on 1) A method according to claim 1, wherein multiple communication connection channels are established between the signage control application running on the user mobile device and the signage control processor to carry different types of traffic. ### Claim 5 (depends on 1) A method according to claim 1, wherein signage control processor displays the video information on the digital signage display. ### Claim 6 (depends on 1) A method according to claim 1, wherein a processor separate from the signage control processor displays the video information on the digital signage display. ### Claim 7 (depends on 1) A method according to claim 1, wherein the signage control application running on the mobile device performs voice recognition on the user voice signals to interpret the user voice signals and wherein the voice control information transmitted by the signage control application includes non-voice data representative of the interpreted user voice signals. ### Claim 8 (depends on 7) A method according to claim 7, wherein the non-voice data comprises voice-to-text data. ### Claim 9 (depends on 1) A method according to claim 1, wherein the signage control processor performs voice recognition on the voice control information. ### Claim 10 (depends on 1) A method according to claim 1, further comprising: receiving, by the signage control application running on the mobile device, a non-voice user input relating to the information displayed on the digital signage display; and transmitting, by the signage control application to the signage control processor, non-voice user input information based on the non-voice user input. ### Claim 11 (depends on 10) A method according to claim 10, wherein the non-voice user input comprises at least one of a motion input or a mobile device touchscreen input. ### Claim 12 (depends on 1) A method according to claim 1, further comprising: transmitting, by the signage control processor to the signage control application running on the mobile device, non-audio output information relating to the information displayed on the digital signage display for rendering on the mobile device. ### Claim 13 (depends on 12) A method according to claim 12, wherein the non-audio output information comprises information for display on a display screen of the mobile device. ### Claim 14 (depends on 1) A method according to claim 1, further comprising: accessing, by the signage control processor, Internet connectivity of the mobile device to perform Internet-related processing for at least one of determining the user voice control command or controlling operation of the digital signage system. ### Claim 15 (depends on 1) A method according to claim 1, wherein the user voice control command comprises at least one of a voice-based query, a voice-based selection of an item displayed on the digital signage display, or voice-based navigation of information displayed on the digital signage display. ### Claim 16 (depends on 1) A method according to claim 1, wherein the signage control processor monitors the audio information for triggers and further controls operation of the digital signage system based on the triggers. ### Claim 17 A digital signage system with interaction from a mobile device, the digital signage system comprising: a digital signage display; and at least one processor including a signage control processor coupled to at least one memory containing instructions executable by the at least one processor to cause the system to: establish one or more wireless communication connections between the signage control processor and a signage control application running on a user mobile device; display video information on the digital signage display for interaction by a user; transmit, by the signage control processor to the signage control application running on the mobile device, over at least one of the one or more wireless communication connections, audio information associated with the video information for rendering through an audio output of the mobile device; render the audio information by the signage control application running on the mobile device through an audio output of the mobile device in synchronization with the displayed video information; receive, by the signage control processor, from the signage control application, voice control information based on user voice signals captured by the signage control application from a microphone of the mobile device; determine a user voice control command based on the voice control information; and control operation of the digital signage system based on the user voice control command and the video information displayed on the digital signage display, wherein the user voice control command is synchronized with the video information being displayed on the digital signage display for interaction with said video information. ### Claim 18 (depends on 17) A system according to claim 17, wherein: the video information displayed on the digital signage display is delayed relative to corresponding audio information by a predetermined amount of delay greater than or equal to an amount of delay added to the audio information by the at least one of the one or more wireless communication connections so that the audio information is received by the signage control application at or before the corresponding video information is displayed on the display device; and the signage control application running on the mobile device adds delay to the audio information if needed to render the audio information in synchronization with the delayed video information being displayed on the digital signage display. ### Claim 19 (depends on 17) A system according to claim 17, wherein the one or more communication connections are automatically established when the mobile device running the signage control application is in communicative proximity with the signage control processor. ### Claim 20 (depends on 17) A system according to claim 17, wherein multiple communication connection channels are established between the signage control processor and the signage control application running on the user mobile device to carry different types of traffic. ### Claim 21 (depends on 17) A system according to claim 17, wherein signage control processor displays the video information on the digital signage display. ### Claim 22 (depends on 17) A system according to claim 17, wherein a processor separate from the signage control processor displays the video information on the digital signage display. ### Claim 23 (depends on 17) A system according to claim 17, wherein the signage control application running on the mobile device performs voice recognition on the user voice signals to interpret the user voice signals and wherein the voice control information transmitted by the signage control application includes non-voice data representative of the interpreted user voice signals. ### Claim 24 (depends on 23) A system according to claim 23, wherein the non-voice data comprises voice-to-text data. ### Claim 25 (depends on 17) A system according to claim 17, wherein the signage control processor performs voice recognition on the voice control information. ### Claim 26 (depends on 17) A system according to claim 17, further comprising: receiving, by the signage control processor, from the signage control application running on the mobile device, non-voice user input information based on a non-voice user input received by the signage control application. ### Claim 27 (depends on 26) A system according to claim 26, wherein the non-voice user input comprises at least one of a motion input or a mobile device touchscreen input. ### Claim 28 (depends on 17) A system according to claim 17, further comprising: transmitting, by the signage control processor to the signage control application running on the mobile device, non-audio output information relating to the information displayed on the digital signage display for rendering on the mobile device. ### Claim 29 (depends on 28) A system according to claim 28, wherein the non-audio output information comprises information for display on a display screen of the mobile device. ### Claim 30 (depends on 17) A system according to claim 17, further comprising: accessing, by the signage control processor, Internet connectivity of the mobile device to perform Internet-related processing for at least one of determining the user voice control command or controlling operation of the digital signage system. ### Claim 31 (depends on 17) A system according to claim 17, wherein the user voice control command comprises at least one of a voice-based query, a voice-based selection of an item displayed on the digital signage display, or voice-based navigation of information displayed on the digital signage display. ### Claim 32 (depends on 17) A system according to claim 17, wherein the signage control processor monitors the audio information for triggers and further controls operation of the digital signage system based on the triggers.