# US 10,871,935 B2 · Audio cancellation and content recognition of audio received over HDMI/ARC > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,871,935 B2 - Document ID: US10871935B2 - Status: Granted - Kind: B2 - Application number: 16/133,797 - Filed: 2018-09-18 - Published: 2020-03-19 - Granted: 2020-12-22 - Earliest claimed: 2018-09-18 - Assignees: Roku, Inc. - Applicants: Roku, Inc. - Inventors: Robert Caston Curtis - Web page: https://patentgrep.com/patent/US10871935B2 ## CPC - G06F 3/162: {Interface to dedicated audio devices, e.g. audio drivers, interface to CODECs} - H04N 21/43635: {HDMI} - G06F 16/635: Filtering based on additional data, e.g. user or group profiles - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 19/008: Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing - G10L 21/0208: Noise filtering - G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) - H04N 21/439: Processing of audio elementary streams - H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/4668: {for recommending content, e.g. movies} - H04R 3/12: for distributing signals to two or more loudspeakers {(specially adapted for hearing aids H04R25/407)} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G10L 19/167: {Audio streaming, i.e. formatting and decoding of an encoded audio signal representation into a data stream for transmission or storage purposes} - G10L 2015/223: {Execution procedure of a spoken command} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract Apparatus, systems, and methods are disclosed for cancellation and content recognition of audio received over a High-Definition Multimedia Interface (HDMI)/Audio Return Channel (ARC) port of a display device. The media device receives audio over the HDMI/ARC port from the display device, processes the audio by identifying, decoding, re-encoding, and fingerprinting the audio with identification information, and subsequently streams the audio within the media streaming environment, including to external speakers, such as voice-enabled speakers. Based on the identification information and recognized content of the processed audio, the media device offers additional content for streaming to a user. The media device continuously monitors for spoken voice received over a voice-enabled speaker while the media device simultaneously receives, processes, and streams audio from the display device. By cancelling the part of the streaming audio received simultaneously with the spoken voice received, the media device enables the voice-enabled speaker to distinguish a voice command. ## Claims ### Claim 1 A method in a media device communicatively coupled to a High-Definition Multimedia Interface (HDMI)/Audio Return Channel (ARC) interface of a display device, comprising: receiving a first audio output from an external voice-enabled smart speaker that is communicatively coupled to the media device, wherein the first audio output is generated by the external voice-enabled smart speaker and comprises a voice command; receiving a second audio output from the display device over the HDMI/ARC interface, wherein the second audio output is generated by the display device; processing the second audio output received from the display device over the HDMI/ARC interface, wherein the processing comprises: decoding the second audio output into multi-channel audio; and encoding the multi-channel audio to a format of streaming audio compatible with the media device and the external voice-enabled smart speaker; cancelling the streaming audio from the first audio output thereby isolating the voice command; and providing the isolated voice command to the external voice-enabled smart speaker thereby causing the external voice-enabled smart speaker to perform an action based on the isolated voice command, wherein the media device continues receiving the second audio output from the display device over the HDMI/ARC interface. ### Claim 2 (depends on 1) The method of claim 1, wherein the display device is a non-smart television. ### Claim 3 (depends on 1) The method of claim 1, wherein the processing of the second audio output further comprises: selecting an audio processing algorithm from a plurality of audio processing algorithms based on an audio compression format of the second audio output; and wherein the decoding the second audio output further comprises processing the second audio output using the selected audio processing algorithm. ### Claim 4 (depends on 1) The method of claim 1, further comprising: transmitting the streaming audio to an additional external speaker, wherein the additional external speaker is communicatively coupled to the media device and is separate from the external voice-enabled smart speaker. ### Claim 5 (depends on 1) The method of claim 1, wherein the received voice command is a far-field voice. ### Claim 6 (depends on 1) The method of claim 1, further comprising: buffering the cancelled streaming audio; and buffering additional streaming audio while the external voice-enabled smart speaker performs the action based on the isolated voice command. ### Claim 7 (depends on 6) The method of claim 6, further comprising: transmitting the buffered cancelled streaming audio and the buffered additional streaming audio to the external voice-enabled smart speaker after the external voice-enabled smart speaker performs the action based on the isolated voice command. ### Claim 8 A media device, comprising: a transceiver; a memory; and a digital signal processor coupled to the memory and configured to: receive a first audio output from an external voice-enabled smart speaker that is communicatively coupled to the media device via the transceiver, wherein the first audio output is generated by the external voice-enabled smart speaker and comprises a voice command; receive a second audio output from a display device coupled to the media device by a High-Definition Multimedia Interface (HDMI)/Audio Return Channel (ARC) interface, wherein the second audio output is generated by the display device; process the second audio output received from the display device over the HDMI/ARC interface, wherein the digital signal processor is configured to: decode the second audio output into multi-channel audio; and encode the multi-channel audio to a format of streaming audio compatible with the media device and the external voice-enabled smart speaker; cancel the streaming audio from the first audio output thereby isolating the voice command; and provide the isolated voice command to the external voice-enabled smart speaker thereby causing the external voice-enabled smart speaker to perform an action based on the isolated voice command. ### Claim 9 (depends on 8) The media device of claim 8, wherein the digital signal processor is further configured to: select an audio processing algorithm from a plurality of audio processing algorithms based on an audio compression format of the second audio output; and wherein the decoding the second audio output further comprises processing the second audio output using the selected audio processing algorithm. ### Claim 10 (depends on 8) The media device of claim 8, wherein the digital signal processor is further configured to: transmit the streaming audio to an additional external speaker, wherein the additional external speaker is communicatively coupled to the media device and is separate from the external voice-enabled smart speaker. ### Claim 11 (depends on 8) The media device of claim 8, wherein the digital signal processor is further configured to: buffer the cancelled streaming audio; and buffer additional streaming audio while the external voice-enabled smart speaker performs the action based on the isolated voice command. ### Claim 12 (depends on 11) The media device of claim 11, wherein the digital signal processor is further configured to: transmit the buffered cancelled streaming audio and the buffered additional streaming audio to the external voice-enabled smart speaker after the external voice-enabled smart speaker performs the action based on the isolated voice command. ### Claim 13 A non-transitory computer readable medium having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising: receiving a first audio output from an external voice-enabled smart speaker speaker that is communicatively coupled to the at least one computing device, wherein the first audio output is generated by the external voice-enabled smart speaker and comprises a voice command; receiving a second audio output from a display device over a High-Definition Multimedia Interface (HDMI)/Audio Return Channel (ARC) interface, wherein the second audio output is generated by the display device; processing the second audio output received from the display device over the HDMI/ARC interface, wherein the processing comprises: decoding the second audio output into multi-channel audio; and encoding the multi-channel audio to a format of streaming audio compatible with the at least one computing device and the external voice-enabled smart speaker; cancelling the streaming audio from the first audio output thereby isolating the voice command; and providing the isolated voice command to the external voice-enabled smart speaker thereby causing the external voice-enabled smart speaker to perform an action based on the isolated voice command. ### Claim 14 (depends on 13) The non-transitory computer readable medium of claim 13, the operations for processing the second audio output further comprising: selecting an audio processing algorithm from a plurality of audio processing algorithms based on an audio compression format of the second audio output; and wherein the decoding the second audio output further comprises processing the second audio output using the selected audio processing algorithm. ### Claim 15 (depends on 13) The non-transitory computer readable medium of claim 13, the operations further comprising: causing to transmit the streaming audio to an additional external speaker, wherein the additional external speaker is separate from the external voice-enabled smart speaker. ### Claim 16 (depends on 13) The non-transitory computer readable medium of claim 13, the operations further comprising: buffering the cancelled streaming audio; and buffering additional streaming audio while the external voice-enabled smart speaker performs the action based on the isolated voice command. ### Claim 17 (depends on 16) The non-transitory computer readable medium of claim 16, the operations further comprising: causing to transmit the buffered cancelled streaming audio and the buffered additional streaming audio to the external voice-enabled smart speaker after the external voice-enabled smart speaker performs the action based on the isolated voice command.