# US 12,764,387 B2 · Contextual user interface element detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,764,387 B2 - Document ID: US12764387B2 - Status: Granted - Kind: B2 - Application number: 18/869,704 - Filed: 2023-05-16 - Published: 2025-09-25 - Granted: 2026-10-13 - Earliest claimed: 2022-05-31 - Assignees: Sonos, Inc. - Applicants: Sonos, Inc. - Inventors: Paul Andrew Bates - Web page: https://patentgrep.com/patent/US12764387B2 ## CPC - H04N 21/472: End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content {(end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612)} - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A control device such as a mobile device can be used as a secondary display to provide a controller user interface (UI) element corresponding to a contextual display user interface element displayed via a television or other device. An example method includes receiving a media data stream from a media content provider and transmitting the media data stream to a display device for playback. After determining that the media data stream comprises a contextual display user interface element, the method involves causing the controller device to present a controller user interface element corresponding to the contextual display user interface element. An indication of a user input via the controller user interface element is received, after which a signal is transmitted to the media provider that corresponds to the received indication of user input. ## Claims ### Claim 1 A media playback system comprising: one or more processors; data storage having instructions thereon that, when executed by the one or more processors, cause the media playback system to perform operations comprising: receiving, at a computing device, a media data stream from a media content provider; transmitting the media data stream from the computing device to a display device for playback; determining that the media data stream comprises a contextual display user interface element, wherein determining that the media data stream comprises the contextual display user interface element comprises analyzing pixels displayed via the display device to identify the contextual display user interface element; after the determination, causing a controller device to present a contextual controller user interface element corresponding to the contextual display user interface element; receiving, at the computing device, an indication of user input via the contextual controller user interface element; and after receiving the indication, transmitting, via the computing device, a signal corresponding to the received indication of user input to the media content provider. ### Claim 2 (depends on 1) The media playback system of claim 1, wherein the contextual display user interface element comprises a selectable button, and wherein transmitting the signal corresponding to the received indication of user input to the media content provider comprises selecting the button. ### Claim 3 (depends on 1) The media playback system of claim 1, wherein the contextual display user interface element is a first contextual display user interface element, wherein the contextual controller user interface element is a first contextual controller user interface element, the operations further comprising: at a time after transmitting the signal corresponding to the received indication of user input to the media content provider, determining that the media data stream comprises a second contextual display user interface element; and presenting, via the controller device, a second contextual controller user interface element corresponding to the second contextual display user interface element, wherein the second contextual display user interface element is different from the first contextual display user interface element, and wherein the second contextual controller user interface element is different from the first contextual controller user interface element. ### Claim 4 A media playback system comprising: one or more processors; data storage having instructions thereon that, when executed by the one or more processors, cause the media playback system to perform operations comprising: receiving, at a computing device, a media data stream from a media content provider; transmitting the media data stream from the computing device to a display device for playback; determining that the media data stream comprises a contextual display user interface element, wherein determining that the media data stream comprises the contextual display user interface element comprises utilizing an accessibility service to identify the contextual display user interface element; after the determination, causing a controller device to present a contextual controller user interface element corresponding to the contextual display user interface element; receiving, at the computing device, an indication of user input via the contextual controller user interface element; and after receiving the indication, transmitting, via the computing device, a signal corresponding to the received indication of user input to the media content provider. ### Claim 5 (depends on 1) The media playback system of claim 1, wherein determining that the media data stream comprises the contextual display user interface element further comprises inferring a skip option based at least in part on a timestamp of video data being played back. ### Claim 6 (depends on 1) The media playback system of claim 1, wherein the controller device comprises a touch-sensitive display, and wherein the contextual controller user interface element comprises a selectable button displayed via the touch-sensitive display. ### Claim 7 (depends on 1) The media playback system of claim 1, further comprising, after determining that the media data stream comprises the contextual display user interface element, determining, from among a set of one or more user interface elements associated with the media content provider, a user interface element associated with the media content provider corresponding to the contextual display user interface element. ### Claim 8 (depends on 1) The media playback system of claim 1, wherein causing the controller device to present the contextual controller user interface element corresponding to the contextual display user interface element comprises causing the controller device to present a controller user interface that comprises a subset of one or more user interface elements associated with the media content provider, wherein the subset of one or more user interface elements comprises the contextual controller user interface element. ### Claim 9 (depends on 8) The media playback system of claim 8, wherein causing the controller device to present the contextual controller user interface element comprises suppressing display of user interface elements associated with the media content provider unrelated to the contextual controller user interface element. ### Claim 10 A method performed by a media playback system, the method comprising: receiving, at a computing device, a media data stream from a media content provider; transmitting the media data stream from the computing device to a display device for playback; determining that the media data stream comprises a contextual display user interface element, wherein determining that the media data stream comprises the contextual display user interface element comprises utilizing an accessibility service to identify the contextual display user interface element; after the determination, causing a controller device to present a contextual controller user interface element corresponding to the contextual display user interface element; receiving, at the computing device, an indication of a user input via the contextual controller user interface element; and after receiving the indication, transmitting, via the computing device, a signal corresponding to the received indication of user input to the media content provider. ### Claim 11 (depends on 10) The method of claim 10, wherein the contextual display user interface element is a first contextual display user interface element, wherein the contextual controller user interface element is a first contextual controller user interface element, the method further comprising: at a time after transmitting the signal corresponding to the received indication of user input to the media content provider, determining that the media data stream comprises a second contextual display user interface element; and presenting, via the controller device, a second contextual controller user interface element corresponding to the second contextual display user interface element, wherein the second contextual display user interface element is different from the first contextual display user interface element, and wherein the second contextual controller user interface element is different from the first contextual controller user interface element. ### Claim 12 (depends on 10) The method of claim 10, wherein determining that the media data stream comprises the contextual display user interface element further comprises inferring a skip option based at least in part on a timestamp of video data being played back. ### Claim 13 A method performed by a media playback system, the method comprising: receiving, at a computing device, a media data stream from a media content provider; transmitting the media data stream from the computing device to a display device for playback; determining that the media data stream comprises a contextual display user interface element, wherein determining that the media data stream comprises the contextual display user interface element comprises analyzing pixels displayed via the display device to identify the contextual display user interface element; after the determination, causing a controller device to present a contextual controller user interface element corresponding to the contextual display user interface element; receiving, at the computing device, an indication of a user input via the contextual controller user interface element; and after receiving the indication, transmitting, via the computing device, a signal corresponding to the received indication of user input to the media content provider. ### Claim 14 One or more tangible, non-transitory computer-readable media storing instructions that, when executed by one or more processors of a media playback system, cause the media playback system to perform operations comprising: receiving, at a computing device, a media data stream from a media content provider; transmitting the media data stream from the computing device to a display device for playback; determining that the media data stream comprises a contextual display user interface element, wherein determining that the media data stream comprises the contextual display user interface element comprises analyzing pixels displayed via the display device to identify the contextual display user interface element; after the determination, causing a controller device to present a contextual controller user interface element corresponding to the contextual display user interface element; receiving, at the computing device, an indication of a user input via the contextual controller user interface element; and after receiving the indication, transmitting, via the computing device, a signal corresponding to the received indication of user input to the media content provider. ### Claim 15 (depends on 14) The one or more tangible, non-transitory computer-readable media of claim 14, wherein the contextual display user interface element is a first contextual display user interface element, wherein the contextual controller user interface element is a first contextual controller user interface element, the operations further comprising: at a time after transmitting the signal corresponding to the received indication of user input to the media content provider, determining that the media data stream comprises a second contextual display user interface element; and presenting, via the controller device, a second contextual controller user interface element corresponding to the second contextual display user interface element, wherein the second contextual display user interface element is different from the first contextual display user interface element, and wherein the second contextual controller user interface element is different from the first contextual controller user interface element. ### Claim 16 One or more tangible, non-transitory computer-readable media storing instructions that, when executed by one or more processors of a media playback system, cause the media playback system to perform operations comprising: receiving, at a computing device, a media data stream from a media content provider; transmitting the media data stream from the computing device to a display device for playback; determining that the media data stream comprises a contextual display user interface element, wherein determining that the media data stream comprises the contextual display user interface element comprises utilizing an accessibility service to identify the contextual display user interface element; after the determination, causing a controller device to present a contextual controller user interface element corresponding to the contextual display user interface element; receiving, at the computing device, an indication of a user input via the contextual controller user interface element; and after receiving the indication, transmitting, via the computing device, a signal corresponding to the received indication of user input to the media content provider. ### Claim 17 (depends on 14) The one or more tangible, non-transitory computer-readable media of claim 14, wherein determining that the media data stream comprises the contextual display user interface element further comprises inferring a skip option based at least in part on a timestamp of video data being played back. ### Claim 18 (depends on 4) The media playback system of claim 4, wherein the contextual display user interface element is a first contextual display user interface element, wherein the contextual controller user interface element is a first contextual controller user interface element, the operations further comprising: at a time after transmitting the signal corresponding to the received indication of user input to the media content provider, determining that the media data stream comprises a second contextual display user interface element; and presenting, via the controller device, a second contextual controller user interface element corresponding to the second contextual display user interface element, wherein the second contextual display user interface element is different from the first contextual display user interface element, and wherein the second contextual controller user interface element is different from the first contextual controller user interface element. ### Claim 19 (depends on 4) The media playback system of claim 4, wherein the controller device comprises a touch-sensitive display, and wherein the contextual controller user interface element comprises a selectable button displayed via the touch-sensitive display. ### Claim 20 (depends on 4) The media playback system of claim 4, further comprising, after determining that the media data stream comprises the contextual display user interface element, determining, from among a set of one or more user interface elements associated with the media content provider, a user interface element associated with the media content provider corresponding to the contextual display user interface element.