# US 12,602,237 B2 · User interface extendability over wireless protocol > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,237 B2 - Document ID: US12602237B2 - Status: Granted - Kind: B2 - Application number: 17/805,462 - Filed: 2022-06-05 - Published: 2023-12-07 - Granted: 2026-04-14 - Earliest claimed: 2022-06-05 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Jeremy Jie You Foo; Matthew M. Sun - Web page: https://patentgrep.com/patent/US12602237B2 ## CPC - G06F 9/451: Execution arrangements for user interfaces - H04N 21/43072: {of multiple content streams on the same device} - H04N 21/4316: {for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window} - H04N 21/4788: communicating with other users, e.g. chatting {(arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Generating unified content includes receiving, at a first device, first media content comprising a media stream from one or more source devices, and receiving, at the first device, user interface overlay instructions. The first device generates unified content having the first media content and a user interface overlay, and provides the unified content for display on a display device. ## Claims ### Claim 1 A method for processing content, comprising: receiving, at a first device, a playback request for a first media content comprising a media stream; in response to the playback request: transmitting an instruction to a receiver device to retrieve the media stream from a media server over a first data pipeline, determining one or more user interface (UI) overlay elements for the media stream based on a local media application on the first device; generating user interface overlay instructions for drawing the one or more UI overlay elements, and for retrieving sensor data from an accessory device; and transmitting a UI overlay stream over a second data pipeline comprising the user interface overlay instructions to the receiver device, wherein the user interface overlay instructions comprise instructions to render the one or more UI overlay elements over the media stream in accordance with the user interface overlay instructions, and instructions to populate the one or more UI overlay elements with the sensor data retrieved from the accessory device. ### Claim 2 (depends on 1) The method of claim 1, wherein the receiver device comprises a display device. ### Claim 3 (depends on 1) The method of claim 1, further comprising: rendering unified content based on the user interface overlay instructions, and transmitting the unified content to a display device for display. ### Claim 4 (depends on 1) The method of claim 1, wherein the first media content is received via a first data pipeline and the user interface overlay instructions are received via a second data pipeline. ### Claim 5 (depends on 1) The method of claim 1, wherein the user interface overlay instructions comprise instructions to draw a graphical representation of the sensor data. ### Claim 6 A non-transitory computer readable medium comprising computer readable code for processing content, executable by one or more processors to: receive, at a first device, a playback request for a first media content comprising a media stream; in response to the playback request: transmit an instruction to a receiver device to retrieve the media stream from a media server over a first data pipeline, determine one or more user interface (UI) overlay elements for the media stream based on a local media application on the first device; generate user interface overlay instructions for drawing the one or more UI overlay elements, and for retrieving sensor data from an accessory device; and transmit a UI overlay stream over a second data pipeline comprising the user interface overlay instructions to the receiver device, wherein the user interface overlay instructions comprise instructions to render the one or more UI overlay elements over the media stream in accordance with the user interface overlay instructions, and instructions to populate the one or more UI overlay elements with the sensor data retrieved from the accessory device. ### Claim 7 (depends on 6) The non-transitory computer readable medium of claim 6, wherein the receiver device comprises a display device. ### Claim 8 (depends on 6) The non-transitory computer readable medium of claim 6, further comprising computer readable code executable by the one or more processors to: render unified content based on the user interface overlay instructions, and transmit the unified content to a display device for display. ### Claim 9 (depends on 6) The non-transitory computer readable medium of claim 6, wherein the first media content is received via a first data pipeline and the user interface overlay instructions are received via a second data pipeline. ### Claim 10 (depends on 6) The non-transitory computer readable medium of claim 6, wherein the user interface overlay instructions comprise instructions to draw a graphical representation of the sensor data. ### Claim 11 A system comprising: one or more processors; and one or more computer readable media comprising computer readable code executable by the one or more processors to: receive, at a first device, a playback request for a first media content comprising a media stream; in response to the playback request: transmit an instruction to a receiver device to retrieve the media stream from a media server over a first data pipeline, determine one or more user interface (UI) overlay elements for the media stream based on a local media application on the first device; generate user interface overlay instructions for drawing the one or more UI overlay elements, and for retrieving sensor data from an accessory device; and transmit a UI overlay stream over a second data pipeline comprising the user interface overlay instructions to the receiver device, wherein the user interface overlay instructions comprise instructions to render the one or more UI overlay elements over the media stream in accordance with the user interface overlay instructions, and instructions to populate the one or more UI overlay elements with the sensor data retrieved from the accessory device. ### Claim 12 (depends on 11) The system of claim 11, wherein the receiver device comprises a display device. ### Claim 13 (depends on 11) The system of claim 11, further comprising computer readable code executable by the one or more processors to: render unified content based on the user interface overlay instructions, and transmit the unified content to a display device for display. ### Claim 14 (depends on 11) The system of claim 11, wherein the first media content is received via a first data pipeline and the user interface overlay instructions are received via a second data pipeline.