# US 10,771,823 B1 · Presentation of composite streams to users > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,771,823 B1 - Document ID: US10771823B1 - Status: Granted - Kind: B1 - Application number: 16/264,474 - Filed: 2019-01-31 - Granted: 2020-09-08 - Earliest claimed: 2016-12-30 - Assignees: Facebook, Inc. - Applicants: Facebook, Inc. - Inventors: Mathieu R. Henaire; Amit Puntambekar - Web page: https://patentgrep.com/patent/US10771823B1 ## CPC - H04N 21/2187: Live feed - H04L 65/75: Media network packet handling - H04L 65/756: adapting media to device capabilities - H04L 65/765: {intermediate} - H04L 67/565: Conversion or adaptation of application format or content (adding application control or application functional data H04L67/561) - H04N 21/2353: {specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata} - H04N 21/236: Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream {(multiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00)} - H04N 21/2368: Multiplexing of audio and video streams - H04N 21/482: End-user interface for programme selection - H04L 65/61: for supporting one-way streaming services, e.g. Internet radio - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract An online system receives a media stream from a host of a live broadcasting stream and a second media stream from a guest of the host, and generates a single composite stream that includes the multiple live streams. The online system decodes and aligns the video and audio frames of each live stream to ensure that the composite stream displays the multiple live streams in a synchronized fashion. Additionally, the composite stream can display the multiple live streams in a particular format (e.g. side-by-side, or picture-in-picture), and therefore, the online system adjusts the video and audio frames of each live stream to fit that format. For each composite stream, the online system employs a decision engine that identifies the optimal set of encoded composite streams based on a variety of characteristics. The online system encodes the composite stream and distributes the appropriately encoded composite stream to client devices for display to viewers. ## Claims ### Claim 1 A method comprising: receiving, from a first source representing a first user of a social networking system, a live broadcast of a first media stream; receiving, from a second source representing a second user of the social networking system, a second media stream; generating a composite stream by compositing the second media stream with the first media stream; accessing historical data of the first user and the second user that describes views of previous streams hosted by the first user and views of previous streams hosted by the second user; selecting a set of encoders, wherein the encoders in the selected set is based on the historical data; encoding the composite stream into content encodings using the selected set of encoders, each content encoding comprising information describing presentation orientation of the first media stream with respect to presentation orientation of the second media stream; and broadcasting, to one or more client devices, one of the content encodings. ### Claim 2 (depends on 1) The method of claim 1, wherein the historical data comprises data indicating types of client devices that have previously viewed media streams hosted by the first user and data indicating types of client devices that have previously viewed media streams hosted by the second user, such that the set of content encoders is selected to ensure all client devices of users accessing the composite stream can receive the encoded composite stream. ### Claim 3 (depends on 1) The method of claim 1, wherein the historical data comprises popularity data, the popularity data indicating the popularity of previous media streams of at least one of the first user and the second user. ### Claim 4 (depends on 1) The method of claim 1, wherein generating a composite stream by compositing the second media stream with the first media stream comprises: receiving an indication from the first source that the second source has accepted an invitation from the first source; decoding the first media stream and the second media stream; determining one or more required settings for the composite stream; and adjusting the decoded first media stream and the decoded second media stream to meet the required settings of the composite stream. ### Claim 5 (depends on 4) The method of claim 4, wherein the determined one or more required settings for the composite stream comprises a final resolution of a video from the first media stream and a video from the second media stream. ### Claim 6 (depends on 1) The method of claim 1, wherein generating a composite stream by compositing the second media stream with the first media stream comprises: decoding the first media stream and the second media stream; identifying timing information associated with a video frame of the decoded first media stream and timing information associated with a video frame of the decoded second media stream; and aligning the video frame of the decoded first media stream with the video frame of the decoded second media stream based on their associated identified timing information. ### Claim 7 (depends on 6) The method of claim 6, wherein aligning the video frame of the decoded first media stream with the video frame of the decoded second media stream based on their associated identified timing information further comprises: determining a difference between the timing information associated with the video frame of the decoded first media stream and the timing information associated with the video frame of the decoded second media stream. ### Claim 8 (depends on 7) The method of claim 7, further comprising: responsive to the determined difference being below a threshold difference: adjusting the timing information associated with one of the video frame of the decoded first media stream or the video frame of the decoded second media stream; and altering timing information associated with either an audio excerpt from the decoded first media stream or an audio excerpt from the decoded second media stream to reflect the adjusted timing information associated with one of the video frame of the decoded first media stream or the video frame of the decoded second media stream. ### Claim 9 (depends on 7) The method of claim 7, further comprising: responsive to the determined difference being above a threshold difference, identifying timing information associated with a subsequent video frame of the first media stream; and determining a difference between the timing information associated with the subsequent video frame of the decoded first media stream and the timing information associated with the video frame of the decoded second media stream. ### Claim 10 (depends on 1) The method of claim 1, wherein the composite stream comprises a plurality of composite video frames, and each composite video frame comprises a video frame from the second media stream displayed within a corresponding video frame from the first media stream. ### Claim 11 (depends on 1) The method of claim 1, wherein the first media stream comprises a first audio and a first video and the second media stream comprises a second audio and a second video. ### Claim 12 A method comprising: receiving, from a first source representing a first user of a social networking system, a live broadcast of a first media stream comprising a first video and a first audio, the first user of the social networking system being a host of the live broadcasting stream; broadcasting the first media stream to one or more client devices; receiving, from a second source representing a second user of the social networking system, a second media stream comprising a second video and a second audio, the second user being a guest of the host of the live broadcasting stream; generating a composite stream by compositing the second media stream with the first media stream; accessing intrinsic characteristics describing properties of one or more of the first video, the first audio, the second video, and the second audio; selecting a set of encoders, wherein the encoders are selected based on the intrinsic characteristics; encoding the composite stream into content encodings using the selected set of encoders, each content encoding comprising information describing presentation orientation of the first media stream with respect to presentation orientation of the second media stream; and broadcasting, to one of the one or more client devices, one of the content encodings. ### Claim 13 (depends on 12) The method of claim 12, wherein generating a composite stream by compositing the second media stream with the first media stream comprises: receiving an indication from the first source that the second source has accepted an invitation from the first source; decoding the first media stream and the second media stream; determining one or more required settings for the composite stream; and adjusting the decoded first media stream and the decoded second media stream to meet the required settings of the composite stream. ### Claim 14 (depends on 13) The method of claim 13, wherein the determined one or more required settings for the composite stream comprises a final resolution of a video from the first media stream and a video from the second media stream. ### Claim 15 (depends on 12) The method of claim 12, wherein generating a composite stream by compositing the second media stream with the first media stream comprises: decoding the first media stream and the second media stream; identifying timing information associated with a video frame of the decoded first media stream and timing information associated with a video frame of the decoded second media stream; and aligning the video frame of the decoded first media stream with the video frame of the decoded second media stream based on their associated identified timing information. ### Claim 16 (depends on 15) The method of claim 15, wherein aligning the video frame of the decoded first media stream with the video frame of the decoded second media stream based on their associated identified timing information further comprises: determining a difference between the timing information associated with the video frame of the decoded first media stream and the timing information associated with the video frame of the decoded second media stream. ### Claim 17 (depends on 16) The method of claim 16, further comprising: responsive to the determined difference being below a threshold difference: adjusting the timing information associated with one of the video frame of the decoded first media stream or the video frame of the decoded second media stream; and altering timing information associated with either an audio excerpt from the decoded first media stream or an audio excerpt from the decoded second media stream to reflect the adjusted timing information associated with one of the video frame of the decoded first media stream or the video frame of the decoded second media stream. ### Claim 18 (depends on 16) The method of claim 16, further comprising: responsive to the determined difference being above a threshold difference, identifying timing information associated with a subsequent video frame of the first media stream; and determining a difference between the timing information associated with the subsequent video frame of the decoded first media stream and the timing information associated with the video frame of the decoded second media stream. ### Claim 19 (depends on 12) The method of claim 12, wherein the composite stream comprises a plurality of composite video frames, and each composite video frame comprises a video frame from the second media stream displayed within a corresponding video frame from the first media stream. ### Claim 20 (depends on 12) The method of claim 12, wherein the selection of encoders is further based on historical data describing views of previous streams hosted by the first user.