# US 11,310,294 B2 · Companion devices for real-time collaboration in communication sessions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,310,294 B2 - Document ID: US11310294B2 - Status: Granted - Kind: B2 - Application number: 15/480,332 - Filed: 2017-04-05 - Published: 2018-05-03 - Granted: 2022-04-19 - Earliest claimed: 2016-10-31 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Jason Thomas Faulkner; Kenton O'Hara; Ewin Davis Kannuthottiyil; Edward Sean Lloyd Rintel; Kevin Morrison; Robert Corish; Anthony Wieser - Web page: https://patentgrep.com/patent/US11310294B2 ## CPC - H04L 65/1093: by adding participants; by removing participants - H04L 65/4015: {where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference} - H04L 65/403: Arrangements for multi-party communication, e.g. for conferences (data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15) - H04L 67/303: Terminal profiles - H04L 67/52: specially adapted for the location of the user terminal - H04L 67/75: Indicating network or usage conditions on the user display - H04M 3/42263: {where the same subscriber uses different terminals, i.e. nomadism} - H04M 3/568: {audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants (echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30)} - H04M 7/0027: {Collaboration services where a computer is used for data transfer and the telephone is used for telephonic communication} - H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)} - H04N 7/15: Conference systems - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - H04L 67/306: User profiles - H04M 3/563: {User guidance or feature selection} - H04M 3/564: {whereby the feature is a sub-conference} - H04M 2203/2094: Proximity - H04N 2007/145: {Handheld terminals} - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The present disclosure provides a number of techniques for enhancing a user's experience when joining teleconference sessions with multiple devices. When a user attempts to join a teleconference session using the same user identity for multiple devices, a system differentiates the devices as a primary device and at least one companion device. The primary device has a first control set for controlling a teleconference session and the at least one companion device has a companion control set for sharing content. In some embodiments, the primary device also has one set of selected streams, e.g., a stage view, and the companion device has a subset of those streams or other streams based on an activity level. In addition, the present disclosure provides a number of techniques for enabling users to share content using the companion devices. ## Claims ### Claim 1 A method, comprising: generating communication data comprising a plurality of streams associated with a communication session; receiving a first request to join the communication session from a first computing device associated with a user identity; communicating the communication data to the first computing device causing the first computing device to display a primary user interface, the primary user interface including a rendering of the plurality of streams; receiving a second request to join the communication session from a second computing device associated with the user identity; in response to receiving the second request from the second computing device to join the communication session with the user identity associated with the first computing device concurrently joined with the communication session, selecting at least one stream of the plurality of streams based on an activity level of a participant depicted in the at least one stream; and in response to receiving the second request from the second computing device to join the communication session with the user identity associated with the first computing device concurrently joined with the communication session, communicating the communication data to the second computing device causing the second computing device to display a companion user interface, the companion user interface including a rendering of the at least one stream that is selected based on the activity level of the participant depicted in the at least one stream, wherein a layout of the companion user interface of a second application operating on the second computing device is different from a layout of the primary user interface of a first application operating on the first computing device using the same user identity as the second application operating on the second computing device. ### Claim 2 (depends on 1) The method of claim 1, wherein the primary user interface comprises a plurality of core communication control elements, the core communication control elements including user interface elements configured to control a state of the first computing device and a state of the communication session. ### Claim 3 (depends on 1) The method of claim 1, wherein the companion user interface comprises a plurality of companion communication control elements, the companion communication control elements including user interface elements configured to provide content sharing functionality, wherein the content sharing functionality enables the second computing device to share at least one of a selected file, a desktop view, or a stream generated by a camera of the second computing device. ### Claim 4 (depends on 1) The method of claim 1, wherein the at least one stream that is selected based on the activity level is displayed on the second computing device, and wherein the at least one stream that is selected based on the activity level is displayed concurrently with the rendering of the plurality of streams on the first computing device. ### Claim 5 (depends on 1) The method of claim 1, wherein the at least one stream that is selected based on the activity level brings focus to content of the at least one stream on the second computing device while the rendering of the plurality of streams is concurrently displayed on the first computing device. ### Claim 6 (depends on 1) The method of claim 1, wherein the activity level is based on the participant depicted in the at least one stream performing a predetermined action. ### Claim 7 (depends on 1) The method of claim 1, wherein the activity level is based on a number of people depicted with the participant depicted in the at least one stream. ### Claim 8 (depends on 1) The method of claim 1, wherein the activity level is based on at least one of a motion of the participant depicted in the at least one stream or a facial expression of the participant depicted in the at least one stream. ### Claim 9 (depends on 1) The method of claim 1, wherein the second computing device transitions from the primary user interface to the companion user interface in response to determining that the first computing device and the second computing device are in the communication session using the same user identity. ### Claim 10 (depends on 1) The method of claim 1, wherein the layout of the companion user interface is different from the layout of the primary user interface in response to determining that the first computing device and the second computing device are in the communication session using the same user identity. ### Claim 11 A method, comprising: generating communication data comprising a plurality of streams associated with a communication session; receiving a first request to join the communication session from a first computing device associated with a user identity; receiving a second request to join the communication session from a second computing device associated with the user identity; communicating the communication data to the first computing device; and in response to receiving the second request from the second computing device to join the communication session with the user identity associated with the first computing device concurrently joined with the communication session, communicating a sub-portion of the communication data to the second computing device, wherein the sub-portion comprises at least one stream that is selected from the plurality of streams based on an activity level of a participant depicted in the at least one stream, wherein the at least one stream is selected in response to the second request from the second computing device to join the communication session with the user identity associated with the first computing device that is concurrently joined with the communication session, wherein a layout of a user interface of a second application operating on the second computing device is different from a layout of the primary user interface of a first application operating on the first computing device using the same user identity as the second application operating on the second computing device. ### Claim 12 A method, comprising: transmitting a first request to join a communication session from a first computing device associated with a user identity; receiving first communication data comprising a plurality of streams associated with the communication session at the first computing device; displaying a rendering of the plurality of streams on the first computing device; transmitting a second request to join the communication session from a second computing device associated with the user identity; receiving second communication data comprising a subset of streams of the plurality of streams at the second computing device; and in response to receiving the second request from the second computing device to join the communication session with the user identity associated with the first computing device concurrently joined with the communication session, displaying a rendering of the subset of streams of the plurality of streams on the second computing device, wherein the subset of streams of the plurality of streams is selected based at least in part on data indicating an activity level of a participant depicted in at least one stream of the subset of streams, wherein the subset of streams is selected in response to the second request from the second computing device to join the communication session with the user identity associated with the first computing device that is concurrently joined with the communication session, wherein a layout of a user interface of the second computing device is different from a layout of a user interface of the first computing device, wherein a layout of a companion user interface of a second application operating on the second computing device is different from a layout of a primary user interface of a first application operating on the first computing device using the same user identity as the second application operating on the second computing device. ### Claim 13 (depends on 12) The method of claim 12, wherein the plurality of streams comprise a plurality of participant streams and a plurality of content streams, wherein the plurality of participant streams comprise a user media data of the communication session and the plurality of content streams comprise content media data of the communication session. ### Claim 14 (depends on 12) The method of claim 12, wherein the layout of the user interface of the second computing device is different from the layout of the user interface of the first computing device in response to determining that the first computing device and the second computing device are in the communication session using the same user identity. ### Claim 15 (depends on 11) The method of claim 11, wherein the layout of the user interface of the second computing device is different from the layout of the user interface of the first computing device in response to determining that the first computing device and the second computing device are in the communication session using the same user identity. ### Claim 16 (depends on 11) The method of claim 11, wherein the method further comprises: receiving a control command for causing a transition of the first computing device and the second computing device; and in response to receiving the control command, communicating the communication data to the second computing device for displaying a rendering of the plurality of streams; and communicating the sub-portion of the communication data to the first computing device for displaying a rendering of the at least one stream. ### Claim 17 A system comprising: one or more processing units; and a computer-readable medium having encoded thereon computer-executable instructions to cause the one or more processing units to generate communication data comprising a plurality of streams associated with a communication session; receive a first request to join the communication session from a first computing device associated with a user identity; communicate the communication data to the first computing device for displaying a primary user interface, the primary user interface including a plurality of core communication control elements; receive a second request to join the communication session from a second computing device associated with the user identity; in response to receiving the second request from the second computing device to join the communication session with the user identity associated with the first computing device concurrently joined with the communication session, select at least one stream of the plurality of streams based on an activity level of a participant depicted in the at least one stream; and in response to receiving the second request from the second computing device to join the communication session with the user identity associated with the first computing device concurrently joined with the communication session, communicate the communication data to the second computing device causing the second computing device to display a companion user interface, the companion user interface including a plurality of companion communication control elements and a rendering of the at least one stream of the plurality of streams that is selected based on the activity level of the participant depicted in the at least one stream, wherein a layout of the companion user interface of a second application operating on the second computing device is different from a layout of the primary user interface of a first application operating on the first computing device using the same user identity as the second application operating on the second computing device. ### Claim 18 (depends on 17) The system of claim 17, wherein the plurality of streams comprise at least one participant stream and at least one content stream, wherein the computer-executable instructions further cause the one or more processing units to communicate the at least one participant stream and the at least one content stream to the first computing device; and communicate the at least one content stream to the second computing device. ### Claim 19 (depends on 17) The system of claim 17, wherein the computer-executable instructions further cause the one or more processing units to receive a control command for causing a transition of the first computing device and the second computing device; and in response to receiving the control command, communicate the communication data to the second computing device causing a rendering of the plurality of streams; and communicate the communication data to the first computing device causing a rendering of the at least one stream. ### Claim 20 (depends on 17) The system of claim 17, wherein the layout of the companion user interface is different from the layout of the primary user interface in response to determining that the first computing device and the second computing device are in the communication session using the same user identity.