# US 11,909,825 B2 · Multi-master hybrid telerobotics system with federated avatar control > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,909,825 B2 - Document ID: US11909825B2 - Status: Granted - Kind: B2 - Application number: 17/813,457 - Filed: 2022-07-19 - Published: 2023-04-20 - Granted: 2024-02-20 - Earliest claimed: 2021-10-19 - Assignees: Tata Conulstancy Services Limited - Applicants: Tata Consultancy Services Limited - Inventors: Abhijan Bhattacharyya; Ashis Sau; Madhurima Ganguly - Web page: https://patentgrep.com/patent/US11909825B2 ## CPC - H04L 67/104: Peer-to-peer [P2P] networks - H04L 67/141: Setup of application sessions (admission control or resource allocation in data switching networks H04L47/70) - H04L 61/2589: over a relay server, e.g. traversal using relay for network address translation [TURN] - H04L 67/1042: using topology management mechanisms - H04L 67/1089: {Hierarchical topologies} - H04L 67/125: involving control of end-device applications over a network - H04L 67/143: Termination or inactivation of sessions, e.g. event-controlled end of session - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract The present disclosure addresses an issue of inherent delay in a cloud-centric architecture in a scenario where multiple remote users join a teleoperation session with a robot such that any one of the users may take exclusive control of the robot and remotely maneuver it as the avatar of a current master. A Web Real-Time Communication (WebRTC) based signaling protocol with a hybrid topology for a multi-user session is provided, wherein Audio-Visual (A/V) data streaming happens over a public cloud while each master creates an on-demand peer-to-peer (P2P) channel with the robot for a desired duration of maneuvering the robot that represents the avatar of the current master. Thus, low-latency delivery of control commands results in good user experience. The provided method and system may be applied to teleoperation sessions like telemedicine sessions. ## Claims ### Claim 1 A processor implemented method comprising the steps of: sending a peer-to-peer (P2P) connection request, from a master amongst a plurality of masters being user devices associated with a corresponding user, to a broadcaster via a signaling server, wherein the signaling server and the broadcaster are hosted on a public cloud and wherein the broadcaster is connected to a plurality of peers including (i) the plurality of masters and (ii) a robot in a star topology, each peer being identified by an associated peer-type and interact with each other via messaging; notifying (i) an identifier (ID) associated with the robot and the master, and (ii) a current state of the robot, by the broadcaster to the signaling server, wherein the broadcaster comprises a state-register for maintaining the current state of the robot and a media-manager for combining an Audio-Visual (A/V) data received from the plurality of peers via the public cloud; responding, by the broadcaster via the signaling server, to the P2P connection request based on the current state of the robot, wherein the current state of the robot is set by the state-register of the broadcaster as one of (i) state_0 indicative of the robot not having joined a teleoperation session, (ii) state_1 indicative of the robot having joined the teleoperation session but not connected with the master using the P2P connection, and (iii) state_2 indicative of the robot being connected to the master; if the current state of the robot is state_1, performing in parallel, by the signaling server, the steps of: passing the ID associated with the robot and the current state of the robot received from the broadcaster to the master; and requesting the robot to initiate a P2P connection with the master associated with the ID received from the broadcaster; sending, by the robot, a P2P connection offer to the master associated with the ID received, via the signaling server; accepting the P2P connection offer, by the master via the signaling server; establishing the P2P connection, on-demand between the master and the robot, upon exchanging of an Interactive Connectivity Establishment (ICE) candidate therebetween via the signaling server, wherein the robot represents an avatar of the master configured to control the robot via the P2P connection in a federated manner and the master is an active master of the teleoperation session; notifying that the P2P connection is established, by the active master and the robot to the signaling server and by the signaling server to the broadcaster; and broadcasting (i) the current state of the robot being state_2 and (ii) ID of the active master in P2P connection with the robot, to remaining masters amongst the plurality of masters, by the broadcaster, wherein, one or more hardware processors serve as the plurality of masters, the broadcaster, the signaling server and the robot. ### Claim 2 (depends on 1) The processor implemented method of claim 1, wherein the step of establishing the P2P connection is via a Traversal Using Relays around NAT (TURN) server based on whether the active master and the avatar are behind a restrictive Network Address Translators (NAT). ### Claim 3 (depends on 1) The processor implemented method of claim 1, further comprising rejecting the P2P connection request, from the master, by the broadcaster via the signaling server, if the current state of the robot is state_0 or state_2. ### Claim 4 (depends on 1) The processor implemented method of claim 1, further comprising: sending a P2P disconnection request, from the active master, to the broadcaster via the signaling server; notifying the avatar of the P2P disconnection request, by the broadcaster via the signaling server; and broadcasting the current state of the robot being state_1, by the broadcaster via the signaling server to the active master that has relinquished control and the remaining masters amongst the plurality of masters. ### Claim 5 (depends on 1) The processor implemented method of claim 1, further comprising streaming of the Audio-Visual (A/V) data amongst the plurality of peers via the public cloud in the star topology, while control commands are delivered with low latency between the active master and the avatar via the established P2P connection on-demand, thereby providing a hybrid topology. ### Claim 6 (depends on 1) The processor implemented method of claim 1, wherein the messaging between the plurality of peers is in a JavaScript Object Notation (JSON) format with four parameters represented as: { “connection-type”: <>, “room-id”: <>, “message-type”: <>, “message”: <> }. ### Claim 7 (depends on 6) The processor implemented method of claim 6, wherein the four parameters are characterized as: (i) the “connection-type” is a Boolean, wherein 0 indicates a broadcast session and 1 indicates the P2P connection; (ii) the “room-id” is a string value representing an identifier (ID) for the teleoperation session; (iii) the “message-type” is a three-bit string, wherein 000 represents request connection or disconnection, 001 represents send connection request acceptance, 010 represents initiate connection, 011 represents send connection establishment status, 100 represents send robot-connection-status, 101 represents send offer, 110 represents send answer, and 111 represents send ICE candidate; and (iv) the “message” comprises messages specific to the “message-type”. ### Claim 8 A telerobotics system comprising: a memory storing instructions; one or more communication interfaces; and one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors serving as a plurality of masters being user devices associated with a corresponding user, a broadcaster, a signaling server, and a robot are configured by the instructions to: send a peer-to-peer (P2P) connection request, from a master amongst the plurality of masters, to the broadcaster via the signaling server, wherein the signaling server and the broadcaster are hosted on a public cloud and wherein the broadcaster is connected to a plurality of peers including (i) the plurality of masters and (ii) the robot in a star topology, each peer being identified by an associated peer-type; notify (i) an identifier (ID) associated with the robot and the master, and (ii) a current state of the robot, by the broadcaster to the signaling server, wherein the broadcaster comprises a state-register for maintaining the current state of the robot and a media-manager for combining an Audio-Visual (A/V) data received from the plurality of peers via the public cloud; respond, by the broadcaster via the signaling server, to the P2P connection request based on the current state of the robot, wherein the current state of the robot is set by the state-register of the broadcaster as one of (i) state_0 indicative of the robot not having joined a teleoperation session, (ii) state_1 indicative of the robot having joined the teleoperation session but not connected with the master using the P2P connection, and (iii) state_2 indicative of the robot being connected to the master; if the current state of the robot is state_1, perform in parallel, by the signaling server, the steps of: passing the ID associated with the robot and the current state of the robot received from the broadcaster to the master; and requesting the robot to initiate a P2P connection with the master associated with the ID received from the broadcaster; send, by the robot, a P2P connection offer to the master associated with the ID received, via the signaling server; accept the P2P connection offer, by the master via the signaling server; establish the P2P connection, on-demand between the master and the robot, upon exchanging of an Interactive Connectivity Establishment (ICE) candidate therebetween via the signaling server, wherein the robot represents an avatar of the master configured to control the robot via the P2P connection in a federated manner and the master is an active master of the teleoperation session; notify that the P2P connection is established, by the active master and the robot to the signaling server and by the signaling server to the broadcaster; and broadcast (i) the current state of the robot being state_2 and (ii) ID of the active master in P2P connection with the robot, to remaining masters amongst the plurality of masters, by the broadcaster. ### Claim 9 (depends on 8) The telerobotics system of claim 8, wherein the one or more processors are further configured to establish the P2P connection via a Traversal Using Relays around NAT (TURN) server based on whether the active master and the avatar are behind a restrictive Network Address Translators (NAT). ### Claim 10 (depends on 8) The telerobotics system of claim 8, wherein the one or more processors serving as the broadcaster are further configured to reject the P2P connection request, from the master, via the signaling server, if the current state of the robot is state_0 or state_2. ### Claim 11 (depends on 8) The telerobotics system of claim 8, wherein the one or more processors are further configured to: send a P2P disconnection request, from the active master, to the broadcaster via the signaling server; notify the avatar of the P2P disconnection request, by the broadcaster via the signaling server; and broadcast the current state of the robot being state_1, by the broadcaster via the signaling server to the active master that has relinquished control and the remaining masters amongst the plurality of masters. ### Claim 12 (depends on 8) The telerobotics system of claim 8, wherein the one or more processors are further configured to stream the Audio-Visual (A/V) data amongst the plurality of peers via the public cloud in the star topology, while control commands are delivered with low latency between the active master and the avatar via the established P2P connection on-demand, thereby providing a hybrid topology. ### Claim 13 (depends on 8) The telerobotics system of claim 8, wherein the messaging between the plurality of peers is in a JavaScript Object Notation (JSON) format with four parameters represented as: { “connection-type”: <>, “room-id”: <>, “message-type”: <>, “message”: <> }. ### Claim 14 (depends on 13) The telerobotics system of claim 13, wherein the four parameters are characterized as: (i) the “connection-type” is a Boolean, wherein 0 indicates a broadcast session and 1 indicates the P2P connection; (ii) the “room-id” is a string value representing an identifier (ID) for the teleoperation session; (iii) the “message-type” is a three-bit string, wherein 000 represents request connection or disconnection, 001 represents send connection request acceptance, 010 represents initiate connection, 011 represents send connection establishment status, 100 represents send robot-connection-status, 101 represents send offer, 110 represents send answer, and 111 represents send ICE candidate; and (iv) the “message” comprises messages specific to the “message-type”. ### Claim 15 One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause: sending a peer-to-peer (P2P) connection request, from a master amongst a plurality of masters being user devices associated with a corresponding user, to a broadcaster via a signaling server, wherein the signaling server and the broadcaster are hosted on a public cloud and wherein the broadcaster is connected to a plurality of peers including (i) the plurality of masters and (ii) a robot in a star topology, each peer being identified by an associated peer-type and interact with each other via messaging; notifying (i) an identifier (ID) associated with the robot and the master, and (ii) a current state of the robot, by the broadcaster to the signaling server, wherein the broadcaster comprises a state-register for maintaining the current state of the robot and a media-manager for combining an Audio-Visual (A/V) data received from the plurality of peers via the public cloud; responding, by the broadcaster via the signaling server, to the P2P connection request based on the current state of the robot, wherein the current state of the robot is set by the state-register of the broadcaster as one of (i) state_0 indicative of the robot not having joined a teleoperation session, (ii) state_1 indicative of the robot having joined the teleoperation session but not connected with the master using the P2P connection, and (iii) state_2 indicative of the robot being connected to the master; if the current state of the robot is state_1, performing in parallel, by the signaling server, the steps of: passing the ID associated with the robot and the current state of the robot received from the broadcaster to the master; and requesting the robot to initiate a P2P connection with the master associated with the ID received from the broadcaster; sending, by the robot, a P2P connection offer to the master associated with the ID received, via the signaling server; accepting the P2P connection offer, by the master via the signaling server; establishing the P2P connection, on-demand between the master and the robot, upon exchanging of an Interactive Connectivity Establishment (ICE) candidate therebetween via the signaling server, wherein the robot represents an avatar of the master configured to control the robot via the P2P connection in a federated manner and the master is an active master of the teleoperation session; notifying that the P2P connection is established, by the active master and the robot to the signaling server and by the signaling server to the broadcaster; and broadcasting (i) the current state of the robot being state_2 and (ii) ID of the active master in P2P connection with the robot, to remaining masters amongst the plurality of masters, by the broadcaster, wherein, the one or more hardware processors serve as the plurality of masters, the broadcaster, the signaling server and the robot. ### Claim 16 (depends on 15) The one or more non-transitory machine-readable information storage mediums of claim 15, wherein the one or more instructions which when executed by the one or more hardware processors further cause one or more of: establishing the P2P connection is via a Traversal Using Relays around NAT (TURN) server based on whether the active master and the avatar are behind a restrictive Network Address Translators (NAT); rejecting the P2P connection request, from the master, by the broadcaster via the signaling server, if the current state of the robot is state_0 or state_2; sending a P2P disconnection request, from the active master, to the broadcaster via the signaling server; notifying the avatar of the P2P disconnection request, by the broadcaster via the signaling server; broadcasting the current state of the robot being state_1, by the broadcaster via the signaling server to the active master that has relinquished control and the remaining masters amongst the plurality of masters; and streaming of the Audio-Visual (A/V) data amongst the plurality of peers via the public cloud in the star topology, while control commands are delivered with low latency between the active master and the avatar via the established P2P connection on-demand, thereby providing a hybrid topology. ### Claim 17 (depends on 15) The one or more non-transitory machine-readable information storage mediums of claim 15, wherein the messaging between the plurality of peers is in a JavaScript Object Notation (JSON) format with four parameters represented as: { “connection-type”: <> “room-id”: <>, “message-type”: <>, “message”: <> }. ### Claim 18 (depends on 17) The one or more non-transitory machine-readable information storage mediums of claim 17, wherein the four parameters are characterized as: (i) the “connection-type” is a Boolean, wherein 0 indicates a broadcast session and 1 indicates the P2P connection; (ii) the “room-id” is a string value representing an identifier (ID) for the teleoperation session; (iii) the “message-type” is a three-bit string, wherein 000 represents request connection or disconnection, 001 represents send connection request acceptance, 010 represents initiate connection, 011 represents send connection establishment status, 100 represents send robot-connection-status, 101 represents send offer, 110 represents send answer, and 111 represents send ICE candidate; and (iv) the “message” comprises messages specific to the “message-type”.