# US 12,597,174 B2 · Method and apparatus for delivering 5G AR/MR cognitive experience to 5G devices > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,597,174 B2 - Document ID: US12597174B2 - Status: Granted - Kind: B2 - Application number: 17/977,560 - Filed: 2022-10-31 - Published: 2023-05-04 - Granted: 2026-04-07 - Earliest claimed: 2021-11-03 - Assignees: Tencent America LLC - Applicants: Tencent America LLC - Inventors: Iraj Sodagar - Web page: https://patentgrep.com/patent/US12597174B2 ## CPC - G06T 11/00: Two-dimensional [2D] image generation - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - H04L 65/1069: Session establishment or de-establishment - H04L 65/60: Network streaming of media packets - H04L 65/612: for unicast - H04L 65/80: Responding to QoS - H04N 21/6587: Control parameters, e.g. trick play commands, viewpoint selection - H04N 21/816: {involving special video data, e.g 3D video} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method for delivering a 5th generation (5G) augmented reality (AR)/mixed reality (MR) experience includes establishing an AR/MR session for streaming AR/MR media content for transmission over a 5G media streaming (5GMS) network; receiving an AR/MR media content update that is associated with a plurality of AR/MR scenes; iteratively receiving a respective AR/MR scene included in the plurality of AR/MR scenes; rendering the respective AR/MR scene; acquiring cognitive information and pose information corresponding to the respective AR/MR scene; and transmitting the cognitive information and the pose information to a cognitive immersive server while the AR/MR media content is streamed. ## Claims ### Claim 1 A method for delivering a cognitive immersive experiences to augmented reality (AR)/mixed reality (MR) devices, the method comprising: receiving, by a Cognitive Immersive Server (CIS), captured media data that is associated with a plurality of scenes, the captured media data comprising cognitive media data, interaction information, and pose information; configuring, by the CIS, an immersive media streaming session, the configuring comprising: establishing transport sessions between the CIS and a Media Client of a Cognitive Media Client for exchange of delivery manifests associated with a plurality of immersive media scenes; receiving information associated with a first number of initialized media pipelines based on the delivery manifests where each initialized media pipeline is associated with a respective media stream according to the plurality of immersive media scenes; establishing a first number of transport sessions between the CIS and the Media Client of the Cognitive Media Client for exchange of immersive media associated with the plurality of immersive media scenes; and receiving information on the first number of initialized media pipelines associated with the Media Client of the Cognitive Media Client; processing, by the CIS, the captured media data to generate the immersive media associated with the plurality of immersive media scenes; and transmitting, by the CIS, the immersive media to the Media Client of the Cognitive Media Client based on the first number of media pipelines. ### Claim 2 (depends on 1) The method of claim 1, wherein the cognitive media data is generated by performing at least one of semantical perception, spatial localization, and mapping based on cognitive information and the pose information. ### Claim 3 (depends on 1) The method of claim 1, wherein the captured media data comprises standalone augmented reality (STAR) media content. ### Claim 4 (depends on 3) The method of claim 3, wherein the immersive media streaming session is created in an outer scene session loop, and wherein the immersive media is transmitted from the CIS to the Media Client in an inner media session loop. ### Claim 5 (depends on 1) The method of claim 1, wherein the captured media data comprises edge dependent AR (EDGAR) media content. ### Claim 6 A device for cognitive immersive experiences to augmented reality (AR)/mixed reality (MR) devices, the device comprising: at least one memory configured to store program code; and at least one processor implementing a Cognitive Immersive Server (CIS), configured to read the program code and operate as instructed by the program code, the program code including: first receiving code configured to cause the at least one processor to receive captured media data that is associated with a plurality of scenes, the captured media data comprising cognitive media data, interaction information, and pose information; first configuring code configured to cause the at least one processor to configure an immersive media streaming session, the first configuring code comprising: establishing transport sessions between the CIS and a Media Client of a Cognitive Media Client for exchange of delivery manifests associated with a plurality of immersive media scenes; receiving information associated with a first number of initialized media pipelines based on the delivery manifests where each initialized media pipeline is associated with a respective media stream according to the plurality of immersive media scenes; establishing a first number of transport sessions between the CIS and the Media Client of the Cognitive Media Client for exchange of immersive media associated with the plurality of immersive media scenes; and receiving information on the first number of initialized media pipelines associated with the Media Client of the Cognitive Media Client; processing code configured to cause the at least one processor to process, by the CIS, the captured media data to generate the immersive media associated with the plurality of immersive media scenes; and second transmitting code configured to cause the at least one processor to transmit the cognitive media data to the Media Client of the Cognitive Media Client. ### Claim 7 (depends on 6) The device of claim 6, wherein the cognitive media data is generated by performing at least one of semantical perception, spatial localization, and mapping based on cognitive information and the pose information. ### Claim 8 (depends on 6) The device of claim 6, wherein the captured media data comprises standalone augmented reality (STAR) media content. ### Claim 9 (depends on 8) The device of claim 8, wherein the immersive media streaming session is created in an outer scene session loop, and wherein the immersive media is transmitted from the CIS to the Media Client in an inner media session loop. ### Claim 10 (depends on 6) The device of claim 6, wherein the captured media data comprises edge dependent AR (EDGAR) media content. ### Claim 11 A non-transitory computer-readable medium storing instructions, the instructions comprising: one or more instructions that, when executed by one or more processors of a Cognitive Immersive Server (CIS) for delivering a cognitive immersive experiences to augmented reality (AR)/mixed reality (MR) devices, cause the one or more processors of the CIS to: receive captured media data that is associated with a plurality of scenes, the captured media data comprising cognitive media data, interaction information, and pose information; configure an immersive media streaming session, the configuring comprising: establishing transport sessions between the CIS and a Media Client of a Cognitive Media Client for exchange of delivery manifests associated with a plurality of immersive media scenes; receiving information associated with a first number of initialized media pipelines based on the delivery manifests where each initialized media pipeline is associated with a respective media stream according to the plurality of immersive media scenes; establishing a first number of transport sessions between the CIS and the Media Client of the Cognitive Media Client for exchange of immersive media associated with the plurality of immersive media scenes; and receiving information on the first number of initialized media pipelines associated with the Media Client of the Cognitive Media Client; process, by the CIS, the captured media data to generate the immersive media associated with the plurality of immersive media scenes; and transmit the immersive media to the Media Client of the Cognitive Media Client based on the first number of media pipelines. ### Claim 12 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the cognitive media data is generated by performing at least one of semantical perception, spatial localization, and mapping based on cognitive information and the pose information. ### Claim 13 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the captured media data comprises standalone augmented reality (STAR) media content. ### Claim 14 (depends on 13) The non-transitory computer-readable medium of claim 13, wherein the immersive media streaming session is created in an outer scene session loop, and wherein the immersive media is transmitted from the CIS to the Media Client in an inner media session loop. ### Claim 15 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein the captured media data comprises edge dependent AR (EDGAR) media content.