# US 12,192,434 B2 · Cloud-based rendering of interactive augmented/virtual reality experiences > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,192,434 B2 - Document ID: US12192434B2 - Status: Granted - Kind: B2 - Application number: 18/197,999 - Filed: 2023-05-16 - Published: 2023-09-14 - Granted: 2025-01-07 - Earliest claimed: 2021-06-07 - Assignees: zSpace, Inc. - Applicants: zSpace, Inc. - Inventors: Clifford S. Champion; Jonathan J. Hosenpud; Baifang Lu; Alex Shorey; Robert D. Kalnins - Web page: https://patentgrep.com/patent/US12192434B2 ## CPC - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - H04N 21/816: {involving special video data, e.g 3D video} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - H04L 65/1016: IP multimedia subsystem [IMS] - H04L 65/1063: Application servers providing network services (systems providing special services to telephonic subscribers H04M3/42) - H04L 65/612: for unicast - H04L 65/75: Media network packet handling - H04L 65/80: Responding to QoS - H04L 67/131: Protocols for games, networked simulations or virtual reality - H04N 13/128: Adjusting depth or disparity - H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) - H04N 13/194: Transmission of image signals - H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras} - H04N 21/2402: {Monitoring of the downstream path of the transmission network, e.g. bandwidth available (traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876)} - H04N 21/25825: {involving client display capabilities, e.g. screen resolution of a mobile phone (optimising the visualisation of content during browsing in the Internet G06F16/9577; processing of terminal status or physical abilities in wireless networks H04W8/22; authentication in wireless network security H04W12/06)} - H04N 21/4312: {involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations} - H04N 21/440218: {by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 (conversion of standards in analogue television systems H04N7/01)} - H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)} - H04N 21/6582: {Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number (arrangements where receivers interact with the broadcast H04H20/38)} - H04N 21/6587: Control parameters, e.g. trick play commands, viewpoint selection - H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Systems and methods for implementing methods for cloud-based rendering of interactive augmented reality (AR) and/or virtual reality (VR) experiences. A client device may initiate execution of a content application on a server and provide information associated with the content application to the server. The client device may initialize, while awaiting a notification from the server, local systems associated with the content application and, upon receipt of the notification, provide, to the server, information associated with the local systems. Further, the client device may receive, from the server, data associated with the content application and render an AR/VR scene based on the received data. The data may be based, at least in part, on the information associated with the local system. The providing and receiving may be performed periodically, e.g., at a rate to sustain a comfortable viewing environment of the AR/VR scene by a user of the client device. ## Claims ### Claim 1 A non-transitory computer readable memory medium storing programming instructions executable by processing circuitry of a client device comprising a three-dimensional (3D) display to: receive, from a server executing a content application, data based on the content application, wherein the data is received over a network, and wherein the data is based on information the client device provided to the server; render a 3D scene based on the received data; and incorporate attributes with semantic hints as input options into an artificial intelligence (AI) image enhancement algorithm, wherein the AI image enhancement algorithm is optimized for different object types within the 3D scene, and wherein the different object types within the 3D scene include one or more of text, two-dimensional (2D) images, or 3D images. ### Claim 2 (depends on 1) The non-transitory computer readable memory medium of claim 1, wherein the information includes first information that is associated with the client device, and wherein the first information includes one or more of a display size, a display resolution, a display model, system specifications, a system language, or a system locale. ### Claim 3 (depends on 1) The non-transitory computer readable memory medium of claim 1, wherein the programming instructions are further executable by the processing circuitry of the client device to: create a local stereoscopic-enabled window or full-screen surface; and initialize tracking sub-systems. ### Claim 4 (depends on 1) The non-transitory computer readable memory medium of claim 1, wherein the client device includes one or more of a head tracking sub-system, an eye tracking sub-system, or a user tracking sub-system, and wherein user tracking includes at least one of user hand tracking or stylus tracking. ### Claim 5 (depends on 1) The non-transitory computer readable memory medium of claim 1, wherein the information includes information associated with local systems, and wherein to provide the information associated with the local systems to the server, the programming instructions are further executable by the processing circuitry of the client device to: receive local tracking system state information; encode the local tracking system state information; transmit the local tracking system state information to the server via the network; encode user input events and user input state information; and transmit the user input events and user input state information to the server via the network. ### Claim 6 (depends on 5) The non-transitory computer readable memory medium of claim 5, wherein the local tracking system state information includes one or more of: head tracking state, quality, and poses; eye tracking state, quality, and poses; or user tracking state, quality, and poses. ### Claim 7 (depends on 5) The non-transitory computer readable memory medium of claim 5, wherein the user input events include one or more of keyboard events or mouse events. ### Claim 8 (depends on 1) The non-transitory computer readable memory medium of claim 1, wherein the AI image enhancement algorithm receives a recent history of images and enhances image clarity based on the recent history of images. ### Claim 9 A three-dimensional (3D) display system comprising: at least one radio, wherein the at least one radio is configured to perform wireless communication using at least one radio access technology (RAT); at least one processor coupled to the at least one radio, wherein the one or more processors and the at least one radio are configured to perform data communications; one or more displays, coupled to the at least one processor; a tracking system comprising two or more cameras and in communication with the at least one processor; and a memory in communication with the tracking system and the at least one processor, wherein the at least one processor is configured to: receive, from a server executing a content application, data based on the content application, wherein the data is received over a network, and wherein the data is based on information the 3D display system provided to the server; render a 3D scene based on the received data; and incorporate attributes with semantic hints as input options into an artificial intelligence (AI) image enhancement algorithm, wherein the AI image enhancement algorithm is optimized for different object types within the 3D scene, and wherein the different object types within the 3D scene include one or more of text, two-dimensional (2D) images, or 3D images. ### Claim 10 (depends on 9) The 3D display system of claim 9, wherein the at least one processor is further configured to: predict a movement of a user based on physiological assumptions regarding a target audience of users and one or more network factors, and wherein the one or more network factors include at least one of: measured network reliability according to a trailing window; a geographic location of an edge of a cell relative to a current location of the 3D display system; or a geographic location of a content delivery network cloud facility hosting the server relative to the current location of the 3D display system. ### Claim 11 (depends on 9) The 3D display system of claim 9, wherein the 3D scene includes one or more elements, and wherein, to receive data associated with the content application, the at least one processor is further configured to receive one or more streams of data, wherein each stream of data is associated with a respective element of the one or more elements. ### Claim 12 (depends on 11) The 3D display system of claim 11, wherein the one or more elements include at least one of a menu user interface (UI) element, an interactive 3D model, a static 3D model, a background 3D model, or a background image. ### Claim 13 (depends on 11) The 3D display system of claim 11, wherein each element of the one or more elements is encoded separately according to at least one of a respective frame rate for the element or resolution properties associated with the element. ### Claim 14 A method for rendering a three-dimension (3D) scene, comprising: a client device, receiving, from a server executing a content application, data based on the content application, wherein the data is received over a network, and wherein the data is based on information the client device provided to the server; rendering a 3D scene based on the received data; and incorporating attributes with semantic hints as input options into an artificial intelligence (AI) image enhancement algorithm, wherein the AI image enhancement algorithm is optimized for different object types within the 3D scene, and wherein the different object types within the 3D scene include one or more of text, two-dimensional (2D) images, or 3D images. ### Claim 15 (depends on 14) The method of claim 14, wherein the information includes one or more of a display size of a display of the client device, a display resolution of a display of the client device, a display model of a display of the client device, system specifications of the client device, a system language of the client device, or a system locale of the client device. ### Claim 16 (depends on 14) The method of claim 14, wherein the information includes local tracking system state information, and wherein the local tracking system state information includes one or more of: head tracking state, quality, and poses; eye tracking state, quality, and poses; or user tracking state, quality, and poses. ### Claim 17 (depends on 14) The method of claim 14, wherein quality of service (QoS) attributes are assigned for each channel of communication between the client device and server. ### Claim 18 (depends on 17) The method of claim 17, wherein the QoS attributes are assigned based, at least in part, on user preferences, directives provided by the server, or directives provided by the content application. ### Claim 19 (depends on 17) The method of claim 17, wherein the channels of communication between the client device and server include one or more of data channels for head and/or eye tracking state upload, stylus and/or hand tracking state upload, mouse and/or keyboard state upload, final frame video download, or download of individual layers. ### Claim 20 (depends on 14) The method of claim 14, wherein the network operates according to Third Generation Partnership Project (3GPP) Fifth Generation (5G) New Radio (NR); and wherein the client device comprises a user equipment device (UE).