# US 12,003,792 B2 · Adaptation of 2D video for streaming to heterogenous client end-points > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,003,792 B2 - Document ID: US12003792B2 - Status: Granted - Kind: B2 - Application number: 17/406,268 - Filed: 2021-08-19 - Published: 2022-06-09 - Granted: 2024-06-04 - Earliest claimed: 2020-12-08 - Assignees: Tencent America LLC - Applicants: Tencent America LLC - Inventors: Arianne Hinds; Stephan Wenger - Web page: https://patentgrep.com/patent/US12003792B2 ## CPC - H04N 21/2343: involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements {(video transcoding H04N19/40; media packet handling at the source H04L65/762)} - H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - H04N 13/194: Transmission of image signals - H04N 13/261: with monoscopic-to-stereoscopic image conversion - H04N 13/31: using parallax barriers - H04N 21/2662: Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities - H04N 21/816: {involving special video data, e.g 3D video} - G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)} - G06N 3/08: Learning methods - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method, computer program, and computer system is provided for streaming immersive media. The method includes ingesting content in a two-dimensional format; converting the ingested content to a three-dimensional format based on a neural network; and streaming the converted content to a client end-point. ## Claims ### Claim 1 A method of streaming immersive media, executable by a processor, the method comprising: ingesting content in a two-dimensional format; adapting the ingested content to a three-dimensional format based on a neural network, wherein the neural network is selected based on a neural network token provided by a first client end-point, wherein the neural network token is provided responsive to a request that the first client end-point provide a neural network token associated with a neural network model available at the first client end-point, and wherein the adapting maps a geometry of a first client end-point frustum to interpolation abilities of the ingested content; and streaming the adapted content to the first client end-point and a second client end-point, wherein the first client end-point is unable to support the two-dimensional format, and wherein the second client end-point is able to support the two-dimensional format. ### Claim 2 (depends on 1) The method of claim 1, wherein the adapting of the ingested content comprises deriving, by the neural network, a volumetric representation of a scene depicted in the ingested content based on correlated images included in the ingested content. ### Claim 3 (depends on 1) The method of claim 1, wherein the two-dimensional format is a single-view two-dimensional format. ### Claim 4 (depends on 1) The method of claim 1, wherein the two-dimensional format is a multiple-view two-dimensional format. ### Claim 5 (depends on 1) The method of claim 1, wherein the ingested content is adapted using an adaptation process unit comprising at least one from among a central processing unit and a graphics processing unit. ### Claim 6 (depends on 5) The method of claim 5, wherein the neural network corresponds to a neural network model selected by the adaptation process unit from among a plurality of neural network models stored in a memory included in the adaptation process unit. ### Claim 7 (depends on 1) The method of claim 1, wherein the first client end-point includes at least one from among a television, a computer, a head-mounted display, a lenticular light field display, a holographic display, an augmented reality display, and a dense light field display. ### Claim 8 A device for streaming immersive media, the device comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code including: ingesting code configured to cause the at least one processor to ingest content in a two-dimensional format; adapting code configured to cause the at least one processor to adapt the ingested content to a three-dimensional format based on a neural network, wherein the neural network is selected based on a neural network token provided by a first client end-point, wherein the neural network token is provided responsive to a request that the first client end-point provide a neural network token associated with a neural network model available at the first client end-point, and wherein the adapting maps a geometry of a first client end-point frustum to interpolation abilities of the ingested content; and streaming code configured to cause the at least one processor to stream the adapted content to the first client end-point and a second client end-point, wherein the first client end-point is unable to support the two-dimensional format, and wherein the second client end-point is able to support the two-dimensional format. ### Claim 9 (depends on 8) The device of claim 8, wherein the adapting code comprises deriving code configured to cause the at least one processor to derive, by the neural network, a volumetric representation of a scene depicted in the ingested content based on correlated images included in the ingested content. ### Claim 10 (depends on 8) The device of claim 8, wherein the two-dimensional format is a single-view two-dimensional format. ### Claim 11 (depends on 8) The device of claim 8, wherein the two-dimensional format is a multiple-view two-dimensional format. ### Claim 12 (depends on 8) The device of claim 8, wherein the ingested content is adapted using an adaptation process unit included in the device, the adaptation process unit comprising at least one from among a central processing unit and a graphics processing unit. ### Claim 13 (depends on 12) The device of claim 12, wherein the neural network corresponds to a neural network model selected by the adaptation process unit from among a plurality of neural network models stored in the at least one memory. ### Claim 14 (depends on 8) The device of claim 8, wherein the first client end-point includes at least one from among a television, a computer, a head-mounted display, a lenticular light field display, a holographic display, an augmented reality display, and a dense light field display. ### Claim 15 A non-transitory computer-readable medium storing instructions, the instructions comprising: one or more instructions that, when executed by at least one processor of a device for streaming immersive media, cause the at least one processor to: ingest content in a two-dimensional format; adapt the ingested content to a three-dimensional format based on a neural network, wherein the neural network is selected based on a neural network token provided by a first client end-point, wherein the neural network token is provided responsive to a request that the first client end-point provide a neural network token associated with a neural network model available at the first client end-point, and wherein the adapting maps a geometry of a first client end-point frustum to interpolation abilities of the ingested content; and stream the adapted content to the first client end-point and a second client end-point, wherein the first client end-point is unable to support the two-dimensional format, and wherein the second client end-point is able to support the two-dimensional format. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the one or more instructions further cause the at least one processor to derive, by the neural network, a volumetric representation of a scene depicted in the ingested content based on correlated images included in the ingested content. ### Claim 17 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the two-dimensional format is a single-view two-dimensional format. ### Claim 18 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the two-dimensional format is a multiple-view two-dimensional format.