# US 12,217,389 B2 · Codec for processing scenes of almost unlimited detail > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,389 B2 - Document ID: US12217389B2 - Status: Granted - Kind: B2 - Application number: 18/514,340 - Filed: 2023-11-20 - Published: 2024-03-14 - Granted: 2025-02-04 - Earliest claimed: 2018-05-02 - Assignees: Quidient, LLC - Applicants: Quidient, LLC - Inventors: David Scott Ackerson; Donald J. Meagher; John K. Leffingwell - Web page: https://patentgrep.com/patent/US12217389B2 ## CPC - G06T 3/4092: Image resolution transcoding, e.g. by using client-server architectures - G06T 7/557: from light fields, e.g. from plenoptic cameras - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06T 3/067: Reshaping or unfolding three-dimensional [3D] tree structures onto two-dimensional [2D] planes - G06T 3/4076: using the original low-resolution images to iteratively correct the high-resolution images - G06T 9/001: {Model-based coding, e.g. wire frame} - G06T 9/40: Tree coding, e.g. quadtree, octree - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods and apparatus for systems using a scene codec are described, where systems are either providers or consumers of multi-way, just-in-time, only-as-needed scene data including subscenes and subscene increments. An example system using a scene codec includes a plenoptic scene database containing one or more digital models of scenes, where representations and organization of representations are distributable across multiple systems such that collectively the multiplicity of systems can represent scenes of almost unlimited detail. The system may further include highly efficient means for the processing of these representation and organizations of representation providing the just-in-time, only-as-needed subscenes and scene increments necessary for ensuring a maximally continuous user experience enabled by a minimal amount of newly provided scene information, where the highly efficient means include a spatial processing unit. ## Claims ### Claim 1 A scene decoder system comprising: a communication path configured to: transmit one or more requests for data related to a scene, wherein the one or more requests comprise: a request for data representing a first light field in the scene, and data related to one or more characteristics of the system; and receive a response to the request, wherein the response comprises: data representing the first light field in the scene, and additional data representing at least one additional light field in the scene, wherein the additional data is related to the one or more characteristics of the system; a processor configured to: monitor the characteristics of the system, determine contents of the one or more requests, process light field information, wherein the light field information comprises the data representing the first light field in the scene and the additional data related to the at least one additional light field, determine whether to output any portion of the additional data related to the at least one additional light field; and output at least a portion of the processed light field information, wherein the outputted processed light field information comprises: the processed light field information related to the data representing the first light field in the scene, and if the processor has determined to output at least a portion of the additional data related to the at least one additional light field, the portion of the processed light field information related to the additional data related to the at least one additional light field determined to be output by the processor. ### Claim 2 (depends on 1) The scene decoder system of claim 1 wherein the data related to one or more characteristics of the system comprises a request for data related to a subscene and the response includes a representation of a subscene. ### Claim 3 (depends on 1) The scene decoder system of claim 1 wherein the response further comprises data representing a matter field in the scene. ### Claim 4 (depends on 1) The scene decoder system of claim 1 wherein the system further comprises storage medium comprising a plenoptic scene model database organized in one or more of a spatially-sorted, hierarchical, and multi-resolution manner. ### Claim 5 (depends on 1) The scene decoder system of claim 1 wherein the processor is further configured to calculate a probability of making a future request for data representing a second light field in the scene. ### Claim 6 (depends on 5) The scene decoder system of claim 5 wherein the processor is further configured to initiate a second request for data related to the scene based at least in part on the probability. ### Claim 7 (depends on 1) The scene decoder system of claim 1 wherein data related to one or more characteristics of the system comprises an accumulation of scene experience logging, a direction of travel associated with the request, data provided as part of the request, data provided in a separate request, or one or more pieces of auxiliary scene information. ### Claim 8 (depends on 1) The scene decoder system of claim 1 wherein the additional data represents one or more of a higher resolution representation of a portion of the data representing the first light field in the scene, data representing at least part of the scene from a different viewpoint, or data representing at least part of the scene from a same viewpoint with a different direction of viewing. ### Claim 9 (depends on 1) The scene decoder system of claim 1 wherein the request specifies one or more characteristics of the response, and wherein the response includes data representing the first light field in the scene responsive to the specified characteristics. ### Claim 10 (depends on 9) The scene encoder system of claim 9 wherein the one or more characteristics includes a lighting condition associated with the scene and wherein the response comprises data reflecting the lighting condition. ### Claim 11 A method for decoding scene data comprising: transmitting one or more requests for scene data related to a scene, wherein the one or more requests comprise: a request for data representing a first light field in the scene, and data related to one or more characteristics of a system; and receiving a response to the request, wherein the response comprises: data representing the first light field in the scene, and additional data representing at least one additional light field in the scene, wherein the additional data is related to the one or more characteristics of the system; monitoring the characteristics of the system, determining contents of the one or more requests, processing light field information, wherein the light field information comprises the data representing the first light field in the scene and the additional data related to the at least one additional light field, determining whether to output any portion of the additional data related to the at least one additional light field; and outputting at least a portion of the processed light field information, wherein the outputted processed light field information comprises: the processed light field information related to the data representing the first light field in the scene, and if a processor has determined to output at least a portion of the additional data related to the at least one additional light field, the portion of the processed light field information related to the additional data related to the at least one additional light field determined to be output by the processor. ### Claim 12 (depends on 11) The method of claim 11 wherein the data related to one or more characteristics of the system comprises a request for data related to a subscene and the response includes a representation of a subscene. ### Claim 13 (depends on 11) The method of claim 11 wherein the response further comprises data representing a matter field in the scene. ### Claim 14 (depends on 11) The method of claim 11 wherein the method further comprises storing the response in a storage medium comprising a plenoptic scene model database organized in one or more of a spatially-sorted, hierarchical, and multi-resolution manner. ### Claim 15 (depends on 11) The method of claim 11 wherein the method further includes calculating a probability of making a future request for data representing a second light field in the scene. ### Claim 16 (depends on 15) The method of claim 15 wherein the method further comprises initiating a second request for data related to the scene based at least in part on the probability. ### Claim 17 (depends on 11) The method of claim 11 wherein the data related to one or more characteristics of the system comprises an accumulation of scene experience logging, a direction of travel associated with the request, data provided as part of the request, data provided in a separate request, or one or more pieces of auxiliary scene information. ### Claim 18 (depends on 11) The method of claim 11 wherein the additional data represents one or more of a higher resolution representation of a portion of the data representing the first light field in the scene, data representing at least part of the scene from a different viewpoint, or data representing at least part of the scene from a same viewpoint with a different direction of viewing. ### Claim 19 (depends on 11) The method of claim 11 wherein the request specifies one or more characteristics of the response, and wherein the response includes data representing the first light field in the scene responsive to the specified characteristics. ### Claim 20 A non-transitory computer readable storage medium having instructions stored thereon, wherein the instructions, when executed by at least one processor of a computer system, causes the computer system to perform operations comprising: transmitting one or more requests for scene data related to a scene, wherein the one or more requests comprise: a request for data representing a first light field in the scene, and data related to one or more characteristics of the system; and receiving a response to the request, wherein the response comprises: data representing the first light field in the scene, and additional data representing at least one additional light field in the scene, wherein the additional data is related to the one or more characteristics of the system; monitoring the characteristics of the system, determining contents of the one or more requests, processing light field information, wherein the light field information comprises the data representing the first light field in the scene and the additional data related to the at least one additional light field, determining whether to output any portion of the additional data related to the at least one additional light field; and outputting at least a portion of the processed light field information, wherein the outputted processed light field information comprises: the processed light field information related to the data representing the first light field in the scene, and if the processor has determined to output at least a portion of the additional data related to the at least one additional light field, the portion of the processed light field information related to the additional data related to the at least one additional light field determined to be output by the processor.