# US 12,002,223 B2 · Digital representation of multi-sensor data stream > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,223 B2 - Document ID: US12002223B2 - Status: Granted - Kind: B2 - Application number: 18/196,535 - Filed: 2023-05-12 - Published: 2023-09-07 - Granted: 2024-06-04 - Earliest claimed: 2020-03-31 - Assignees: Nant Holdings IP, LLC - Applicants: Nant Holdings IP, LLC - Inventors: Patrick Soon-Shiong - Web page: https://patentgrep.com/patent/US12002223B2 ## CPC - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06F 16/70: of video data - G09B 19/0038: {Sports} - H04N 7/181: {for receiving images from a plurality of remote sources} - G06T 2207/30221: Sports video; Sports image - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A computer-based filming production system is described. An illustrative system includes a real-world, physical environment having a set of sensors disposed about a filming volume and a production center computing system including at least one computer readable memory that enables a processor to store a prioritization policy including priorities for digital data streams and for each filming production workflow stage of a set of multiple filming production workflow stages, receive a set of sensor data representing a filming production in the filming production from the set of sensors, generate a set of digital data streams of renderable content from the set of sensor data via the filming production workflow stages and according to the priorities for each filming production workflow stage, where each digital data streams comprises prioritized rendering instructions for their respective renderable content derived from the prioritization policy, and then transmit the set of digital data streams. ## Claims ### Claim 1 A computer-based filming production system comprising: a real-world, physical environment having a set of sensors disposed about a filming volume; and a production center computing system including at least one computer readable memory storing software instructions and at least one processor coupled with the at least one computer readable memory and with the set of sensors, wherein the at least one processor performs, upon execution of the software instructions, the operations of: storing, in the at least one computer readable memory, a prioritization policy comprising priorities for digital data streams and for each filming production workflow stage of a set of multiple filming production workflow stages; receiving a set of sensor data representing a filming production in the filming production from the set of sensors; generating a set of digital data streams of renderable content from the set of sensor data via the filming production workflow stages and according to the priorities for each filming production workflow stage, wherein each digital data streams comprises prioritized rendering instructions for their respective renderable content derived from the prioritization policy; and transmitting the set of digital data streams to at least one endpoint device according to the prioritization policy thereby enabling the at least one endpoint device to render layers of the renderable content from each digital stream from the set of digital data streams according to their respective prioritized rendering instructions, wherein the prioritization policy of each digital data stream in the set of digital data streams comprises one or more rules to resolve priority conflicts with other digital data streams in the set of digital data streams. ### Claim 2 (depends on 1) The computer-based filming production system of claim 1, wherein the priorities for each digital data stream in the set of digital data streams represent rendering priorities for each digital data stream's renderable content. ### Claim 3 (depends on 1) The computer-based filming production system of claim 1, wherein each digital data stream in the set of digital data streams are transmitted individually. ### Claim 4 (depends on 3) The computer-based filming production system of claim 3, wherein the operation of transmitting the set of digital data streams includes broadcasting the set of digital data streams to multiple endpoint devices according to each stream's broadcast priorities from the prioritization policy. ### Claim 5 (depends on 1) The computer-based filming production system of claim 1, wherein the prioritization policy comprise at least two dimension of relevance. ### Claim 6 (depends on 5) The computer-based filming production system of claim 5, wherein the prioritization policy comprises a prioritization matrix. ### Claim 7 (depends on 6) The computer-based filming production system of claim 6, wherein the prioritization matrix has at least one axis representing layers of the renderable content and at least one axis representing the filming production workflow stages. ### Claim 8 (depends on 1) The computer-based filming production system of claim 1, wherein the filming production workflow stages include at least one of the following: a sensor data processing stage, a post-processing stage, a sensor data capture stage, an object recognition stage, an object identification stage, a rendering layer construction stage, a digital data stream compilation stage, a transmitting stage, and a receiving feedback stage. ### Claim 9 (depends on 1) The computer-based filming production system of claim 1, wherein each stage of the filming production workflow stages is managed by at least one queue. ### Claim 10 (depends on 9) The computer-based filming production system of claim 9, wherein each queue of the at least one queue manages stream tasks according to the data stream priorities of prioritization policy. ### Claim 11 (depends on 9) The computer-based filming production system of claim 9, wherein queues for the filming production workflow stages are managed according to stage priorities of prioritization policy. ### Claim 12 (depends on 1) The computer-based filming production system of claim 1, wherein the set of sensors includes at least one camera. ### Claim 13 (depends on 1) The computer-based filming production system of claim 1, wherein the filming volume comprises an LED wall. ### Claim 14 (depends on 13) The computer-based filming production system of claim 13, wherein the set of digital data streams include at least one stream representing captured background content displayed on the LED wall. ### Claim 15 (depends on 14) The computer-based filming production system of claim 14, wherein the operations further include receiving a feedback from the at least one endpoint device to alter the filming volume. ### Claim 16 (depends on 15) The computer-based filming production system of claim 15, wherein the feedback instructs the LED wall to take at least one of the following actions: change a rendering behavior for a scene, change perspectives, change parallax, modify a rendered object, change focus, change blur, and change lighting. ### Claim 17 (depends on 1) The computer-based filming production system of claim 1, wherein the prioritization policy comprises differencing instructions that determines differences among units of content in the set of digital data streams. ### Claim 18 (depends on 17) The computer-based filming production system of claim 17, wherein the differences comprise rendering gaps in the layers of renderable content. ### Claim 19 (depends on 18) The computer-based filming production system of claim 18, wherein the at least one endpoint devices is enabled to generate interpolated content to fill the rendering gaps according to interpolation instructions associated with the prioritization policy. ### Claim 20 (depends on 1) The computer-based filming production system of claim 1, wherein the production center system is at least in part located remotely over a network from the filming volume.