# US 2020/0404236 A1 · Methods and Apparatus for Supporting Content Generation, Transmission and/or Playback > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0404236 A1 - Document ID: US20200404236A1 - Status: Published application - Kind: A1 - Application number: 16/917,333 - Filed: 2020-06-30 - Published: 2020-12-24 - Earliest claimed: 2015-03-01 - Assignees: Nevermind Capital LLC - Applicants: Nevermind Capital LLC - Inventors: David Cole; Alan McKay Moss - Web page: https://patentgrep.com/patent/US20200404236A1 ## CPC - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06T 17/05: Geographic models - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G09G 3/003: {to produce spatial visual effects} - G09G 5/001: {Arbitration of resources in a display system, e.g. control of access to frame buffer by video controller and/or main processor} - G09G 5/003: {Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto (specific for a CRT G09G1/165; for a flat panel G09G3/2092)} - G09G 5/006: {Details of the interface to the display terminal (specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04)} - H04N 13/139: Format conversion, e.g. of frame-rate or size - H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) - H04N 13/189: Recording image signals; Reproducing recorded image signals - H04N 13/194: Transmission of image signals - H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00) - H04N 13/232: using fly-eye lenses, e.g. arrangements of circular lenses - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - H04N 13/344: with head-mounted left-right displays - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - H04N 21/4345: {Extraction or processing of SI, e.g. extracting service information from an MPEG stream} - G06T 2207/30244: Camera pose - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G09G 2352/00: Parallel handling of streams of display data - G09G 2360/02: Graphics controller able to handle multiple formats, e.g. input or output formats - G09G 2360/125: using unified memory architecture [UMA] - G09G 2370/02: Networking aspects - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Methods and apparatus for supporting the capture of images of surfaces of an environment visible from a default viewing position and capturing images of surfaces not visible from the default viewing position, e.g., occluded surfaces, are described. Occluded and non-occluded image portions are packed into one or more frames and communicated to a playback device for use as textures which can be applied to a model of the environment where the images were captured. An environmental model includes a model of surfaces which are occluded from view from a default viewing position but which may be viewed is the user shifts the user's viewing location. Occluded image content can be incorporated directly into a frame that also includes non-occluded image data or sent in frames of a separate, e.g., auxiliary content stream that is multiplexed with the main content stream which communicates image data corresponding to non-occluded environmental portions. ## Claims ### Claim 1 (canceled) ### Claim 2 A method of operating a system, comprising: receiving, from a playback device, a first request for content; in accordance with the first request for content and a determination that the playback device is configured to enable multiple viewing positions, determining whether the playback device is configured to demultiplex at least two data streams; and in accordance with a determination that the playback device is not configured to demultiplex at least two data streams, transmitting a first frame to the playback device, wherein the first frame comprises non-occluded image data and occluded image data, wherein the non-occluded image data corresponds to a first portion of an environment visible from a first viewing position, and wherein the occluded image data corresponds to a second portion of the environment not visible from the first viewing position and visible from a second viewing position. ### Claim 2 (depends on 2) The method of claim 2, further comprising: receiving, from the playback device, a second request for content; and in accordance with the second request and a determination that the playback device is not configured to enable multiple viewing positions, transmitting a second frame to the playback device comprising the non-occluded image data. ### Claim 3 (depends on 3) The method of claim 3, wherein the first and second requests for content comprise data indicative of whether the playback device is configured to enable multiple viewing positions. ### Claim 4 (depends on 3) The method of claim 3, wherein the non-occluded image data comprises non-occluded image content, a first UV map, and a first environmental model. ### Claim 5 (depends on 5) The method of claim 5, wherein the occluded image data comprises occluded image content, a second UV map, and a second environmental model. ### Claim 6 (depends on 6) The method of claim 6, further comprising: receiving, from a second playback device, a third request for content; determining whether the second playback device is configured to demultiplex at least two data streams; and in accordance with a determination that the second playback device is configured to demultiplex at least two data streams, transmit a primary data stream and an auxiliary data stream to the second playback device, wherein the primary data stream comprises the second frame, wherein the auxiliary data stream comprises a third frame, and wherein the third frame comprises the occluded image data. ### Claim 7 (depends on 7) The method of claim 7, wherein the occluded image data comprises occluded image content, a third UV map, and a third environmental model. ### Claim 8 (depends on 8) The method of claim 8, wherein: the first environmental model corresponds to the first portion of the environment visible from the first viewing position; the first UV map is indicative of a mapping of the non-occluded image content to the first environmental model; the second environmental model comprises a first segment corresponding to the first portion of the environment and a second segment corresponding to the second portion of the environment that is not visible from the first viewing position and visible from the second viewing position; the second UV map is indicative of a mapping of the non-occluded image content to the first segment and the occluded image content to the second segment; the third environmental model corresponds to the second portion of the environment; and the third UV map is indicative of a mapping of the occluded image content to the third environmental model. ### Claim 9 (depends on 9) The method of claim 9, further comprising: receiving a first image and a second image, wherein the first image is captured by a first camera and comprises the non-occluded image content, wherein the second image is captured by a second camera and comprises the occluded image content; and generating the first frame, the second frame, and the third frame based on the first and second images. ### Claim 10 (depends on 10) The method of claim 10, further comprising generating the first, second, and third UV maps. ### Claim 11 (depends on 9) The method of claim 9, further comprising: receiving a depth map of an environment; and generating the first, second, and third environmental models based on the depth map. ### Claim 12 A non-transitory computer readable medium comprising computer code, executable by one or more processors to: receive, from a playback device, a request for content; determine whether the playback device is configured to enable multiple viewing positions; in accordance with a determination that the playback device is not configured to enable multiple viewing positions, transmit a first frame to the playback device, wherein the first frame comprises non-occluded image data corresponding to a first portion of an environment visible from a first viewing position; in accordance with a determination that the playback device is configured to enable multiple viewing positions, determine whether the playback device is configured to demultiplex at least two data streams; in accordance with a determination that the playback device is not configured to demultiplex at least two data streams, transmit a second frame to the playback device, wherein the second frame comprises the non-occluded image data and occluded image data corresponding to a second portion of the environment not visible from the first viewing position and visible from a second viewing position; and in accordance with a determination that the playback device is configured to demultiplex at least two data streams, transmit a primary data stream and an auxiliary data stream to the playback device, wherein the primary data stream comprises the first frame, wherein the auxiliary data stream comprises a third frame, and wherein the third frame comprises the occluded image data. ### Claim 13 (depends on 13) The non-transitory computer readable medium of claim 13, wherein the request for content comprises data indicative of whether the playback device is configured to enable multiple viewing positions. ### Claim 14 (depends on 13) The non-transitory computer readable medium of claim 13, wherein the first frame further comprises a first UV map and a first environmental model. ### Claim 15 (depends on 15) The non-transitory computer readable medium of claim 15, wherein: the first environmental model corresponds to a first portion of an environment visible from the first viewing position; and the first UV map is indicative of a mapping of non-occluded image content to the first environmental model. ### Claim 16 (depends on 13) The non-transitory computer readable medium of claim 13, wherein the second frame further comprises a second UV map and a second environmental model. ### Claim 17 (depends on 17) The non-transitory computer readable medium of claim 17, wherein: the second environmental model comprises a first segment corresponding to a first portion of an environment visible from the first viewing position and a second segment corresponding to a second portion of the environment that is not visible from the first viewing position and visible from the second viewing position; and the second UV map is indicative of a mapping of non-occluded image content to the first segment and occluded image content to the second segment. ### Claim 18 (depends on 13) The non-transitory computer readable medium of claim 13, wherein the third frame further comprises a third UV map and a third environmental model. ### Claim 19 (depends on 19) The non-transitory computer readable medium of claim 19, wherein: the third environmental model corresponds to a second portion of an environment that is not visible from the first viewing position and visible from the second viewing position; and the third UV map is indicative of a mapping of occluded image content to the third environmental model. ### Claim 20 A system, comprising: one or more processors; and one or more computer readable media comprising computer readable code executable by the one or more processors to: receive, from a playback device, a request for content; determine whether the playback device is configured to enable multiple viewing positions; in accordance with a determination that the playback device is not configured to enable multiple viewing positions, transmit a first frame to the playback device, wherein the first frame comprises non-occluded image data corresponding to a first portion of an environment visible from a first viewing position; in accordance with a determination that the playback device is configured to enable multiple viewing positions, determine whether the playback device is configured to demultiplex at least two data streams; in accordance with a determination that the playback device is not configured to demultiplex at least two data streams, transmit a second frame to the playback device, wherein the second frame comprises the non-occluded image data and occluded image data corresponding to a second portion of the environment not visible from the first viewing position and visible from a second viewing position; and in accordance with a determination that the playback device is configured to demultiplex at least two data streams, transmit a primary data stream and an auxiliary data stream to the playback device, wherein the primary data stream comprises the first frame, wherein the auxiliary data stream comprises a third frame, and wherein the third frame comprises the occluded image data. ### Claim 21 (depends on 21) The system of claim 21, wherein the request for content comprises data indicative of whether the playback device is configured to enable multiple viewing positions. ### Claim 22 (depends on 21) The system of claim 21, wherein the first frame further comprises a first UV map and a first environmental model. ### Claim 23 (depends on 21) The system of claim 21, wherein the second frame further comprises a second UV map and a second environmental model. ### Claim 24 (depends on 21) The system of claim 21, wherein the third frame further comprises a third UV map and a third environmental model.