# US 10,311,768 B2 · Virtual window > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,768 B2 - Document ID: US10311768B2 - Status: Granted - Kind: B2 - Application number: 14/604,750 - Filed: 2015-01-25 - Published: 2015-11-05 - Granted: 2019-06-04 - Earliest claimed: 2014-05-02 - Applicants: Paul Lapstun - Inventors: Paul Lapstun - Web page: https://patentgrep.com/patent/US10311768B2 ## CPC - G09G 3/02: by tracing or scanning a light beam on a screen - G02B 26/08: for controlling the direction of light (in light guides G02B6/35) - G02B 26/0833: {the reflecting element being a micromechanical device, e.g. a MEMS mirror, DMD (G02B26/0825 takes precedence; micromechanical devices in general B81B)} - G02B 26/085: {the reflecting means being moved or deformed by electromagnetic means} - G02B 26/0875: {by means of one or more refracting elements} - G02B 26/10: Scanning systems - G02B 26/101: {with both horizontal and vertical deflecting means, e.g. raster or XY scanners (colour television using laser beams scanning a display screen H04N9/3129)} - G02B 27/0075: {with means for altering, e.g. increasing, the depth of field or depth of focus} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 30/10: using integral imaging methods - G02B 30/27: involving lenticular arrays - G02F 1/11: based on acousto-optical elements, e.g. using variable diffraction by sound or like mechanical waves (acousto-optical deflection G02F1/33) - G02F 1/29: for the control of the position or the direction of light beams, i.e. deflection - G02F 1/33: Acousto-optical deflection devices {(circuit or control arrangements therefor G02F1/113)} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G09G 3/003: {to produce spatial visual effects} - G09G 5/10: Intensity circuits - H04N 13/243: using three or more two-dimensional [2D] image sensors - H04N 13/302: for viewing without the aid of special glasses, i.e. using autostereoscopic displays - H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes - H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils - H04N 23/56: provided with illuminating means - H04N 23/61: based on recognised objects - H04N 23/67: Focus control based on electronic image sensor signals - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - H04N 23/957: Light-field or plenoptic cameras or camera modules - G02F 1/294: {Variable focal length devices} - G09G 2320/064: by time modulation of the brightness of the illumination source - G09G 2354/00: Aspects of interface with display user - G09G 2360/141: the light conveying information used for selecting or modulating the light emitting or modulating element - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G02F: OPTICAL DEVICES OR ARRANGEMENTS FOR THE CONTROL OF LIGHT BY MODIFICATION OF THE OPTICAL PROPERTIES OF THE MEDIA OF THE ELEMENTS INVOLVED THEREIN; NON-LINEAR OPTICS; FREQUENCY-CHANGING OF LIGHT; OPTICAL LOGIC ELEMENTS; OPTICAL ANALOGUE/DIGITAL CONVERTERS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A virtual window system, the system comprising a light field camera device and a light field display device arranged back-to-back, the light field camera device configured to capture an input light field video stream, the light field display device configured to display an output light field video stream based on the input light field video stream. ## Claims ### Claim 1 A virtual window system, the system comprising a light field camera device and a light field display device arranged back-to-back, the light field camera device configured to sample a continuous input light field at an input light field boundary to produce a discrete input light field video stream, the light field display device configured to reconstruct a continuous output light field at an output light field boundary from a discrete output light field video stream, wherein the system is configured to resample the discrete input light field video stream to translate it from the input light field boundary to the output light field boundary to produce the discrete output light field video stream, as if the discrete output light field video stream had been sampled from the continuous input light field at the output light field boundary. ### Claim 2 (depends on 1) The system of claim 1, wherein the light field camera device comprises at least one input focus modulator, whereby the focus of at least part of the input light field video stream is modulated according to a fixation depth of a viewer of the light field display device. ### Claim 3 (depends on 1) The system of claim 1, wherein the light field display device comprises at least one output focus modulator, whereby the focus of at least part of the output light field video stream is modulated according to a fixation depth of a viewer of the light field display device. ### Claim 4 (depends on 1) The system of claim 1, wherein the light field camera device comprises at least one input focus modulator and the light field display device comprises at least one output focus modulator, whereby the focus of at least part of the input light field video stream and the focus of at least part of the output light field video stream are modulated, respectively, according to a fixation depth of a viewer of the light field display device. ### Claim 5 (depends on 1) The system of claim 1, wherein the input light field video stream is incorporated into a rendered light field video stream to produce the output light field video stream, the rendered light field video stream comprising at least one virtual 3D object. ### Claim 6 (depends on 5) The system of claim 5, wherein the rendered light field video stream comprises at least one virtual 3D object which interacts with part of the input light field video stream, the interaction selected from the group comprising: reflection and refraction. ### Claim 7 (depends on 1) The system of claim 1, wherein the light field camera device comprises an array of light field camera elements, each camera element comprising: (a) an array of radiance sensors for sensing the radiance of a set of input beams over time, the input beams spanning a two-dimensional angular field; and (b) an input focus modulator for modulating the focus of the input beams over time. ### Claim 8 (depends on 1) The system of claim 1, wherein the light field camera device comprises an array of light field camera elements, each camera element comprising: (a) a scanner for scanning an input beam across a two-dimensional angular field; (b) an input focus modulator for modulating the focus of the input beam over time; and (c) a radiance sensor for sensing the radiance of the input beam over time. ### Claim 9 (depends on 1) The system of claim 1, wherein the light field display device comprises an array of light field display elements, each display element comprising: (a) an array of beam generators and radiance modulators for generating and modulating the radiance of a set of output beams over time, the output beams spanning a two-dimensional angular field; and (b) an output focus modulator for modulating the focus of the output beams over time. ### Claim 10 (depends on 1) The system of claim 1, wherein the light field display device comprises an array of light field display elements, each display element comprising: (a) a beam generator for generating an output beam of light; (b) a radiance modulator for modulating the radiance of the output beam over time; (c) an output focus modulator for modulating the focus of the output beam over time; and (d) a scanner for scanning the output beam across a two-dimensional angular field. ### Claim 11 A method of providing a virtual window through a space, the window having an input light field boundary and an output light field boundary, the light field boundaries separated by the space, the method comprising: (a) sampling, using a light field camera device, a continuous input light field at the input light field boundary to produce a discrete input light field video stream; (b) resampling the discrete input light field video stream to translate it from the input light field boundary to the output light field boundary to produce a discrete output light field video stream, as if the discrete output light field video stream had been sampled from the continuous input light field at the output light field boundary; and (c) reconstructing, using a light field display device, a continuous output light field at the output light field boundary from the discrete output light field video stream.