# US 12,178,403 B2 · Light field capture and rendering for head-mounted displays

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,178,403 B2
- Document ID: US12178403B2
- Status: Granted
- Kind: B2
- Application number: 18/188,774
- Filed: 2023-03-23
- Published: 2024-01-04
- Granted: 2024-12-31
- Earliest claimed: 2023-03-23
- Assignees: University of Washington
- Applicants: University of Washington
- Inventors: Joshua R. Smith; Samuel R. Browd; Rufus Griffin Nicoll; James Andrew Youngquist
- Web page: https://patentgrep.com/patent/US12178403B2

## CPC

- A61B 1/00193: {adapted for stereoscopic vision}
- A61B 1/000095: {for image enhancement}
- A61B 1/00048: {Constructional features of the display}
- A61B 1/00101: {the distal tip features being detachable}
- A61B 1/00181: {for multiple fixed viewing angles (A61B1/00193 takes precedence)}
- A61B 1/00194: {adapted for three-dimensional imaging}
- A61B 1/045: Control thereof
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/254: in combination with electromagnetic radiation sources for illuminating objects
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 13/344: with head-mounted left-right displays
- H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
- H04N 23/957: Light-field or plenoptic cameras or camera modules
- A61B 90/30: Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
- A61B 2017/00216: {with eye tracking or head position tracking control}
- A61B 2034/2048: {using an accelerometer or inertia sensor}
- A61B 2034/2051: {Electromagnetic tracking systems}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/2063: {Acoustic tracking systems, e.g. using ultrasound}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2090/372: {Details of monitor hardware}
- A61B 2576/00: Medical imaging apparatus involving image processing or analysis (A61B1/00009, A61B6/52 and A61B8/52 take precedence)
- G16H 30/40: for processing medical images, e.g. editing
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Systems and methods for capturing and rendering light fields for head-mounted displays are disclosed. A mediated-reality visualization system includes a head-mounted display assembly comprising a frame configured to be mounted to a user's head and a display device coupled to the frame. An imaging assembly separate and spaced apart from the head-mounted display assembly is configured to capture light-field data. A computing device in communication with the imaging assembly and the display device is configured to receive light-field data from the imaging assembly and render one or more virtual cameras. Images from the one or more virtual cameras are presented to a user via the display device.

## Claims

### Claim 1

A method of imaging a physical scene, the method comprising:
receiving image data of the physical scene from a camera assembly having a plurality of cameras configured to capture the image data;
rendering a virtual camera;
generating an image from the virtual camera by determining a value for each of a plurality of virtual pixels in the image from the virtual camera, wherein determining the value for each of the plurality of virtual pixels comprises—
determining a point in the physical scene corresponding to the virtual pixel along a virtual ray;
determining a first pixel in the image data from the first one of the cameras that images the point in the physical scene along a first ray;
determining a second pixel in the image data from the second one of the imaging devices that images the point in the physical scene along a second ray; and
weighting a value of the first pixel and a value of the second pixel to determine a value of the virtual pixel based on (i) a first angular distance between the first ray and the virtual ray, (ii) a second angular distance between the second ray and the virtual ray, and (iii) a third angular distance between the virtual ray and a surface normal at the point in the physical scene; and
displaying the image from the virtual camera at a display, wherein the camera assembly is separate and spaced apart from the display.

### Claim 2 (depends on 1)

The method of claim 1
wherein the cameras are each directed toward and focused on a generally common point in the physical scene.

### Claim 3 (depends on 1)

The method of claim 1 wherein the image data comprises light field image data.

### Claim 4 (depends on 1)

The method of claim 1 wherein the method further comprises determining a surface geometry of the physical scene.

### Claim 5 (depends on 1)

The method of claim 1 wherein the image from the virtual camera comprises a non-planar focal surface.

### Claim 6 (depends on 5)

The method of claim 5 wherein the method further comprises determining a surface geometry of the physical scene, and wherein the non-planar focal surface corresponds to the determined surface geometry of the physical scene.

### Claim 7 (depends on 1)

The method of claim 1 wherein the camera assembly comprises a rigid frame, and wherein the cameras are mounted to frame.

### Claim 8 (depends on 1)

The method of claim 1 wherein the method further comprises continuously receiving the image data, processing the image data, generating the image from the virtual camera, and displaying the image from the virtual camera at the display.

### Claim 9 (depends on 1)

The method of claim 1 wherein the camera assembly is coupled to an articulated arm positionable over the physical scene.

### Claim 10 (depends on 1)

The method of claim 1 wherein the cameras each have a different perspective of the physical scene, and wherein the virtual camera has a perspective of the physical scene different from any of the cameras.

### Claim 11 (depends on 1)

The method of claim 1 wherein the physical scene comprises a surgical scene.

### Claim 12 (depends on 1)

The method of claim 1 wherein the method further comprises tracking a position of the camera assembly relative to the physical scene.

### Claim 13 (depends on 1)

The method of claim 1 wherein the display is a head-mounted display.

### Claim 14

A system for imaging a physical scene, the system comprising:
a camera assembly including a plurality of cameras configured to capture image data of the physical scene;
a computing device communicatively coupled to the camera assembly, wherein the computing device has a memory containing computer-executable instructions and a processor for executing the computer-executable instructions contained in the memory, and wherein the computer-executable instructions, when executed by the processor, cause the processor to—
receive the image data from the camera assembly;
render a virtual camera; and
generate an image from the virtual camera by determining a value for each of a plurality of virtual pixels in the image from the virtual camera, wherein determining the value for each of the plurality of virtual pixels comprises—
determining a point in the physical scene corresponding to the virtual pixel along a virtual ray;
determining a first pixel in the image data from a first one of the cameras corresponding to the point in the physical scene along a first ray;
determining a second pixel in the image data from a second one of the cameras corresponding to the point in the physical scene along a second ray; and
weighting a value of the first pixel and a value of the second pixel to determine a value of the virtual pixel based on (i) a first angular distance between the first ray and the virtual ray, (ii) a second angular distance between the second ray and the virtual ray, and (iii) a third angular distance between the virtual ray and a surface normal at the point in the physical scene; and
a display communicatively coupled to the computing device, wherein the display is separate and spaced apart from the camera assembly, and wherein the display is configured to receive the image from the virtual camera from the computing device and display the image from the virtual camera.

### Claim 15 (depends on 14)

The system of claim 14
wherein the cameras are each directed toward and focused on a generally common point in the physical scene.

### Claim 16 (depends on 15)

The system of claim 15 wherein the camera assembly comprises a rigid frame, wherein the cameras are mounted to frame, and wherein the camera assembly is coupled to an articulated arm positionable over the physical scene.

### Claim 17 (depends on 14)

The system of claim 14 wherein the cameras each have a different perspective of the physical scene, and wherein the virtual camera has a perspective of the physical scene different from any of the cameras.

### Claim 18 (depends on 14)

The system of claim 14 wherein the physical scene comprises a surgical scene.

### Claim 19 (depends on 14)

The system of claim 14 wherein the image data comprises light field image data.

### Claim 20 (depends on 14)

The system of claim 14, further comprising a tracking device communicatively coupled to the computing device, wherein the tracking device is configured to track a position of the camera assembly relative to the physical scene.