# US 2019/0379878 A1 · See-through operator awareness tool using synthesized viewpoints

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0379878 A1
- Document ID: US20190379878A1
- Status: Published application
- Kind: A1
- Application number: 16/006,726
- Filed: 2018-06-12
- Published: 2019-12-12
- Earliest claimed: 2018-06-12
- Assignees: Disney Enterprises, Inc.
- Applicants: Disney Enterprises, Inc.
- Inventors: Steven CHAPMAN; Mark ARANA; Michael GOSLIN; Joseph POPP; Mehul PATEL
- Web page: https://patentgrep.com/patent/US20190379878A1

## CPC

- G06F 3/012: {Head tracking input arrangements}
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation}
- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/344: with head-mounted left-right displays
- H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The disclosure is directed to providing operator visibility through an object that occludes the view of the operator by using a HMD system in communication with one or more imaging devices. A method of doing so may include: using one or more imaging devices coupled to an exterior of an object to capture image data of a real-world environment surrounding the object: calculating an orientation of a HMD positioned in an interior of the object; using at least the calculated orientation of the HMD, synthesizing a FOV of the image data with a FOV of the HMD; and rendering the synthesized FOV of the image data to the HMD. In some instances, the captured image data is used to create a three-dimensional model of the real-world environment, and synthesizing the FOV of the image data includes synthesizing a FOV of the 3D model with the FOV of the HMD.

## Claims

### Claim 1

A method, comprising:
using one or more imaging devices coupled to an exterior of an object to capture image data of a real-world environment surrounding the object;
calculating an orientation of a head mounted display (HMD) positioned in an interior of the object;
using at least the calculated orientation of the HMD, synthesizing a field of view (FOV) of the image data with a FOV of the HMD; and
rendering the synthesized FOV of the image data to the HMD.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more image devices comprise one or more video cameras, wherein the image data of the real-world environment comprises video of the real-world environment captured by the one or more video cameras, wherein synthesizing the FOV of the image data comprises synthesizing a FOV of the video captured by the one or more video cameras with the FOV of the HMD.

### Claim 3 (depends on 2)

The method of claim 2, wherein the one or more video cameras comprise a plurality of omnidirectional cameras.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
overlaying the synthesized FOV of the video captured by the one or more video cameras over video of a FOV of the HMD; or
overlaying video of a FOV of the HMD over the synthesized FOV of the video captured by the one or more video cameras.

### Claim 5 (depends on 4)

The method of claim 4, further comprising: render the overlay of the synthesized FOV of the video captured by the one or more video cameras and the video of the FOV of the HMD.

### Claim 6 (depends on 5)

The method of claim 5 wherein the video of the FOV of the HMD is ghosted in the overlay.

### Claim 7 (depends on 5)

The method of claim 5, wherein the object comprises a vehicle or costumed character.

### Claim 8 (depends on 5)

The method of claim 5, further comprising: presenting a graphical user interface to a user of the HMD, the graphical user interface comprising controls to modify the appearance of the overlay.

### Claim 9 (depends on 1)

The method of claim 1, further comprising: using the captured image data to create a three-dimensional model of the real-world environment, wherein synthesizing the FOV of the image data comprises synthesizing a FOV of the 3D model with the FOV of the HMD.

### Claim 10 (depends on 9)

The method of claim 9, wherein the one or more image devices comprise one or more video cameras, wherein the image data of the real-world environment comprises video of the real-world environment captured by the one or more video cameras.

### Claim 11

A non-transitory computer-readable medium having executable instructions stored thereon that, when executed by one or more processors, cause a system to:
use one or more imaging devices coupled to an exterior of an object to capture image data of a real-world environment surrounding the object;
calculate an orientation of a head mounted display (HMD) positioned in an interior of the object;
use at least the calculated orientation of the HMD to synthesize a field of view (FOV) of the image data with a FOV of the HMD; and
render the synthesized FOV of the image data to the HMD.

### Claim 12 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein the one or more image devices comprise one or more video cameras, wherein the image data of the real-world environment comprises video of the real-world environment captured by the one or more video cameras, wherein synthesizing the FOV of the image data comprises synthesizing a FOV of the video captured by the one or more video cameras with the FOV of the HMD.

### Claim 13 (depends on 12)

The non-transitory computer-readable medium of claim 12, wherein the one or more video cameras comprise a plurality of omnidirectional cameras.

### Claim 14 (depends on 13)

The non-transitory computer-readable medium of claim 13, wherein the instructions, when executed by the one or more processors, further cause the system to:
overlay the synthesized FOV of the video captured by the one or more video cameras over video of a FOV of the HMD; or
overlay video of a FOV of the HMD over the synthesized FOV of the video captured by the one or more video cameras.

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the instructions, when executed by the one or more processors, further cause the system to: render the overlay of the synthesized FOV of the video captured by the one or more video cameras and the video of the FOV of the HMD.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15 wherein the video of the FOV of the HMD is ghosted in the overlay.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the object comprises a vehicle or costumed character.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed by the one or more processors, further cause the system to: present a graphical user interface to a user of the HMD, the graphical user interface comprising controls to modify the appearance of the overlay.

### Claim 19 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein the instructions, when executed by the one or more processors, further cause the system to: use the captured image data to create a three-dimensional model of the real-world environment, wherein synthesizing the FOV of the image data comprises synthesizing a FOV of the 3D model with the FOV of the HMD.

### Claim 20 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the one or more image devices comprise one or more video cameras, wherein the image data of the real-world environment comprises video of the real-world environment captured by the one or more video cameras.

### Claim 21

A system, comprising:
a plurality of omnidirectional video cameras adapted to be mounted on an exterior of an object; and
a head mounted display (HMD) system comprising a HMD, the HMD system configured to:
receive video captured by the plurality of omnidirectional video cameras of a real-world environment surrounding the object;
calculate an orientation of the HMD while it is positioned in an interior of the object;
using at least the calculated orientation of the HMD, synthesize a field of view (FOV) of the captured video with a FOV of the HMD; and
render the synthesized FOV of the video to the HMD.
