# US 12,118,673 B2 · Method and system for displaying a virtual scene

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,673 B2
- Document ID: US12118673B2
- Status: Granted
- Kind: B2
- Application number: 17/864,070
- Filed: 2022-07-13
- Published: 2024-01-18
- Granted: 2024-10-15
- Earliest claimed: 2022-07-13
- Assignees: Orqa Holding LTD
- Applicants: Orqa Holding LTD
- Inventors: Srdjan Kovacevic; Ana Petrinec
- Web page: https://patentgrep.com/patent/US12118673B2

## CPC

- G06T 19/003: {Navigation within 3D models or images}
- G03B 37/04: with cameras or projectors providing touching or overlapping fields of view
- G06T 15/005: {General purpose rendering architectures}
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- H04N 7/181: {for receiving images from a plurality of remote sources}
- H04N 7/183: {for receiving images from a single remote source}
- H04N 9/3179: {Video signal processing therefor}
- H04N 9/3194: {including sensor feedback}
- H04N 23/662: {by using camera arrangements where one camera controls another camera to affect the control of camera image capture, e.g. placing the camera in a desirable condition to capture a desired image}
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method for positioning of cameras on an object that enables accurate rendering of the scene around the object on a dome accurately in real time. The method involves providing a 3D model of the object having a surface, and selecting locations on the surface where the cameras are to be placed to provide a camera rig. The choice of locations is such that every camera has a field of view that overlaps with at least one other camera. The cameras are designated as Direct View Camera (DVC) or a Secondary View Camera (SVC). The method to render a virtual scene includes providing a projection rig that includes a plurality of projectors within a hollow half-sphere, wherein the hollow half-sphere includes an inner surface and an outer surface. Each of the plurality of projectors is designated as a Direct View Projector (DVP) or a Secondary View Projector (SVP).

## Claims

### Claim 1

A method for displaying a virtual scene, the method comprising:
forming a camera rig by positioning a plurality of cameras on an object; the plurality of cameras comprising a plurality of Direct View Cameras and at least one Secondary View Camera;
positioning the plurality of Direct View Cameras and the at least one Secondary View Camera at selected locations on the surface of the object; wherein the plurality of Direct View Cameras are positioned in a non-circular arrangement;
forming a projecting rig that comprises a plurality of projectors and a dome; the plurality of projectors comprising a plurality of Direct View Projectors and at least one Secondary View Projector,
associating each of the plurality of Direct View Projectors with a corresponding one of the plurality of Direct View Cameras, and positioning each of the plurality of Direct View Projectors such that each of the plurality of Direct View Projectors comprises a relative direction and a spatial position identical to a direction and a spatial position of the corresponding Direct View Camera, wherein each of the plurality of Direct View Projectors is capable of receiving images from the corresponding Direct View Camera,
associating the at least one Secondary View Projector with a corresponding one of the at least one Secondary View Camera, wherein each Secondary View Projector is capable of receiving images from the corresponding Secondary View Camera; and
projecting by all of the plurality of projectors the images received from each corresponding camera onto an inner surface of the dome.

### Claim 2 (depends on 1)

The method of claim 1 further comprising predefining a number of the Direct View Cameras and the Secondary View Cameras to be placed.

### Claim 3 (depends on 1)

The method of claim 1 wherein the Direct View Camera is positioned such that there is an overlap between a field of view of the Direct View Camera with at least one other of the cameras.

### Claim 4 (depends on 1)

The method of claim 1 wherein the selected locations are selected to align to a specific geometry.

### Claim 5 (depends on 1)

The method of claim 1 wherein the object is a vehicle.

### Claim 6 (depends on 1)

The method of claim 1 wherein the object is a 3D model.

### Claim 7 (depends on 1)

The method of claim 1 further comprising positioning the at least one Secondary View Projector such that the at least one Secondary View Projector comprises a direction and spatial position that is different than a direction and spatial position of the corresponding Secondary View Camera.

### Claim 8

A method of displaying a virtual scene, the method comprising:
providing a camera rig with a plurality of cameras, comprising:
providing a 3D model of an object, wherein the 3D model comprises a surface;
designating each camera of the plurality of cameras as a Direct View Camera or a Secondary View Camera; the plurality of cameras comprising a plurality of Direct View Cameras and at least one Secondary View Camera,
specifying a direction, a rotation along a longitudinal axis and a field of view for each camera of the plurality of cameras; and
positioning the Direct View Cameras and the at least one Secondary View Camera at selected locations on the surface of the 3D model of the object; wherein the plurality of Direct View Cameras are positioned in a non-circular arrangement;
providing a projection rig that comprises a plurality of projectors within a hollow half-sphere, wherein the hollow half-sphere comprises an inner surface and an outer surface;
designating each projector of the plurality of projectors as a Direct View Projector or a Secondary View Projector;
positioning each of the Direct View Projector such that each Direct View Projector corresponds to a Direct View Camera by having a spatial position, direction, rotation along a longitudinal axis and field of view identical to the spatial position, direction, rotation along the longitudinal axis and field of view of the corresponding Direct View Camera and capable of receiving images from the corresponding Direct View Camera;
associating each of the Secondary View Projectors with a corresponding Secondary View Camera such that each of the Secondary View Projectors is capable of receiving images from the corresponding Secondary View Camera;
positioning of each of the Secondary View Projectors in a manner such that at least one of the Secondary View Projectors comprises a spatial position, direction, rotation along a longitudinal axis and field of view that is different than a spatial position, direction, rotation along a longitudinal axis and field of view of the corresponding Secondary View Camera; and
projecting by all the projectors the images obtained from each corresponding camera onto the inner surface of the hollow half-sphere.

### Claim 9 (depends on 8)

The method of claim 8 wherein the spatial position is defined by Cartesian coordinates.

### Claim 10 (depends on 8)

The method of claim 8 wherein the direction is defined by pitch and yaw.

### Claim 11 (depends on 8)

The method of claim 8 wherein the rotation along the longitudinal axis is defined by roll.

### Claim 12 (depends on 8)

The method of claim 8 wherein for an on-board vehicle operation, where a dome is effectively fixed to a vehicle's frame of reference by virtue of the camera rig being fixed to the vehicle and a projection rig being fixed to the dome, movements of an operator's viewpoint in the dome are driven by the movements of a headset relative to the vehicle, while the movements of the vehicle relative to a global frame of reference are ignored.

### Claim 13 (depends on 12)

The method of claim 12, wherein the movements of the operator's viewpoint in the dome are determined either by relying on visual sensors in 6 degrees of freedom (6DoF) head tracking, or by subtracting the movement of the vehicle from the movement of the headset by combining measurement data from inertial sensors in the headset with measure data from the one affixed to the vehicle.

### Claim 14

A system for displaying a virtual scene, the system comprising:
a 3D model of an object comprising a surface;
a camera rig that comprises a plurality of cameras positioned at selected locations on the surface of the 3D model of the object, wherein each camera of the plurality of cameras is designated as a Direct View Camera or a Secondary View Camera, the plurality of cameras comprises a plurality of Direct View Cameras and at least one Secondary View Camera, wherein each camera of the plurality of cameras is associated with a direction, a rotation along a longitudinal axis and a field of view; the plurality of Direct View Cameras being positioned in a non-circular arrangement; and
a projection rig that comprises a plurality of projectors within a hollow half-sphere, wherein the hollow half-sphere comprises an inner surface and an outer surface, wherein each projector of the plurality of projectors is designated as a Direct View Projector or a Secondary View Projector, the plurality of projectors comprises a plurality of Direct View Projectors and at least one Secondary View Projector, wherein
the plurality of projectors are positioned such that each Direct View Projector corresponds to a Direct View Camera comprising identical spatial position, direction, rotation along a longitudinal axis and field of view of the corresponding Direct View Camera and capable of receiving images from the corresponding Direct View Camera,
the at least one Secondary View Projector corresponds to the at least one Secondary View Camera on the camera rig and is capable of receiving images from the corresponding Secondary View Camera, wherein the at least one Secondary View Projector comprises a spatial position that is different than the spatial position of the corresponding Secondary View Camera, and
all the projectors of the plurality of projectors are configured to project the images obtained from the corresponding cameras onto the inner surface of the hollow half-sphere.

### Claim 15 (depends on 14)

The system of claim 14 wherein the spatial position is defined by Cartesian coordinates.

### Claim 16 (depends on 14)

The system of claim 14 wherein the direction is defined by pitch and yaw.

### Claim 17 (depends on 14)

The system of claim 14 wherein the rotation along the longitudinal axis is defined by roll.

### Claim 18 (depends on 14)

The system of claim 14 wherein for remote vehicle operation, where the camera rig is attached to a vehicle, a degree of projection stabilization is implemented in order to reduce intensity of movement of virtual horizon by moving the projection rig in an opposite direction in relations to the vehicle.

### Claim 19 (depends on 18)

The system of claim 18, wherein the movement of the projection rig is determined by equipping the vehicle with an Inertial Measurement Unit (the “IMU”), and/or using a live video feed from DVCs and SVCs to perform Visual Inertial Odometry (“VIO”) in order to track the movement of the vehicle, and then using pitching data and rolling data of the vehicle to rotate the projection rig within a dome in the opposite direction.