US 2010/0245387 A1 · SYSTEMS AND METHODS FOR COMBINING VIRTUAL AND REAL-TIME PHYSICAL ENVIRONMENTS

Abstract

Systems, methods and structures for combining virtual reality and real-time environment by combining captured real-time video data and real-time 3D environment renderings to create a fused, that is, combined environment, including capturing video imagery in RGB or HSV/HSV color coordinate systems and processing it to determine which areas should be made transparent, or have other color modifications made, based on sensed cultural features, electromagnetic spectrum values, and/or sensor line-of-sight, wherein the sensed features can also include electromagnetic radiation characteristics such as color, infra-red, ultra-violet light values, cultural features can include patterns of these characteristics, such as object recognition using edge detection, and whereby the processed image is then overlaid on, and fused into a 3D environment to combine the two data sources into a single scene to thereby create an effect whereby a user can look through predesignated areas or “windows” in the video image to see into a 3D simulated world, and/or see other enhanced or reprocessed features of the captured image.

Claims20

Dependent claims

A virtual reality system comprising:

a head mounted camera and one or more fixed cameras adapted to receive images of a physical environment and to produce a frame of the physical environment that contains pixels representing at least a first range of electromagnetic spectrum values;
at least one range of predetermined target threshold electromagnetic spectrum values;
a means for processing the frame and rendering pixels having values within the target threshold range of values to be transparent pixels;
a means for generating a virtual image;
a means for combining the virtual image with the frame of the physical environment to form a combined image whereby the virtual image is visible at all positions of the transparent pixels;
and,
a means for displaying the combined image.

4 more elements

A method for combining virtual reality environments and real-time physical environments for a user comprising:

identifying a target pixel depth;
providing the real-time physical environment, the real-time physical environment having a predesignated area to be overlaid with virtual reality images and having the target pixel depth corresponding a predetermined distance to an object in the predesignated area;
identifying the predesignated area via pixel depth, whereby the depth is determined by distance from the object in the predesignated area to a first distance sensor;
determining the distance from the first distance sensor to the object with at least one distance sensor selected from the group consisting of camera, laser, lidar, sonar and stereoscopy devices;
providing virtual reality images;
providing the user with a head mounted display adapted to have a perspective during use corresponding to a perspective of the user's eyes along a predetermined line of sight;
providing a first depth sensor, the first depth sensor being a video camera and mounting the video camera at a location from which the video camera has a perspective substantially similar to the perspective of the user's eyes;
providing a second depth sensor and a third depth sensor;
mounting the second depth sensor and mounting the third depth sensor at various locations around the user;
adapting the video camera, the second depth sensor and the third depth sensor to provide data representative of the distance from the object to the video camera along the line of sight;
operating the video camera to provide real video images of the real-time physical environment;
capturing the real video images in digital pixels;
identifying areas of the real video images to be overlaid with virtual video images by comparing the pixel depth of the digital pixels to a predetermined target pixel depth;
making transparent all digital pixels of the real video images whose pixel depth exceeds the predetermined target pixel depth;
overlaying the real video images onto the virtual video images to form a combined image;
and,
providing to the head mounted display the combined image.

14 more elements

A computer-implemented system for combining a virtual reality environment and a physical environment for a user comprising:

a computer;
a camera operatively connected to the computer and adapted to provide to the computer real-time physical environment video images in digital pixels using a hue, saturation and brightness color coordinate system, and adapted to be mounted on the head of a user at a location from which the camera has a view substantially similar to the view of the user's eyes;
a physical object operatively connected to the computer and adapted to interact with the user and to provide input to the computer in response to interaction with the user;
a virtual image generator operatively connected to the computer and adapted to provide to the computer virtual video images in digital pixels;
a position detector operatively connected to the computer and adapted to be mounted on the head of the user and to provide to the computer three-dimensional, spatial information about the location and direction of the user's head;
an image display operatively connected to the computer and adapted to be mounted on the head of the user and adapted to receive video images from the computer;
a real-time physical environment;
a pre-determined target masking color covering a predesignated area of the real-time physical environment;
the computer programmed to recognize the target masking color in the hue, saturation and brightness color coordinate system;
a range of predetermined target threshold values of hue corresponding to the pre-determined target masking color hue;
a range of predetermined target threshold values of saturation corresponding to the pre-determined target masking color saturation;
a range of predetermined target threshold values of brightness corresponding to the pre-determined target masking color brightness;
the computer programmed to make transparent areas of the real-time physical environment video images in which the color of the pixels of the real-time physical environment video images fall within the predetermined target threshold values of hue, of saturation and of brightness of the pixels of the target masking color;
the computer adapted to change the virtual video images in response to the input to the computer from the physical object;
the computer programmed to overlay the real-time physical environment video images onto the virtual video images to form combined video images;
and,
the computer programmed to provide to the image display the combined video images.

14 more elements

Dependencies are inferred from the claim text.

Systems and methods for combining virtual and real-time physical environments

Life of this patent

Systems and methods for combining virtual and real-time physical environments

Life of this patent

Abstract

Claims20