Back to results
US RE50,637 E1Granted
Augmented reality using projector-camera enabled devices
AssigneeThe Regents of the University of California
InventorsAditi Majumder, Behzad Sajadi
Life of this patent
Estimating the term…
Abstract
An augmented reality scene may be registered onto an arbitrary surface. A camera may capture an image of the arbitrary surface. The camera may analyze the surface geometry of the arbitrary surface. In some embodiments, a processing computing device may analyze data captured by the camera and an adjacent camera to reconstruct the surface geometry of the arbitrary surface. A scene may be registered to a three dimensional coordinate system corresponding to the arbitrary surface. A projector may project the scene onto the arbitrary surface according to the registration so that the scene may not display as being distorted.
Claims54
A system for projecting an image of a scene onto an arbitrary surface that is non-planar in both horizontal and vertical directions, the system comprising:
- a first un-calibrated camera coupled to un-calibrated in relative camera calibration parameters:
- a first projector, the first un-calibrated camera being disposed to capture at least a part of at least a first portion of the image projected onto the arbitrary surface by the first projector, wherein the first portion of the image includes a first point;
- an adjacent un-calibrated a second camera coupled to un-calibrated in relative camera calibration parameters;
- a second projector, wherein the adjacent un-calibrated second camera is disposed to capture at least a part of at least a second portion of the image projected onto the arbitrary surface by the second projector, wherein the second portion of the image includes the first point;
- wherein the first camera and the second camera are interchangeably coupled and uncoupled from the first projector and the second projector such that any camera can be coupled to any projector;
- and
- a one or more processing computing device coupled to the first camera configured to coupled to a non-transitory computer readably storage medium that includes computer readable code that, when executed by the one or more processing computing devices, causes the one or more processing computing devices to:
- perform configuration identification of the projectors and cameras based on an adjacency graph representation in context of overlap in their display space or sensing space,
- recognize reconstruct an underlying surface geometry of the arbitrary surface by analyzing data in the at least the part of at least the first portion of the image captured by the first camera and, the at least the part of at least the second portion of the image captured by the adjacent second camera, or a combination thereof,
- register an overlapping portion of the image of the scene, that is projected by the first projector and projected by the second projector, within a three dimensional coordinate system for display on the arbitrary surface based on an analysis of the data in the at least the part of at least the first portion of the image captured by the first camera, the at least the part of at least the second portion of the image captured by the second camera, or a combination thereof, and
- control the at least one of the first projector and the second projector to display the overlapping portion of the image of the registered scene onto the arbitrary surface based on an analysis of the data in the at least first portion of the image captured by the first camera and the at least second portion of the image captured by the adjacent camera.
8 more elements
A method for projecting an image of a scene onto an arbitrary surface that is non-planar in both horizontal and vertical directions, the method comprising:
- performing configuration identification of projectors and cameras based on an adjacency graph representation in context of overlap in their display space or sensing space, wherein the cameras are interchangeably coupled and uncoupled from the at least two projectors such that any camera can be coupled to any projector;
- capturing, from at least one un-calibrated camera un-calibrated in relative camera calibration parameters, a set of images of the arbitrary surface while patterns are projected on the arbitrary surface by at least one of at least two projectors, wherein the arbitrary surface is horizontally and vertically irregular, and wherein at least a first portion of a first image projected by a first projector of the at least two projectors overlaps on the arbitrary surface with at least a second portion of a second image projected by a second projector of the at least two projectors;
- recognizing reconstructing a surface geometry of the arbitrary surface, based on the patterns projected on the arbitrary surface, using multiview stereo reconstruction to determine three dimensional locations of data points within the set of images;
- and
- registering the first portion of the first image and the second portion of the second image to project the image of the scene corresponding onto the surface geometry of the arbitrary surface.
2 more elements
A non-transitory computer readable storage medium comprising:
- a computer readable code, which when executed by a processing computing device, causes the processing computing device to:
- perform configuration identification of projectors and cameras based on an adjacency graph representation in context of overlap in their display space or sensing space, wherein a first camera and a second camera are interchangeably coupled and uncoupled from a first projector and a second projector such that any camera can be coupled to any projector;
- capture an image a scene of an arbitrary surface that is non-planar in both horizontal and vertical directions, wherein at least one point within the captured image scene is:
- (i) within a field of view of an un-calibrated a camera un-calibrated in relative camera calibration parameters, within a first image projected by a first projector and within a second image projected by a second projector, or (ii) within an image projected by a projector and, within a first field of view of image captured by a first un-calibrated camera un-calibrated in relative camera calibration parameters and within a second field of view of image captured by a second un-calibrated camera un-calibrated in relative camera calibration parameters;
- analyze a surface geometry of the arbitrary surface, from the image scene of the arbitrary surface, for surface height variations;
- and
- register on the arbitrary surface (a) a first portion of the first image, that is projected by the first projector, with a second portion of the second image, that is projected by the second projector, corresponding to the surface height variations of the arbitrary surface or (b) register a first portion of the first image captured by the first camera with a second portion of the second image captured by the second camera such that first height variations in the first portion of the first image and second height variations in the second portion of the second image are indiscernible from the surface height variations of the arbitrary surface on which the first portion and the second portion are protected projected or captured.
4 more elements
A projection system, comprising:
- a first projector configured to project a first image of a scene onto an arbitrary surface that is non-planar in both horizontal and vertical directions;
- a second projector configured to project a second image of the scene onto the arbitrary surface that is non-planar in both horizontal and vertical directions;
- a first camera, un-calibrated in relative camera calibration parameters, said first camera configured to capture at least a part of the first image, wherein the part of the first image includes a first point;
- a second camera, un-calibrated in relative camera calibration parameters, said second camera configured to capture at least a part of the second image, wherein the part of the second image includes the first point, wherein the first camera and the second camera are interchangeably coupled and uncoupled from the first projector and the second projector such that any camera can be coupled to any projector;
- and
- one or more processing computing devices coupled to a non-transitory computer readable storage medium that includes computer readable code that, when executed by the one or more processing computing devices, causes the one or more processing computing devices to:
- perform configuration identification of the projectors and cameras based on an adjacency graph representation in context of overlap in their display space or sensing space, and
- perform geometric correction of the projected images such that the projected images are geometrically registered onto the arbitrary surface.
5 more elements
A system for projecting an image of a scene onto an arbitrary surface that is non-planar in both horizontal and vertical directions, the system comprising:
- a first camera un-calibrated in relative camera calibration parameters, said first camera configured to capture at least a portion of a first image projected onto the arbitrary surface by a first projector, wherein the portion of the first image includes a first point;
- a second camera un-calibrated in relative camera calibration parameters, said second camera configured to capture at least a portion of a second image projected onto the arbitrary surface by a second projector, wherein the first camera and the second camera are interchangeably coupled and uncoupled from the first projector and the second projector such that any camera can be coupled to any projector, wherein the portion of the second image includes the first point;
- and
- one or more processing computing devices coupled to a non-transitory computer readable storage medium that includes computer readable code that, when executed by the one or more processing computing devices, causes the one or more processing computing devices to:
- perform configuration identification of the projectors and cameras based on an adjacency graph representation in context of overlap in their display space or sensing space;
- and
- control at least one of the first projector and the second projector such that an overlapping portion of the first and the second images projected by the first projector and the second projector respectively is geometrically registered onto the arbitrary surface,
- wherein control of at least one of the first projector and the second projector is based on a reconstruction of an underlying surface geometry of the arbitrary surface from an analysis of the data in at least the portion of the first image captured by the first camera, at least the portion of the second image captured by the second camera, or a combination thereof and associating one or more points in the first and the second images to corresponding three dimensional coordinates along the surface geometry of the arbitrary surface.
5 more elements
Dependencies are inferred from the claim text.