US 11,455,739 B2Granted

Method for augmenting a scene in real space with projected visual content

AssigneeLightform, Inc.

InventorsKevin Karsch, Rajinder Sodhi, Brett Jones, Pulkit Budhiraja, Phil Reyneri, Douglas Rieck, Andrew Kilkenny, Ehsan Noursalehi, Derek Nedelman, Laura LaPerche, Brittany Factura

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

One variation of method includes: serving setup frames to a projector facing a scene; at a peripheral control module comprising a camera facing the scene, recording a set of images during projection of corresponding setup frames onto the scene by the projector and a baseline image depicting the scene in the field of view of the camera; calculating a pixel correspondence map based on the set of images and the setup frames; transforming the baseline image into a corrected color image—depicting the scene in the field of view of the camera—based on the pixel correspondence map; linking visual assets to discrete regions in the corrected color image; generating augmented reality frames depicting the visual assets aligned with these discrete regions; and serving the augmented reality frames to the projector to cast depictions of the visual assets onto surfaces, in the scene, corresponding to these discrete regions.

Claims19

Dependent claims

A method for augmenting a scene in real space with projected content comprising:

serving a sequence of setup frames to a light projector facing the scene;
at a camera facing the scene:
capturing a set of scan images of the scene, each scan image in the set of scan images captured during projection of a corresponding setup frame, in the sequence of setup frames, onto the scene by the light projector;
and
capturing a baseline image depicting the scene in a field of view of the camera;
calculating a pixel correspondence map based on pixel values in scan images in the set of scan images and pixel values in corresponding setup frames in the sequence of setup frames;
deriving a three-dimensional representation of the scene based on the pixel correspondence map;
transforming the baseline image into a corrected color image based on the pixel correspondence map, the corrected color image depicting the scene from a perspective of the light projector;
detecting a color gradient in the corrected color image;
distinguishing a first region of the scene from a second region of the scene based on the color gradient;
and
accessing a first visual asset assigned to the first region of the scene;
aligning the first visual asset to the color gradient;
warping the first visual asset according to a profile of the first region of the scene characterized in the three-dimensional representation of the scene;
and
projecting the first visual asset, warped according to the profile, onto the first region of the scene via the light projector.

13 more elements

A method for augmenting a scene in real space with projected content comprising:

serving a sequence of setup frames to a light projector facing the scene;
at a camera facing the scene, capturing a set of scan images of the scene, each scan image in the set of scan images captured during projection of a corresponding setup frame, in the sequence of setup frames, onto the scene by the light projector;
calculating a pixel correspondence map based on pixel values in scan images in the set of scan images and pixel values in corresponding setup frames in the sequence of setup frames;
deriving a three-dimensional representation of the scene based on the pixel correspondence map;
identifying a first region of the scene in the three-dimensional representation of the scene;
identifying a second region of the scene, distinct from the first region of the scene, in the three-dimensional representation of the scene;
assigning a first content application to the first region of the scene;
assigning a second content application to the second region of the scene;
warping a first visual asset, output by the first content application, according to a first profile of the first region of the scene characterized in the three-dimensional representation of the scene;
warping a second visual asset, output by the second content application, according to a second profile of the first region of the scene characterized in the three-dimensional representation of the scene;
during a first time period, projecting the first visual asset, warped according to the first profile, onto the first region of the scene via the light projector;
and
during a second time period succeeding the first time period, projecting the second visual asset, warped according to the second profile, onto the first region of the scene via the light projector in response to detecting a trigger event.

10 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/872,237Filed 11 May 2020 · continuation
Priority: 28 Sep 2018
Published: US 2020/0105006 A124 Dec 2020
Granted: 27 Sep 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification10 codes

G01B 11/2536
By projecting a pattern, e.g. one or more lines, moiré fringes on the object: using several gratings with variable grating pitch, projected on the…
G01B 11/2513
By projecting a pattern, e.g. one or more lines, moiré fringes on the object: with several lines being projected in more than one direction, e.g.…
G06T 7/521
Depth or shape recovery: from laser ranging, e.g. using interferometry; from the projection of structured light
G06T 7/536
Depth or shape recovery: from perspective effects, e.g. by using vanishing points

Family

Application16/146,679Parent of this continuation · filed 28 Sep 2018
Application16/447,792Parent of this continuation · filed 20 Jun 2019
US 10,373,325 B1Parent of this continuation · filed 6 Aug 2019
US 2020/0402247 A1Pre-grant publication · published 24 Dec 2020
US 10,692,233 B2Parent of this continuation
US 2020/0105006 A1Published application
US 11,455,739 B2This patent

From the US filing history. Foreign family members aren’t listed.