US 9,996,945 B1Granted

Live augmented reality guides

AssigneeFyusion, Inc.

InventorsStefan Johannes Josef Holzer, Alexander Jay Bruen Trevor, Michelle Jung-Ah Ho, David Klein, Stephen David Miller, Shuichi Tsutsumi, Radu Bogdan Rusu

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Various embodiments of the present invention relate generally to systems and methods for analyzing and manipulating images and video. According to particular embodiments, the spatial relationship between multiple images and video is analyzed together with location information data, for purposes of creating a representation referred to herein as a surround view for presentation on a device. A visual guide can provided for capturing the multiple images used in the surround view. The visual guide can be a synthetic object that is rendered in real-time into the images output to a display of an image capture device. The visual guide can help user keep the image capture device moving along a desired trajectory.

Claims21

Dependent claims

A method comprising:

receiving live images from a camera on a mobile device comprising 2-D pixel data wherein the live images are output to a display on the mobile device and show what is currently being captured by the camera;
capturing a plurality of images from among the live images to generate a multi-view interactive digital media representation of an object appearing in the live images wherein a position and/or orientation of the camera varies during the capturing of the plurality of images;
while the live images are currently being output to the display on the mobile device, receiving, via a touchscreen over the display, a selection of a location on the touchscreen wherein the location is over the object;
based upon the location on the touchscreen, determining a first position of a tracking point in the 2-D pixel data from a first image among the live images;
receiving sensor data indicating the position and/or orientation of the camera;
as additional live images after the first image are captured by the camera and output to the display, determining a current position of the tracking point in each of the additional live images on an image by image basis;
based upon the current position of the tracking point and the sensor data, rendering a virtual object, separate from and addition to the object, into each of the additional live images output to the display to generate a plurality of synthetic images wherein in each of the synthetic images the virtual object is positioned relative to the current position of the tracking point determined for each of the additional live images, wherein the virtual object provides information related to i) a progress of the camera along a path, ii) a current position and/or current orientation of the camera relative to the path during the capture of the plurality of images used to generate the multi-view interactive digital media representation of the object;
and
outputting the plurality of synthetic images to the display wherein each of the synthetic images show the object as currently being captured by the camera and the virtual object.

6 more elements

A method comprising:

receiving live images from a camera on a mobile device comprising 2-D pixel data wherein the live images are output to a display on the mobile device and show what is currently being captured by the camera;
capturing a plurality of images from among the live images to generate a multi-view interactive digital media representation of an object appearing in the live images wherein a position and/or orientation of the camera varies during the capturing of the plurality of images;
while the live images are currently being output to the display on the mobile device, receiving a selection of a location wherein the location is over the object in 2-D pixel data from a first image from among the live images;
based upon the location, determining a first position of a tracking point in the 2-D pixel data from the first image among the live images;
receiving sensor data indicating the position and/or orientation of the camera;
as additional live images after the first image are captured by the camera and output to the display, determining a current position of the tracking point in each of the additional live images on an image by image basis;
based upon the current position of the tracking point and the sensor data, rendering a virtual object, separate from and addition to the object, into each of the additional live images output to the display to generate a plurality of synthetic images wherein in each of the synthetic images the virtual object is positioned relative to the current position of the tracking point determined for each of the additional live images, wherein the virtual object provides information related to i) a progress of the camera along a path, ii) a current position and/or current orientation of the camera relative to the path during the capture of the plurality of images used to generate the multi-view interactive digital media representation of the object;
and
outputting the plurality of synthetic images to the display wherein each of the synthetic images show the object as currently being captured by the camera and the virtual object.

6 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/376,537Filed 12 Dec 2016
Published: US 2018/0165827 A114 Jun 2018
Granted: 12 Jun 2018
Term ends: Not estimatedThe source records don’t support an estimate.

Classification14 codes

G06T 19/006
Manipulating three-dimensional [3D] models or images for computer graphics: mixed reality
G06T 7/33
Determination of transform parameters for the alignment of images, i.e. image registration: using feature-based methods
G06V 10/17
Image acquisition: using hand-held instruments
G06V 20/20
Scenes; Scene-specific elements: in augmented reality scenes