# US 2019/0244372 A1 · System and method for infinite smoothing of image sequences

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244372 A1
- Document ID: US20190244372A1
- Status: Published application
- Kind: A1
- Application number: 16/384,578
- Filed: 2019-04-15
- Published: 2019-08-08
- Earliest claimed: 2014-10-31
- Assignees: Fyusion, Inc.
- Applicants: Fyusion, Inc.
- Inventors: Stefan Johannes Josef Holzer; Stephen David Miller; Radu Bogdan Rusu; Yuheng Ren
- Web page: https://patentgrep.com/patent/US20190244372A1

## CPC

- G06T 7/337: {involving reference images or patches}
- G06F 3/005: {Input arrangements through a video camera}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 16/532: Query formulation, e.g. graphical querying
- G06F 16/5838: {using colour}
- G06F 16/738: Presentation of query results
- G06F 16/783: using metadata automatically derived from the content
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 15/205: {Image-based rendering}
- H04N 5/265: Mixing
- H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06T 2207/10016: Video; Image sequence
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Various embodiments of the present invention relate generally to systems and processes for interpolating images of an object. According to particular embodiments, a sequence of images is obtained using a camera which captures the sequence of images along a camera translation. Each image contains at least a portion of overlapping subject matter. A plurality of keypoints is identified on a first image of the sequence of images. Each keypoint from the first image are kept track of to a second image. Using a predetermined algorithm, a plurality of transformations are computed using two randomly sampled keypoint correspondences, each of which includes a keypoint on the first image and a corresponding keypoint on the second image. An optimal subset of transformations is determined from the plurality of transformations based on predetermined criteria, and transformation parameters corresponding to the optimal subset of transformations is calculated and stored for on-the-fly interpolation.

## Claims

### Claim 1

A method for interpolating images of an object, the method including:
obtaining a sequence of images using a camera, the sequence of images being captured along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including an object;
identifying a plurality of keypoints on a first image of the sequence of images;
keeping track of each keypoint from the first image to a second image;
computing, using a predetermined algorithm, a plurality of transformations, wherein each transformation is computed using two randomly sampled keypoint correspondences, wherein each keypoint correspondence includes a keypoint on the first image and a corresponding keypoint on the second image;
determining an optimal subset of transformations from the plurality of transformations based on predetermined criteria, wherein determining the optimal subset of transformations includes selecting an optimal transformation candidate for each pixel of the first image, wherein selecting an optimal transformation candidate for each pixel is processed as a separate thread of a graphics processing unit (GPU) such that the processing time for each pixel is bounded by O(1) rather than O(n), wherein the plurality of transformations includes a subset of transformation candidates, wherein determining the optimal subset of transformations includes applying each transformation candidate in the subset of transformation candidates to each pixel in the first image, thereby generating a plurality of first interpolated pixel candidates, and to a corresponding pixel in the second image, thereby generating a plurality of second interpolated pixel candidates, and then comparing the plurality of first interpolated pixel candidates to the corresponding plurality of second interpolated pixel candidates to select an optimal transformation candidate having the smallest corresponding pixel candidate difference;
calculating transformation parameters corresponding to the optimal subset of transformations; and
storing the transformation parameters for on-the-fly interpolation.
