# US 11,443,444 B2 · Interior photographic documentation of architectural and industrial environments using 360 panoramic videos

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,444 B2
- Document ID: US11443444B2
- Status: Granted
- Kind: B2
- Application number: 17/304,534
- Filed: 2021-06-22
- Published: 2021-11-04
- Granted: 2022-09-13
- Earliest claimed: 2019-12-13
- Assignees: Reconstruct Inc.
- Applicants: Reconstruct Inc.
- Inventors: Derek Hoiem; Shengze Wang; Abhishek Bhatia
- Web page: https://patentgrep.com/patent/US11443444B2

## CPC

- G06T 7/33: using feature-based methods
- G06T 7/579: from motion
- G06T 3/06: Topological mapping of higher dimensional structures onto lower dimensional surfaces
- G06T 3/14: Transformations for image registration, e.g. adjusting or mapping for alignment of images
- G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 3/60: Rotation of whole images or parts thereof
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/73: using feature-based methods
- G06T 11/23: using straight lines or curves
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30108: Industrial image inspection
- G06T 2207/30204: Marker
- G06T 2210/04: Architectural design, interior design
- G06T 2210/56: Particle system, point based geometry or rendering
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06T 2219/2016: Rotation, translation, scaling
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A 360 panoramic video sequence of an architectural or industrial site includes a plurality of 360 images. 3D poses of the 360 images are determined with respect to one another, and a subset of the images extracted for further processing according to selection criteria. The 3D poses of the extracted images is refined based on determined correspondences of the features in the images, and a 3D point cloud of the site is developed from the extracted images with refined pose estimates. An “as-built” representation of the site captured in the panoramic video sequence is created from the sparse 3D point cloud and aligned to a 2D or 3D plan of the site. Once so aligned, the extracted images may be presented to a user in conjunction with their plan positions (e.g., overlayed on the site plan). Optionally, a point cloud or mesh view of the site may also be returned.

## Claims

### Claim 1

A method of producing a photographic record of a site, comprising extracting a plurality of 360 images from a 360 panoramic video sequence, determining three-dimensional (“3D”) poses of the extracted plurality of 360 images relative to each other and to a plan of the site, and producing a model of the site from the extracted plurality of 360 images, said model aligned to a plan of the site, wherein:
the plan is one of a two-dimensional (“2D”) drawing or a 3D computer aided design (“CAD”) or building information model (“BIM”) of the site;
the 3D poses of the extracted plurality of 360 images are determined by first determining an approximate pose based on features sequentially tracked between sequential images of the 360 panoramic video sequence and then refined based on features matched across a subset of images selected from the 360 panoramic video sequence; and
if the plan of the site is a 2D drawing, alignment to the plan of the site is obtained by determining correspondences between lines in the 2D drawing and lines derived from a sparse 3D point cloud produced from the subset of images selected from the 360 panoramic video sequence, said derivation performed by projecting 3D planes from the 3D point cloud determined to be perpendicular to a ground plane into lines on the ground plane, otherwise, if the plan of the site is a 3D CAD or BIM, alignment to the plan of the site is obtained by determining correspondences between points or planes derived from the sparse 3D point cloud and planes derived from the 3D CAD or BIM using ground plane projections of said points or planes derived from the 3D point cloud and ground plane projections of said planes derived from the 3D CAD or BIM.

### Claim 2 (depends on 1)

The method of claim 1, wherein the subset of images selected from the 360 panoramic video sequence is a subset selected based on intensity contrast within each image of the 360 panoramic video sequence or differences between images the 360 panoramic video sequence in timestamps or spatial positions.

### Claim 3 (depends on 2)

The method of claim 2, wherein positions and orientations of images within the subset of images selected from the 360 panoramic video sequence with respect to the plan of the site are obtained automatically.

### Claim 4 (depends on 1)

The method of claim 1, further comprising obtaining an as-built drawing of the site automatically from the sparse 3D point cloud.

### Claim 5 (depends on 4)

The method of claim 4, further comprising automatically aligning the as-built drawing to the 2D or 3D CAD or BIM.

### Claim 6 (depends on 5)

The method of claim 5, wherein positions and orientations of the images selected from the 360 panoramic video sequence with respect to the as-built drawing are obtained automatically.

### Claim 7 (depends on 1)

The method of claim 1, further comprising performing a multi-view stereo process on ones of the extracted plurality of 360 images to produce a dense 3D point cloud model of the site and obtaining an as-built drawing of the site automatically from the dense 3D point cloud.

### Claim 8

A computer system comprising a processor coupled to a storage device, the storage device storing processor-implemented instructions for producing a photographic record of a site, said instructions for causing said processor to perform steps including:
extract a plurality of 360 images from a 360 panoramic video sequence, determine three-dimensional (“3D”) poses of the extracted plurality of 360 images relative to each other and to a plan of the site, and produce a model of the site from the extracted plurality of 360 images, said model aligned to a plan of the site, wherein:
the 3D poses of the extracted plurality of 360 images are determined by first determining an approximate pose based on features sequentially tracked between sequential images of the 360 panoramic video sequence and then refined based on features matched across a subset of images selected from the 360 panoramic video sequence; and
if the plan of the site is a two-dimensional (“2D”) drawing, alignment to the plan of the site is obtained by determining correspondences between lines in the 2D drawing and lines derived from a sparse 3D point cloud produced from the subset of images selected from the 360 panoramic video sequence, said derivation performed by projecting 3D planes from the 3D point cloud determined to be perpendicular to a ground plane into lines on the ground plane, otherwise, if the plan of the site is a 3D computer aided design (“CAD”) or building information model (“BIM”), alignment to the plan of the site is obtained by determining correspondences between points or planes derived from the sparse 3D point cloud and planes derived from the 3D CAD or BIM using ground plane projections of said points or planes derived from the 3D point cloud and ground plane projections of said planes derived from the 3D CAD or BIM.

### Claim 9 (depends on 8)

The computer system of claim 8, wherein the subset of images selected from the 360 panoramic video sequence is a subset selected based on intensity contrast within each image of the 360 panoramic video sequence or differences between images the 360 panoramic video sequence in timestamps or spatial positions.

### Claim 10 (depends on 9)

The computer system of claim 9, wherein positions and orientations of images within the subset of images selected from the 360 panoramic video sequence with respect to the plan of the site are obtained automatically.

### Claim 11 (depends on 8)

The computer system of claim 8, wherein said instructions further include instructions for causing said processor to obtain an as-built drawing of the site automatically from the sparse 3D point cloud.

### Claim 12 (depends on 11)

The computer system of claim 11, wherein said instructions further include instructions for causing said processor to automatically align the as-built drawing to the 2D or 3D CAD or BIM.

### Claim 13 (depends on 11)

The computer system of claim 11, wherein said instructions further include instructions for causing said processor to automatically obtain positions and orientations of the images selected from the 360 panoramic video sequence with respect to the as-built drawing.

### Claim 14 (depends on 8)

The computer system of claim 8, wherein said instructions further include instructions for causing said processor to perform a multi-view stereo process on ones of the extracted plurality of 360 images to produce a dense 3D point cloud model of the site and obtain an as-built drawing of the site automatically from the dense 3D point cloud.

### Claim 15

A system, comprising a camera for producing a 360 video and a computer system coupled to receive said 360 video from said camera, said computer system including a processor and a storage device, the storage device storing processor-implemented instructions for producing a photographic record of a site from said 360 video, said instructions for causing said processor to perform steps including:
extract a plurality of 360 images from the video, determine three-dimensional (“3D”) poses of the extracted plurality of 360 images relative to each other and to a plan of the site, and produce a model of the site from the extracted plurality of 360 images, said model aligned to a plan of the site, wherein:
the 3D poses of the extracted plurality of 360 images are determined by first determining an approximate pose based on features sequentially tracked between sequential images of the 360 panoramic video sequence and then refined based on features matched across a subset of images selected from the 360 panoramic video sequence; and
if the plan of the site is a two-dimensional (“2D”) drawing, alignment to the plan of the site is obtained by determining correspondences between lines in the 2D drawing and lines derived from a sparse 3D point cloud produced from the subset of images selected from the 360 panoramic video sequence, said derivation performed by projecting 3D planes from the 3D point cloud determined to be perpendicular to a ground plane into lines on the ground plane, otherwise, if the plan of the site is a 3D computer aided design (“CAD”) or building information model (“BIM”), alignment to the plan of the site is obtained by determining correspondences between points or planes derived from the sparse 3D point cloud and planes derived from the 3D CAD or BIM using ground plane projections of said points or planes derived from the 3D point cloud and ground plane projections of said planes derived from the 3D CAD or BIM.