# US 10,455,134 B2 · Temporal processes for aggregating multi dimensional data from discrete and distributed collectors to provide enhanced space-time perspective

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,455,134 B2
- Document ID: US10455134B2
- Status: Granted
- Kind: B2
- Application number: 15/521,871
- Filed: 2015-10-26
- Published: 2018-09-06
- Granted: 2019-10-22
- Earliest claimed: 2014-10-26
- Assignees: Galileo Group, Inc.
- Applicants: Galileo Group, Inc.
- Inventors: Donald Michael Barnes; James Michael Grichnik
- Web page: https://patentgrep.com/patent/US10455134B2

## CPC

- H04N 23/661: Transmitting camera control signals through networks, e.g. control via the Internet
- G01C 19/5776: Signal processing not specific to any of the devices covered by groups G01C19/5607 - G01C19/5719
- G06F 16/58: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F 16/587: using geographical or spatial information, e.g. location
- G06F 18/2137: based on criteria of topology preservation, e.g. multidimensional scaling or self-organising maps
- G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529)
- G06T 7/74: {involving reference images or patches}
- G06T 15/04: Texture mapping
- G06T 15/08: Volume rendering
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- H04N 23/61: based on recognised objects
- H04N 23/611: where the recognised objects include parts of the human body
- H04W 4/70: Services for machine-to-machine communication [M2M] or machine type communication [MTC]
- G06T 2200/04: involving 3D image data
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30196: Human being; Person
- G06T 2207/30241: Trajectory
- G06T 2207/30244: Camera pose
- G06T 2215/16: Using real world measurements to influence rendering
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

At a central controlling system, a composite map is constructed based on a previous dataset that includes (A) images of a target obtained by imaging devices at a first time, and (B) respective meta data associated with the images. Imaging devices collect image data during a second time after the first time. In accordance with the composite map, the imaging devices obtain images of the target and associate meta data with the images. The images and respective meta data are communicated to the central controlling system. The positions and orientations of the imaging devices when the respective images were obtained, and when the images were obtained, are used to index the images against the target, thereby aggregating multi-dimensional data for the target. Temporal information about a characteristic of the target over time is then extracted from the aggregated multi-dimensional data.

## Claims

### Claim 1

A computer-implemented method of aggregating multi-dimensional data for a target at a central controlling system, the method comprising:
constructing a composite map based on a previous dataset, wherein the previous dataset includes:
a plurality of two-dimensional pixilated images of the target obtained by one or more computer-enabled imaging devices during a first time, and
respective meta data associated with each of the plurality of two-dimensional pixilated images, wherein the respective meta data indicates (i) positional and orientation data that includes a position and orientation of a respective computer-enabled imaging device when the respective image is obtained, (ii) an indication of when the respective image is obtained, and an (iii) identity of the respective computer-enabled imaging device;
using each of the one or more computer-enabled imaging devices to collect respective image data of the target during a second time subsequent to the first time, by causing each respective computer-enabled imaging device of the one or more computer-enabled imaging devices to execute a method of data capture, including:
in accordance with the composite map based on the previous dataset, obtaining a respective two-dimensional pixilated image of the target;
associating meta data with the respective two-dimensional pixilated image, wherein the respective meta data indicates respective (i) positional and orientation data that includes a position and orientation of the respective computer-enabled imaging device when the respective image is obtained, (ii) an indication of when the respective image is obtained, and an (iii) identity of the respective computer-enabled imaging device; and
communicating to the central controlling system the respective two-dimensional pixilated image of the target and the respective meta data;
at the central controlling system, the central controlling system having one or more processors and memory for storing one or more programs for execution by the one or more processors, executing the method of:
receiving one or more respective two-dimensional pixilated images and associated meta data collected from the second time from each of the one or more computer-enabled imaging devices;
using the position and orientation of each of the respective computer-enabled imaging devices when the respective images were obtained, and the indication of when the images were obtained, to index the images against the target, thereby aggregating multi-dimensional data for the target;
extracting, from the aggregated multi-dimensional data for the target, temporal information about a characteristic of the target over time; and
identifying a deficiency in the composite map, wherein the obtaining, for each respective computer-enabled imaging device, of the respective two-dimensional pixilated image is responsive to and in accordance with the identified deficiency.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising identifying a two-dimensional feature of the target using the composite map, wherein the obtaining, for each respective computer-enabled imaging device, of the respective two-dimensional pixilated image is in accordance with the identified two-dimensional feature.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising identifying a three-dimensional feature of the target using the composite map, wherein the obtaining, for each respective computer-enabled imaging device, of the respective two-dimensional pixilated image is in accordance with the identified three-dimensional feature.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein obtaining, for each respective computer-enabled imaging device, the respective two-dimensional pixilated image in accordance with the composite map comprises:
obtaining the respective two-dimensional pixilated image in accordance with the positional and orientation data associated with the plurality of two-dimensional pixilated images of the previous dataset.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the position and orientation of the respective computer-enabled imaging device, when the respective two-dimensional pixilated image is obtained, are substantially aligned with the positional and orientation data associated with at least one of the plurality of two-dimensional pixilated images from the previous dataset.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein:
the image data collected by each of the one or more computer-enabled imaging devices comprises a plurality of two-dimensional pixilated images obtained during the second time; and
a position and orientation of a respective one of the one or more computer-enabled imaging devices, for each of the plurality of two-dimensional pixilated images obtained during the second time, are substantially aligned with the positional and orientation data associated with a respective one of the plurality of two-dimensional pixilated images from the previous dataset.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein using each respective computer-enabled imaging device to collect respective image data further comprises:
providing information indicating a proximity of a current position and orientation of a respective computer-enabled imaging device, to a respective position and orientation associated with at least one of the plurality of two-dimensional pixilated images from the previous dataset.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein providing information indicating the proximity comprises displaying, on the respective computer-enabled imaging device, visual information indicating the proximity so as to enable alignment of the respective position and orientation of the respective computer-enabled imaging device to match the position and orientation associated with the at least one of the plurality of two-dimensional pixilated images from the previous dataset.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, further comprising:
using the composite map, spatially registering the one or more respective two-dimensional pixilated images obtained during the second time.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein constructing the composite map comprises:
extracting two-dimensional features from each image of the previous dataset; and
using the two-dimensional features extracted from the previous dataset, creating a three-dimensional composite map comprising a plurality of three-dimensional points, wherein each respective three-dimensional point of the plurality of three-dimensional points is associated with one of a plurality of respective sets of the two-dimensional features extracted from the previous dataset,
each of the respective sets representing an appearance of an associated respective three-dimensional point across the previous dataset.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein the spatially registering comprises:
extracting two-dimensional features from each image obtained during the second time; and
matching a set of the two-dimensional features extracted from the images obtained during the second time, with one of the plurality of respective sets of the two-dimensional features extracted from the previous dataset,
thereby associating the set of the two-dimensional features extracted from the images obtained during the second time with a respective three-dimensional point, of the three-dimensional composite map, that is associated with the matched set of the two-dimensional features extracted from the previous dataset.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein constructing the composite map comprises:
matching a two-dimensional feature in a first image and in a second image of the plurality of two-dimensional pixilated images;
estimating a parallax between the first image and the second image using respective meta data associated with the first image and the second image; and
adding, when the parallax between the first image and the second image satisfies a parallax threshold, a two or three-dimensional point to the composite map at a distance obtained by triangulating the first image and the second image using respective meta data associated with the first image and the second image.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein:
the composite map based on the previous dataset is a first composite map, and
the image data collected during the second time by using each of the one or more computer-enabled imaging devices comprises a plurality of two-dimensional pixilated images of the target and the associated meta data,
the method further comprising:
constructing a second composite map based on the image data collected during the second time; and
comparing the second composite map with the first composite map to identify respective differences in one or more three-dimensional features from the first time to the second time.

### Claim 14 (depends on 13)

The computer-implemented method of claim 13, wherein the respective identified differences include a respective shape of the one or more three-dimensional features.

### Claim 15 (depends on 13)

The computer-implemented method of claim 13, wherein the respective identified differences include a respective volume of the one or more three-dimensional features.

### Claim 16 (depends on 13)

The computer-implemented method of claim 13, wherein the respective identified differences include a respective surface contour of the one or more three-dimensional features.

### Claim 17 (depends on 13)

The computer-implemented method of claim 13, wherein the respective identified differences include a respective smoothness or roughness of the one or more three-dimensional features.

### Claim 18 (depends on 13)

The computer-implemented method of claim 13, wherein the respective identified differences include a respective opacity of the one or more three-dimensional features.

### Claim 19 (depends on 13)

The computer-implemented method of claim 13, wherein the respective identified differences include a respective color gradient of the one or more three-dimensional features.