# US 12,008,800 B2 · Conflation of geospatial points of interest and ground-level imagery

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,800 B2
- Document ID: US12008800B2
- Status: Granted
- Kind: B2
- Application number: 18/383,633
- Filed: 2023-10-25
- Published: 2024-05-02
- Granted: 2024-06-11
- Earliest claimed: 2022-10-27
- Assignees: UT-Battelle, LLC
- Applicants: UT-Battelle, LLC
- Inventors: Debraj De; Rutuja Gurav; Junchuan Fan; Gautam Thakur
- Web page: https://patentgrep.com/patent/US12008800B2

## CPC

- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/29: {Graphical models, e.g. Bayesian networks}
- G06V 10/426: Graphical representations
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06V 10/84: using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
- G06V 20/50: Context or environment of the image
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A prediction system harvests geo-tagged ground-level images through one or more algorithms. The system receives point of interest data representing structures or events and tags the geo-tagged ground-level images with a probability describing a classification. The system tags point of interest data with a hierarchical genre classification and encodes the tagged geo-tagged ground-level images as vectors to form nodes and edges in a proximity graph. The system encodes tagged points of interest data as similarity vectors to render more nodes and more edges on the proximity graph associated with the tagged geo-tagged ground-level images nodes by calculated semantic distances. The system splits the proximity graph into a training subgraph and a testing subgraph and trains a neural network by aggregating and sampling information from neighboring nodes within the training subgraph graph and validates through the testing subgraph. Training ends when a loss measurement is below a threshold.

## Claims

### Claim 1

A non-transitory machine-readable medium encoded with machine-executable instructions, wherein execution of the machine-executable instructions is for:
harvesting geo-tagged ground-level images through a scene detection algorithm executed by an image classifier;
receiving point of interest data representing structures or events through a transceiver;
tagging each of the geo-tagged ground-level images with a probability object describing a tag classification;
tagging each of the point of interest data with a hierarchical level genre classification object;
encoding tagged geo-tagged ground-level images as hot encoded vectors to render a first plurality of nodes and a first edge on a proximity graph;
encoding tagged point of interest data as a similarity vector to render a second plurality of nodes and a second edge on the proximity graph associated with the first node by a calculated semantic distance measurement;
splitting the proximity graph into a training subgraph and a testing subgraph;
initiating a training session of a neural network by processing the training subgraph during a training session that includes aggregating information at a selected node within the training subgraph by sampling information from a neighboring node within the training subgraph;
terminating the training session when a binary cross-entropy loss measurement based on the training subgraph falls below a predetermined threshold; and
initiating an evaluation of the neural network by processing the testing subgraph.

### Claim 2 (depends on 1)

The non-transitory machine-readable medium of claim 1, where the calculated semantic distance measurement comprises a calculated Euclidean distance measurement.

### Claim 3 (depends on 1)

The non-transitory machine-readable medium of claim 1, where the proximity graph is constructed from a plurality of dot products.

### Claim 4 (depends on 1)

The non-transitory machine-readable medium of claim 1, where the binary cross-entropy loss measurement comprises an average loss measurement based on a predicted edge value and a calculated edge value.

### Claim 5 (depends on 4)

The non-transitory machine-readable medium of claim 4, where the binary cross-entropy loss measurement comprises:
-
∑
n
i
,
n
j
y
n
i
,
n
j
⁢
log
⁢
(
y
ˆ
n
i
,
n
j
)
+
(
1
-
y
n
i
,
n
j
)
⁢
log
⁡
(
1
-
y
ˆ
n
i
,
n
j
)
where ŷni,nj represents a calculated edge value for a node pair ni, nj and ŷni,nj represents a predicted edge value for the node pair input for the node pair ni,nj.

### Claim 6 (depends on 1)

The non-transitory machine-readable medium of claim 1, where the neural network comprises a scalable graph neural network.

### Claim 7

A process comprising:
harvesting geo-tagged ground-level images through a scene scene detection algorithm executed by an image classifier;
receiving point of interest data representing structures or events through a transceiver;
tagging each of the geo-tagged ground-level images with a probability object describing a tag classification;
tagging each of the point of interest data with a hierarchical level genre classification object;
encoding tagged geo-tagged ground-level images as encoded vectors to render a first plurality of nodes and a first edge on a proximity graph;
encoding tagged point of interest data as a similarity vector to render a second plurality of nodes and a second edge on the proximity graph associated with the first node by a calculated semantic distance measurement;
splitting the proximity graph into a training subgraph and a testing subgraph;
initiating a training session of a neural network by processing the training subgraph during a training session that includes aggregating information at a selected node within the training subgraph by sampling information from a neighboring node within the training subgraph; and
terminating the training session when a binary cross-entropy loss measurement based on the training subgraph falls below a predetermined threshold.

### Claim 8 (depends on 7)

The process of claim 7, where the calculated semantic distance measurement comprises a calculated Euclidean semantic distance measurement.

### Claim 9 (depends on 7)

The process of claim 7, further comprising initiating an evaluation session of the neural network by processing the neural network processing the testing subgraph.

### Claim 10 (depends on 7)

The process of claim 7, where the proximity graph is constructed based on a dot product.

### Claim 11 (depends on 7)

The process of claim 7, where the binary cross-entropy loss measurement comprises an average loss measurement based on a predicted edge value and a calculated edge value.

### Claim 12 (depends on 7)

The process of claim 7, where the neural network comprises a scalable graph neural network.

### Claim 13

An edge prediction system comprising:
an image classifier that harvests geo-tagged ground-level images through one or more scene scene detection algorithms;
a transceiver that receives point of interest data representing structures or events through a transceiver;
a processor that tags each of the geo-tagged ground-level images with a probability object describing a tag classification and tags each of the point of interest data with one of a plurality of hierarchical level genre classification objects;
an encoder that encodes tagged geo-tagged ground-level images as hot encoded vectors to render a first plurality of nodes and a first edge on a proximity graph, and encodes tagged point of interest data as a similarity vector to render a second plurality of nodes and a second edge on the proximity graph associated with the first plurality of nodes by a calculated semantic distance measurement;
a partitioning engine that splits the proximity graph into a training subgraph and a testing subgraph;
a training engine that processes the training subgraph through a neural network during a training session that includes aggregating information at a plurality of selected nodes within the training subgraph by sampling information from one more neighboring node within the training subgraph, and terminates the training session when a binary cross-entropy loss measurement based on the training subgraph falls below a predetermined threshold; and
a testing engine that evaluates the neural network by processing the testing subgraph.

### Claim 14 (depends on 13)

The system of claim 13, where the calculated semantic distance measurement comprises a calculated Euclidean semantic distance measurement.

### Claim 15 (depends on 13)

The system of claim 13, where the binary cross-entropy loss measurement comprises an average loss measurement based on a plurality of predicted edge values and a plurality of calculated edge values.

### Claim 16 (depends on 15)

The system of claim 15, where the binary cross-entropy loss measurement is expressed by:
-
∑
n
i
,
n
j
y
n
i
,
n
j
⁢
log
⁢
(
y
ˆ
n
i
,
n
j
)
+
(
1
-
y
n
i
,
n
j
)
⁢
log
⁡
(
1
-
y
ˆ
n
i
,
n
j
)
where yni,nj represents a calculated edge value for a node pair ni, nj and ŷni,nj represents a predicted edge value for the node pair input for the node pair ni,nj.

### Claim 17 (depends on 13)

The system of claim 13, where the proximity graph is constructed based on a dot product of the first plurality of nodes and the second plurality of nodes.