# US 10,373,377 B1 · Augmented reality assisted deliveries

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,373,377 B1
- Document ID: US10373377B1
- Status: Granted
- Kind: B1
- Application number: 15/334,458
- Filed: 2016-10-26
- Granted: 2019-08-06
- Earliest claimed: 2016-10-26
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Robert Niewiadomski
- Web page: https://patentgrep.com/patent/US10373377B1

## CPC

- G06T 19/003: {Navigation within 3D models or images}
- H04W 4/024: Guidance services
- G01C 21/20: Instruments for performing navigational calculations (G01C21/24, G01C21/26 take precedence)
- G01C 21/3632: {Guidance using simplified or iconic instructions, e.g. using arrows (G01C21/365 takes precedence)}
- G01C 21/365: {Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself}
- G01C 21/3697: {Output of additional, non-guidance related information, e.g. low fuel level (G01C21/3679 takes precedence)}
- G02B 27/017: {Head mounted}
- G06Q 10/0631: Resource planning, allocation, distributing or scheduling for enterprises or organisations
- G06Q 10/083: Shipping
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04L 41/046: comprising network management agents or mobile agents therefor
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- H04L 67/52: specially adapted for the location of the user terminal
- H04W 4/029: Location-based management or tracking services
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G06T 2219/004: Annotating, labelling
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Disclosed are various embodiments for providing augmented reality assisted deliveries. Some embodiments include determining a current location of a client device and a next node in a delivery workflow based at least in part on the current location of the client device. Then, the client device renders the next node in the delivery workflow on an augmented reality display linked to the client device. Various embodiments can also include identifying a plurality of delivery workflows that include a destination node that corresponds to a delivery and selecting select a delivery workflow from the plurality of delivery workflows based at least in part on a delivery constraint. In these embodiments, a delivery route to a routing node is generated, wherein the routing node comprises an entry to the selected delivery workflow. The delivery route and the delivery workflow are then sent to an augmented reality client.

## Claims

### Claim 1

A system, comprising:
a computing device comprising a processor and a memory; and
an application comprising machine readable instructions stored in the memory that, when executed by the processor, cause the computing device to at least:
receive an annotation from a first augmented reality client device for a node in a delivery workflow;
add the annotation to the node in the delivery workflow;
identify a plurality of delivery workflows that include the node;
select the delivery workflow from the plurality of delivery workflows based at least in part the delivery workflow having a routing node that is compatible with a vehicle constraint associated with a delivery, the routing node comprising a navigational waypoint on a delivery route to a destination node;
add the delivery workflow to the delivery route; and
send the delivery route to a second augmented reality client device, wherein the second augmented reality client device is configured to:
render a three-dimensional model of an area associated with the node in the delivery workflow on an augmented reality display associated with the second augmented reality client device, wherein the three-dimensional model comprises a marker indicating a physical location corresponding to the node in the delivery workflow;
render the annotation in association with the node on the augmented reality display associated with the second augmented reality client device; and
wherein the augmented reality display is configured to be worn by an operator of the second augmented reality client device.

### Claim 2 (depends on 1)

The system of claim 1, wherein the node represents a parking location.

### Claim 3 (depends on 1)

The system of claim 1, wherein the node represents a delivery location.

### Claim 4 (depends on 1)

The system of claim 1, wherein the annotation comprises a video captured by the first augmented reality client device as a delivery agent moved from a previous node to the node, and the video is stored in association with the node.

### Claim 5

A system, comprising:
a computing device comprising a processor and a memory; and
an application comprising machine readable instructions stored in the memory that, when executed by the processor, cause the computing device to at least:
identify a plurality of delivery workflows that include a destination node that corresponds to a delivery;
select a delivery workflow from the plurality of delivery workflows based at least in part on a delivery constraint and the delivery workflow comprising at least one node that is compatible with a vehicle restriction associated with a vehicle being used for the delivery, the at least one node comprising a navigational routing waypoint on a delivery route to the destination node;
generate at least a portion of the delivery route to a routing node, wherein the routing node comprises an entry to the selected delivery workflow; and
send the delivery route and the selected delivery workflow to an augmented reality client, wherein at least a portion of the selected delivery workflow is rendered using a three-dimensional model that comprises a mark of a physical location of the routing node.

### Claim 6 (depends on 5)

The system of claim 5, wherein the machine readable instructions that cause the computing device to at least identify the plurality of delivery workflows further comprise machine readable instructions stored in the memory that, when executed by the processor, cause the computing device to at least:
match a first address specified in the destination node to a second address corresponding to the delivery; and
match an addressee specified in the destination node to a recipient corresponding to the delivery.

### Claim 7 (depends on 5)

The system of claim 5, wherein:
the selected delivery workflow comprises a first delivery workflow;
the destination node of the first delivery workflow comprises a first destination node; and
the application further comprises machine readable instructions stored in the memory that, when executed by the processor, cause the computing device to further:
identify a second delivery workflow that comprises the routing node of the first delivery workflow, the delivery constraint of the first delivery workflow, and a second destination node; and
add the second delivery workflow to the delivery route to the routing node.

### Claim 8 (depends on 5)

The system of claim 5, wherein the routing node represents a navigational waypoint.

### Claim 9 (depends on 5)

The system of claim 5, wherein the delivery constraint comprises a method of delivery.

### Claim 10 (depends on 5)

The system of claim 5, wherein:
the routing node is limited to a method of delivery; and
the machine readable instructions that cause the computing device to select a delivery workflow from the plurality of delivery workflows based at least in part on the delivery constraint further comprise machine readable instructions stored in the memory that, when executed by the processor, cause the computing device to at least determine that each routing node within the delivery workflow specifies the method of delivery as a valid method of delivery for the routing node.

### Claim 11 (depends on 5)

The system of claim 5, wherein the delivery constraint comprises a time of delivery.

### Claim 12 (depends on 5)

The system of claim 5, wherein the physical location is a first physical location, and the selected delivery workflow further comprises an access node that includes access instructions for a delivery agent to access a second physical location represented by the destination node.

### Claim 13 (depends on 5)

The system of claim 5, wherein the physical location is a first physical location, and wherein the selected delivery workflow further comprises a parking node that specifies a second physical location for a delivery agent to park when attempting delivery to a third physical location represented by the destination node.

### Claim 14

A method, comprising:
determining, via a client device, a current location of the client device using a receiver associated with the client device;
determining, via the client device, a next node in a delivery workflow based at least in part on the current location of the client device and a vehicle restriction associated with a vehicle being used in association with the client device, wherein the next node comprises a navigational routing waypoint on a delivery route to a destination node and is compatible with the vehicle restriction associated with the vehicle;
rendering, via the client device, the next node in the delivery workflow on an augmented reality display linked to the client device; and
rendering, via the client device, a marker on the augmented reality display, wherein the marker comprises a visual indication of a physical location corresponding to the next node in the delivery workflow.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
rendering, via the client device, a data balloon on the augmented reality display in proximity to the marker, wherein the data balloon comprises information stored within the next node in the delivery workflow.

### Claim 16 (depends on 14)

The method of claim 14, wherein the augmented reality display comprises a headset.

### Claim 17 (depends on 14)

The method of claim 14, wherein the marker is rendered on a three-dimensional model comprising a holographic projection of the physical location of the next node.

### Claim 18 (depends on 14)

The method of claim 14, wherein the next node in the delivery workflow comprises a way point on a road network.

### Claim 19 (depends on 14)

The method of claim 14, wherein the next node in the delivery workflow comprises a delivery destination.

### Claim 20 (depends on 14)

The method of claim 14, wherein the next node in the delivery workflow comprises a parking point.