# US 2019/0244428 A1 · Augmented reality based horticultural care tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244428 A1
- Document ID: US20190244428A1
- Status: Published application
- Kind: A1
- Application number: 15/891,110
- Filed: 2018-02-07
- Published: 2019-08-08
- Earliest claimed: 2018-02-07
- Assignees: iUNU, Inc.
- Applicants: iUNU, Inc.
- Inventors: Adam Phillip Takla Greenberg; Matthew Charles King
- Web page: https://patentgrep.com/patent/US20190244428A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A01G 7/06: Treatment of growing trees or plants, e.g. for preventing decay of wood, for tingeing flowers or wood, for prolonging the life of plants
- G06Q 10/06: Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q 50/02: Agriculture; Fishing; Forestry; Mining
- G06T 7/001: {using an image reference approach}
- G06V 20/68: Food, e.g. fruit or vegetables
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A01G: HORTICULTURE; CULTIVATION OF VEGETABLES, FLOWERS, RICE, FRUIT, VINES, HOPS OR SEAWEED; FORESTRY; WATERING (picking of fruits, vegetables, hops or the like A01D46/00; propagating unicellular algae C12N1/12)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An augmented reality-based horticultural care tracking system to collect information about plants on a per plant or near per plant basis in order to identify issues and generate itineraries that include task lists for remedial courses of actions is disclosed. A network-enabled augmented reality device can be employed by operators of various growing operations in order to overlay digital information such as tasks, images, and related information in a real-time environment. The augmented reality device can communicate with administrative personnel at an operating station and/or intermediate servers in the network in order to send and receive status updates and notifications to properly execute the tasks in the itinerary and evaluate the courses of actions' efficacy.

## Claims

### Claim 1

A computer-implemented method, comprising:
receiving, from an image capture device (ICD), images depicting an identified plant or an identified artifact, wherein at least one of the images is an image of interest associated with an identified issue of the identified plant or the identified artifact;
identifying a course of action associated with the identified issue from a solutions table to resolve the identified issue, wherein the course of action comprises one or more tasks;
displaying the one or more tasks on an augmented reality device selected from a plurality of augmented reality devices based at least on an individual operator associated with the selected augmented reality device; and
associating the one or more tasks with at least one graphical indicia that is displayed on the selected augmented reality device.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating an itinerary comprising a task list that includes the one or more tasks, wherein the one or more tasks is sorted in order of priority, wherein the itinerary is associated with the selected augmented reality device.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving a notification from the selected augmented reality device that the one or more tasks is complete;
determining whether the identified issue is resolved;
if the identified issue is not resolved, generating an alternate course of action, wherein the alternate course of action includes one or more new tasks; and
displaying the one or more new tasks on the selected augmented reality device, wherein the one or more new tasks can be associated with the at least one graphical indicia.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving a notification from the selected augmented reality device that the one or more tasks is complete; and
indicating via the at least one graphical indicia on the selected augmented reality device that the one or more tasks is complete.

### Claim 5 (depends on 2)

The computer-implemented method of claim 2, further comprising:
assigning a priority level to each of the one or more tasks; and
displaying on the selected augmented reality device the one or more tasks from a highest priority level to a lowest priority level.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more tasks and the at least one graphical indicia are displayed as an overlay to a real environment or a hologram display.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving a request from the selected augmented reality device to reassign the one or more tasks, wherein the selected augmented reality device is a first augmented reality device; and
displaying the one or more tasks on a second augmented reality device selected from the plurality of augmented reality devices, wherein the first augmented reality device and the second augmented reality device are associated with different operators.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the image of interest comprises annotations associated with the identified issue depicted in the image of interest.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the at least one graphical indicia is location based.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving a notification from the selected augmented reality device that the one or more tasks is complete;
requesting a first image associated with the identified issue of the identified plant or the identified artifact just before the one or more tasks was reported as complete;
requesting a second image associated with the identified issue of the identified plant or the identified artifact just after the one or more tasks was reported as complete; and
comparing the first image and the second image to determine whether the identified issue is resolved.

### Claim 11

A computer-implemented method, comprising:
receiving, from an image capture device (ICD), at least one image of interest depicting an identified artifact of a plant, the at least one image of interest associated with an identified issue of the identified artifact;
identifying a course of action associated with the identified issue from a solutions table to resolve the identified issue, wherein the course of action comprises individual tasks assigned to an operator based on one or more conditions;
displaying the individual tasks on an augmented reality device of a plurality of augmented reality devices associated with the operator, wherein the individual tasks can be associated with at least one graphical indicia; and
receiving annotations from the operator for display on the augmented reality device.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, further comprising:
generating an itinerary comprising a task list that includes individual tasks;
assigning a priority level to the individual tasks; and
displaying, on the augmented reality device, a list of the individual tasks from a highest priority level to a lowest priority level.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein the assigning comprises:
identifying locations where the individual tasks are to be performed;
grouping the individual tasks by the locations; and
assigning the priority level to the individual tasks based at least on the grouping.

### Claim 14 (depends on 11)

The computer-implemented method of claim 11, further comprising:
receiving a request from the augmented reality device to reassign the individual tasks, wherein the augmented reality device is a first augmented reality device; and
displaying the individual tasks on a second augmented reality device, wherein the first augmented reality device and the second augmented reality device are associated with different operators.

### Claim 15 (depends on 11)

The computer-implemented method of claim 11, wherein the one or more tasks and the at least one graphical indicia are displayed as an overlay to a real environment or a hologram display.

### Claim 16

A system, comprising:
at least one augmented reality device;
one or more nontransitory storage mediums configured to provide stored code segments, the one or more nontransitory storage mediums coupled to one or more processors, each configured to execute the code segments and causing the one or more processors to:
receive an image of interest depicting an identified plant or an identified artifact, wherein the image of interest is associated with an identified issue of the identified plant or the identified artifact;
identify a course of action associated with the identified issue from a solutions table to resolve the identified issue, wherein the course of action comprises one or more tasks assigned to an operator; and
display the one or more tasks on the at least one augmented reality device associated with the operator, wherein the one or more tasks can be associated with at least one graphical indicia.

### Claim 17 (depends on 16)

The system of claim 16, wherein the at least one augmented reality device comprises a first augmented reality device and a second augmented reality device, the first augmented reality device and the second augmented reality device configured to screen share.

### Claim 18 (depends on 16)

The system of claim 16, wherein the one or more processors is further configured to:
receive a notification that the one or more tasks is complete;
determine whether the identified issue is resolved;
if the identified issue is not resolved, generate an alternate course of action, wherein the alternate course of action includes one or more new tasks assigned to a second operator; and
display the one or more new tasks on a second augmented reality device associated with the second operator, wherein the one or more new tasks can be associated with the at least one graphical indicia;
if the identified issue is resolved, indicate on the at least one augmented reality device that the one or more tasks is complete.

### Claim 19 (depends on 16)

The system of claim 16, wherein the at least one augmented reality device is configured to display the one or more tasks and the at least one graphical indicia as an overlay to a real environment or a hologram display.

### Claim 20 (depends on 16)

The system of claim 16, wherein the one or more processors is further configured to:
receive a request from the at least one augmented reality device to reassign the one or more tasks, wherein the at least one augmented reality device comprises a first augmented reality device;
reassign the one or more tasks to a second operator; and
display the one or more tasks on a second augmented reality device associated with the second operator.
