# US 2021/0278834 A1 · Method for Exploration and Mapping Using an Aerial Vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0278834 A1
- Document ID: US20210278834A1
- Status: Published application
- Kind: A1
- Application number: 17/260,781
- Filed: 2019-07-17
- Published: 2021-09-09
- Earliest claimed: 2018-07-17
- Assignees: Emesent IP Pty Ltd.
- Applicants: Emesent IP Pty Ltd.
- Inventors: Farid Kendoul; Stefan Hrabar
- Web page: https://patentgrep.com/patent/US20210278834A1

## CPC

- G05D 1/102: {specially adapted for vertical take-off of aircraft}
- G08G 5/32: for flight plan preparation
- G05D 1/0016: {characterised by the operator's input device (input arrangements for computing systems in general G06F3/00)}
- G05D 1/0022: {characterised by the communication link (data switching networks in general H04L12/00)}
- G05D 1/0027: {involving a plurality of vehicles, e.g. fleet or convoy travelling (fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00)}
- G05D 1/0044: {by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps (maps used for automatic navigation G05D1/0274; flight directors G01C23/005)}
- G05D 1/1064: {specially adapted for avoiding collisions with other aircraft}
- B64U 2101/30: for imaging, photography or videography
- B64U 2201/10: autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
- G01S 17/89: for mapping or imaging
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

A method for use in performing exploration and mapping of an environment, the method being performed using an aerial vehicle and a user processing system that wirelessly communicates with the aerial vehicle when the aerial vehicle is within communication range thereof, the method including: the aerial vehicle generating range data using a range sensor; whilst the aerial vehicle is within communication range, the aerial vehicle transmitting, to the user processing system, map data based on the range data; the user processing system displaying a map representation based on the map data; the user processing system obtaining user defined flight instructions; whilst the aerial vehicle is within communication range, the user processing system transmitting, to the aerial vehicle, flight instructions data based on the user defined flight instructions; and the aerial vehicle flying autonomously in accordance with the flight instructions data and the range data.

## Claims

### Claim 1

A method for use in performing exploration and mapping of an environment, the method being performed using an aerial vehicle and a user processing system that wirelessly communicates with the aerial vehicle when the aerial vehicle is within communication range of the user processing system, the method including:
a) the aerial vehicle generating range data using a range sensor, the range data being indicative of a range to the environment;
b) whilst the aerial vehicle is within communication range of the user processing system, the aerial vehicle transmitting, to the user processing system, map data based on the range data;
c) the user processing system displaying, using a graphical user interface, a map representation based on the map data;
d) the user processing system obtaining user defined flight instructions in accordance with user interactions with the graphical user interface;
e) whilst the aerial vehicle is within communication range of the user processing system, the user processing system transmitting, to the aerial vehicle, flight instructions data based on the user defined flight instructions; and
f) the aerial vehicle flying autonomously in accordance with the flight instructions data and the range data.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the method includes generating a map of the environment based on the range data.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the method includes, in one or more vehicle processing devices of the aerial vehicle, determining a flight plan based on the flight instructions data, the aerial vehicle flying autonomously in accordance with the flight plan.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the method includes, in the one or more vehicle processing devices:
a) using the range data to generate pose data indicative of a position and orientation of the aerial vehicle relative to the environment;
b) using the pose data and the flight instructions data to identify manoeuvres that can be used to execute the flight plan;
c) generating control instructions in accordance with the manoeuvres; and
d) transferring the control instructions to a vehicle control system of the aerial vehicle to cause the aerial vehicle to implement the manoeuvres and thereby fly autonomously in accordance with the flight plan.

### Claim 5 (depends on 4)

The system according to claim 4, wherein the method includes, in the one or more vehicle processing devices:
a) using the range data and pose data to generate a depth map indicative of a minimum range to the environment in a plurality of directions; and
b) identifying the manoeuvres in accordance with the depth map to thereby perform collision avoidance.

### Claim 6 (depends on 4)

The method according to claim 4, wherein the method includes, in the one or more vehicle processing devices:
a) using the range data and pose data to generate an occupancy grid indicative of a presence of the environment in different voxels of the occupancy grid; and
b) identifying the manoeuvres using the occupancy grid.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the method includes, while the aerial vehicle is flying autonomously, the aerial vehicle performing collision avoidance in accordance with the range data and at least one of:
a) an extent to the aerial vehicle; and
b) an exclusion volume surrounding an extent of the aerial vehicle.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the user defined flight instructions include one or more user defined waypoints obtained in accordance with user interactions with the graphical user interface.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the method includes the user processing system generating the flight instructions data based on the one or more user defined waypoints and the map data.

### Claim 10 (depends on 9)

The method according to claim 9, wherein the method includes, for each user defined waypoint, the user processing system determining whether the user defined waypoint is separated from the environment by a predefined separation distance.

### Claim 11 (depends on 10)

The method according to claim 10, wherein the method includes, in the event of a determination that the user defined waypoint is separated from the environment by the predefined separation distance, the user processing system generating the flight instructions data using the user defined waypoint.

### Claim 12 (depends on 10)

The method according to claim 10, wherein the method includes, in the event of a determination that the user defined waypoint is not separated from the environment by the predefined separation distance, the user processing system modifying the user defined waypoint and generating the flight instructions data using the resulting modified user defined waypoint.

### Claim 13 (depends on 12)

The method according to claim 12, wherein the method includes the user processing system modifying the user defined waypoint by shifting the user defined waypoint to a nearby point that is separated from the environment at least one of:
a) by a predefined separation distance; and
b) in accordance with defined constraints.

### Claim 14 (depends on 1)

The method according to claim 1, wherein the user defined flight instructions include a predefined flight path segment selected in accordance with user interactions with the graphical user interface.

### Claim 15 (depends on 1)

The method according to claim 1, wherein the user defined flight instructions include a predefined flight plan selected in accordance with user interactions with the graphical user interface.

### Claim 16 (depends on 1)

The method according to claim 1, wherein the method includes the user processing system:
a) generating a preview flight path based on the user defined flight instructions and the map data; and
b) displaying, using the graphical user interface, the preview flight path in the map representation, for approval by the user.

### Claim 17 (depends on 16)

The method according to claim 16, wherein the method includes the user processing system generating the preview flight path by determining flight path segments between waypoints of the user defined flight instructions.

### Claim 18 (depends on 17)

The method according to claim 17, wherein the method includes the user processing system determining each flight path segment so that the flight path segment is separated from the environment by a predefined separation distance.

### Claim 19 (depends on 16)

The method according to claim 16, wherein the method includes the user processing system:
a) obtaining user approval of the preview flight path in accordance with user interactions with the graphical user interface; and
b) in response to the user approval, transmitting the flight instructions data to the aerial vehicle.

### Claim 20 (depends on 16)

The method according to claim 16, wherein the method includes the user processing system:
a) obtaining a user modification input in accordance with user interactions with the graphical user interface, for identifying a desired modification to the user defined flight instructions; and
b) modifying the user defined flight instructions in response to the user modification input.

### Claim 21 (depends on 20)

The method according to claim 20, wherein the user defined flight instructions include waypoints and the method includes modifying the user defined flight instructions by at least one of:
a) removing one of the waypoints;
b) moving one of the waypoints; and
c) adding a new waypoint.

### Claim 22 (depends on 1)

The method according to claim 1, wherein the method includes, whilst the aerial vehicle is flying autonomously:
a) the aerial vehicle continuing to generate range data; and
b) whilst the aerial vehicle is within communication range of the user processing system, the aerial vehicle transmitting, to the user processing system, further map data generated based on the range data.

### Claim 23 (depends on 22)

The method according to claim 22, wherein the further map data includes one of:
a) any updates to the map data;
b) updates to the map data in a predetermined time window;
c) updates to the map data within a predetermined range of the aerial vehicle; and
d) updates to the map data within a predetermined range of waypoints.

### Claim 24 (depends on 1)

The method according to claim 1, wherein the method includes the aerial vehicle, upon completion of autonomous flight in accordance with the flight instructions data, determining whether the aerial vehicle is within communication range of the user processing system at a final position.

### Claim 25 (depends on 24)

The method according to claim 24, wherein the method includes, in the event of a determination that the aerial vehicle is within communication range, the aerial vehicle hovering at the final position to await transmission of further flight instructions data from the user processing system.

### Claim 26 (depends on 24)

The method according to claim 24, wherein the method includes, in the event of a determination that the aerial vehicle is not within communication range, the aerial vehicle autonomously flying to a communications position that is within communication range and hovering at the communications position to await transmission of further flight instructions data from the user processing system.

### Claim 27 (depends on 26)

The method according to claim 26, wherein the method includes, in one or more vehicle processing devices of the aerial vehicle, determining a return flight plan based on the communications position and the range data, the aerial vehicle flying autonomously to the communications position in accordance with the return flight plan.

### Claim 28 (depends on 27)

The method according to claim 27, wherein the method includes, whilst the aerial vehicle is flying autonomously, in the one or more vehicle processing devices:
a) determining whether the aerial vehicle is within communication range of the user processing system; and
b) storing at least an indication of a previous location that was within communication range.

### Claim 29 (depends on 28)

The method according to claim 28, wherein the flight instructions data includes waypoints and the method includes the aerial vehicle storing an indication of whether each waypoint is within communication range after flying autonomously through each waypoint.

### Claim 30 (depends on 1)

The method according to claim 1, wherein the map data includes at least one of:
a) at least some of the range data;
b) a three dimensional map generated based on the range data;
c) an occupancy grid indicative of a presence of the environment in different voxels of the occupancy grid;
d) a depth map indicative of a minimum range to the environment in a plurality of directions; and
e) a point cloud indicative of points in the environment detected by the range sensor.

### Claim 31 (depends on 1)

The method according to claim 1, wherein the map data is at least one of:
a) generated as a down-sampled version of a map generated by the aerial vehicle using the range data;
b) generated using simplified representations of known types of structures determined using the range data; and
c) generated based on a subset of the range data.

### Claim 32 (depends on 1)

The method according to claim 1, wherein the map representation includes at least one of:
a) a two dimensional representation of the environment generated using the map data; and
b) colour coded points where a colour of each point is selected to indicate at least one of:
i) a position of the point in at least one dimension; and
ii) a distance of the point relative to the aerial vehicle in at least one dimension.

### Claim 33 (depends on 1)

The method according to claim 1, wherein the method includes the user processing system dynamically updating the map representation in response to user manipulations of the map representation in accordance with user interactions with the graphical user interface.

### Claim 34 (depends on 1)

The method according to claim 1, wherein the method includes:
a) the aerial vehicle transmitting, to the user processing system, pose data together with the map data; and
b) the user processing system displaying a vehicle representation in the map representation based on the pose data.

### Claim 35 (depends on 1)

The method according to claim 1, wherein the method includes:
a) the aerial vehicle transmitting, to the user processing system, flight plan data indicative of a flight plan determined by the aerial vehicle; and
b) the user processing system displaying a representation of the flight plan in the map representation, based on the flight plan data.

### Claim 36 (depends on 1)

The method according to claim 1, wherein the method includes:
a) the user processing system obtaining at least one user selected heading in accordance with user interactions with the graphical user interface; and
b) the user processing system generating the flight instructions data in accordance with the user selected heading.

### Claim 37 (depends on 1)

The method according to claim 1, wherein the method includes:
a) the user processing system determining flight parameters with regard to the user defined flight instructions; and
b) the user processing system generating the flight instructions data in accordance with the flight parameters.

### Claim 38 (depends on 1)

The method according to claim 1, wherein the method includes:
a) the user processing system obtaining a user command from the user in accordance with user interactions with the graphical user interface;
b) if the aerial vehicle is within communication range of the user processing system, the user processing system transmitting a vehicle command to the aerial vehicle based on the user command; and
c) the aerial vehicle executing the vehicle command.

### Claim 39 (depends on 1)

The method according to claim 1, wherein the method includes:
a) the aerial vehicle transmitting status data to the user processing system, the status data including at least one of:
i) a mission status; and
ii) status of one or more subsystems of the aerial vehicle; and
b) the user processing displaying the status data using the graphical user interface.

### Claim 40 (depends on 1)

The method according to claim 1, wherein the method includes:
a) the aerial vehicle transmitting a completion message to the user processing system upon completion of autonomous flight in accordance with the flight instructions data; and
b) the user processing system generating a user notification in response to receiving the completion message.

### Claim 41 (depends on 1)

The method according to claim 1, wherein the user defined flight instructions are for causing the aerial vehicle to:
a) fly autonomously beyond visual line of sight of the user; and
b) fly autonomously outside of communication range of the user processing system.

### Claim 42 (depends on 1)

The method according to claim 1, wherein the range sensor is a Lidar sensor.

### Claim 43 (depends on 1)

The method according to claim 1, wherein the environment is a GPS-denied environment.

### Claim 44 (depends on 1)

The method according to claim 1, wherein the environment is one of indoors and underground.

### Claim 45 (depends on 1)

The method according to claim 1, wherein the method includes using a simultaneous localisation and mapping algorithm to at least one of:
a) generate a map of the environment based on the range data; and
b) generate pose data indicative of a position and orientation of the aerial vehicle relative to the environment.

### Claim 46 (depends on 1)

The method according to claim 1, wherein the user defined flight instructions are for causing the aerial vehicle to fly autonomously into a region of the environment for which map data is not available.

### Claim 47 (depends on 46)

The method according to claim 46, wherein the user defined flight instructions include a user defined exploration target obtained in accordance with user interactions with the graphical user interface.

### Claim 48 (depends on 47)

The method according to claim 47, wherein the user defined exploration target is at least one of:
a) a target waypoint;
b) a target plane;
c) a target area;
d) a target volume;
e) a target object; and
f) a target point.

### Claim 49 (depends on 47)

The method according to claim 47, wherein the user defined flight instructions are for causing the aerial vehicle to fly autonomously towards the user defined exploration target while performing collision avoidance in accordance with the range data.

### Claim 50

A method for use in performing exploration and mapping of an environment, the method being performed using an aerial vehicle including a range sensor for generating range data indicative of a range to the environment and a user processing system that wirelessly communicates with the aerial vehicle when the aerial vehicle is within communication range of the user processing system, the method including, in the user processing system:
a) receiving map data based on the range data whilst the aerial vehicle is within communication range of the user processing system;
b) displaying a map representation based on the map data using a graphical user interface;
c) obtaining user defined flight instructions in accordance with user interactions with the graphical user interface; and
d) transmitting flight instructions data to the aerial vehicle based on the user defined flight instructions, whilst the aerial vehicle is within communication range of the user processing system, and wherein the aerial vehicle is responsive to fly autonomously in accordance with the flight instructions data and the range data.

### Claim 51

A system for use in performing exploration and mapping of an environment, the system including:
a) an aerial vehicle including a range sensor for generating range data indicative of a range to the environment; and
b) a user processing system configured to wirelessly communicate with the aerial vehicle when the aerial vehicle is within communication range of the user processing system, and wherein the user processing system is configured to:
i) receive map data based on the range data whilst the aerial vehicle is within communication range of the user processing system;
ii) display a map representation based on the map data using a graphical user interface;
iii) obtain user defined flight instructions in accordance with user interactions with the graphical user interface; and
iv) transmit flight instructions data to the aerial vehicle based on the user defined flight instructions, whilst the aerial vehicle is within communication range of the user processing system, and wherein the aerial vehicle is responsive to fly autonomously in accordance with the flight instructions data and the range data.
