# US 2021/0271369 A1 · Tool tracking and task management in a three-dimensional environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0271369 A1
- Document ID: US20210271369A1
- Status: Published application
- Kind: A1
- Application number: 16/802,726
- Filed: 2020-02-27
- Published: 2021-09-02
- Earliest claimed: 2020-02-27
- Assignees: The Boeing Company
- Applicants: The Boeing Company
- Inventors: Kurt Webster; Donald W. Coffland; Natalia Roberts
- Web page: https://patentgrep.com/patent/US20210271369A1

## CPC

- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G05B 19/402: characterised by control arrangements for positioning, e.g. centring a tool relative to a hole in the workpiece, additional detection means to correct position (G05B19/19 takes precedence)
- G05B 2219/32128: Gui graphical user interface
- G05B 2219/35415: 3-D three dimension, space input, spaceball
- G05B 2219/50047: Positioning, indexing
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A system for affixing a fastener or fastener collar is provided including a handheld tool configured to engage a corresponding fastener; a communication device affixed to the handheld tool and configured to communicate with a base station; and a computer. In some example implementations, the computer is configured to receive an identification of a position of the handheld tool within a region of three-dimensional space, and determine, based on the position of the handheld tool and a user input, a location of the corresponding fastener. A digital representation of the region of three-dimensional space with an identification of the location of the corresponding fastener can then be generated; along with a graphical user interface including a visual representation of the map of the three-dimensional space and a visual representation of the location of the corresponding fastener.

## Claims

### Claim 1

A system for affixing a fastener or fastener collar comprising:
a handheld tool configured to engage a corresponding fastener or fastener collar;
a communication device affixed to the handheld tool and configured to wirelessly communicate with a base station; and
a computer configured to:
receive, from the base station, data identifying a position, orientation and movement of the handheld tool within a region of three-dimensional space;
determine a location of the corresponding fastener or fastener collar, based on the position, orientation and movement of the handheld tool and a user input to a user input interface of the communication device or the computer, and without a previously-stored or mapped location of the corresponding fastener or fastener collar;
generate a graphical user interface (GUI) including a digital representation of the region of three-dimensional space with an identification of the location of the corresponding fastener or fastener collar.

### Claim 2 (depends on 1)

The system of claim 1, wherein the handheld tool is a torque wrench, and the communication device is configured to transmit a torque status of the corresponding fastener or fastener collar to the base station; and
wherein the computer is further configured to:
receive the torque status of the corresponding fastener or fastener collar;
store a record of the torque status of the corresponding fastener or fastener collar; and
provide an indication the torque status via the GUI.

### Claim 3 (depends on 1)

The system of claim 1, wherein the communication device is further configured to receive the user input via the user input interface of the communication device and transmit a signal to the base station to cause the computer to record the position of the handheld tool within the region of three-dimensional space on the digital representation.

### Claim 4 (depends on 1)

The system of claim 1, wherein the GUI includes a rendering of the region of three-dimensional space and a rendering of the corresponding fastener or fastener collar applied to the rendering of the region of three-dimensional space in a position on the GUI corresponding to the location of the corresponding fastener or fastener collar within the region of three-dimensional space.

### Claim 5

A system for affixing a fastener or fastener collar comprising:
a handheld tool configured to engage a corresponding fastener or fastener collar;
a communication device affixed to the handheld tool and configured to wirelessly communicate with a base station; and
a computer configured to:
receive, from the base station, data identifying a position, orientation and movement of the handheld tool within a region of three-dimensional space;
access a map of the region of three dimensional space, including an identification of a location of the corresponding fastener or fastener collar;
determine, based on the position of the handheld tool and the identification of the location of the corresponding fastener or fastener collar, a relative position of the handheld tool with respect to the corresponding fastener or fastener collar;
determine, based on one or more force measurements, and the orientation and movement of the handheld tool, one or more forces applied by the handheld tool on the corresponding fastener or fastener collar, the one or more forces including respective directions that indicate tightening and loosening operations performed by the handheld tool;
determine a status of the corresponding fastener or fastener collar based on the one or more forces including the respective directions; and
provide an indication of the relative position of the handheld tool with respect to the corresponding fastener or fastener collar, and the status of the corresponding fastener or fastener collar.

### Claim 6 (depends on 5)

The system of claim 5, wherein the computer being configured to determine the relative position of the handheld tool with respect to the corresponding fastener or fastener collar comprises the computer being configured to:
apply the position of the handheld tool within the region of three-dimensional space to the map; and
determine a difference between the applied position of the handheld tool and the identification of the location of the corresponding fastener or fastener collar.

### Claim 7 (depends on 5)

The system of claim 5, wherein the computer being configured to provide the indication of the relative position of the handheld tool with respect to the corresponding fastener or fastener collar, comprises the computer being configured to:
generate a graphical user interface (GUI) including a digital representation of the map of the three-dimensional space, with the location of the corresponding fastener or fastener collar, and the position of the handheld tool with respect to the corresponding fastener or fastener collar.

### Claim 8 (depends on 5)

The system of claim 5, wherein the computer being configured to provide the indication of the relative position of the handheld tool with respect to the corresponding fastener or fastener collar, comprises the computer being configured to transmit to the communication device, via the base station, the indication of the relative position of the handheld tool with respect to the corresponding fastener or fastener collar; and
wherein the communication device is further configured to provide visual, audio, or haptic feedback to a user based on the indication of the relative position of the handheld tool with respect to the corresponding fastener or fastener collar.

### Claim 9 (depends on 8)

The system of claim 8, wherein a characteristic of the visual, audio or haptic feedback is based at least in part of the relative position of the handheld tool with respect to the corresponding fastener or fastener collar.

### Claim 10 (depends on 5)

The system of claim 5, wherein the map further includes an identification of a location of a second corresponding fastener or fastener collar.

### Claim 11 (depends on 10)

The system of claim 10, wherein the computer is further configured to determine, based on the position of the handheld tool and the identification of the location of the second corresponding fastener or fastener collar, a relative position of the handheld tool with respect to the second corresponding fastener or fastener collar; and
provide an indication of the relative position of the handheld tool with respect to the second corresponding fastener or fastener collar.

### Claim 12

A method for affixing a fastener or fastener collar, the method comprising:
receiving, at a computer, from a base station, data identifying a position, orientation and movement, within a region of three-dimensional space, of a handheld tool to which a communication device is affixed;
determining a location of the corresponding fastener or fastener collar, based on the position of the handheld tool and a user input to a user input interface of the communication device or the computer, and without a previously-stored or mapped location of a corresponding fastener or fastener collar; and
generating a graphical user interface (GUI) including a digital representation of the region of three-dimensional space with an identification of the location of the corresponding fastener or fastener collar.

### Claim 13 (depends on 12)

The method of claim 12, wherein the handheld tool is a torque wrench and the communication device is configured to transmit a torque status of the corresponding fastener or fastener collar to the base station, the method further comprising:
receiving the torque status of the corresponding fastener or fastener collar;
storing a record of the torque status of the corresponding fastener or fastener collar; and
providing an indication of the torque status via the GUI.

### Claim 14 (depends on 12)

The method of claim 12, wherein the communication device is further configured to receive the user input via the user input interface of the communication device and transmit a signal to the base station to cause the computer to record the position of the handheld tool within the region of three-dimensional space on the digital representation.

### Claim 15 (depends on 12)

The method of claim 12, wherein GUI includes a rendering of the region of three-dimensional space and a rendering of the corresponding fastener or fastener collar applied to the rendering of the region of three-dimensional space in a position on the GUI corresponding to the location of the corresponding fastener or fastener collar within the region of three-dimensional space.

### Claim 16

A method for affixing a fastener or fastener collar, the method comprising:
receiving, at a computer, from a base station, data identifying a position, orientation and movement, within a region of three-dimensional space, of a handheld tool to which a communication device is affixed;
accessing a map of the region of three dimensional space, including an identification of a location of a corresponding fastener or fastener collar;
determining, based on the position of the handheld tool and the identification of the location of the corresponding fastener or fastener collar, a relative position of the handheld tool with respect to the corresponding fastener or fastener collar;
determining, based on one or more force measurements, and the orientation and movement of the handheld tool, one or more forces applied by the handheld tool on the corresponding fastener or fastener collar, the one or more forces including respective directions that indicate tightening and loosening operations performed by the handheld tool;
determining a status of the corresponding fastener or fastener collar based on the one or more forces including the respective directions; and
providing an indication of the relative position of the handheld tool with respect to the corresponding fastener or fastener collar, and the status of the corresponding fastener or fastener collar.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
applying the position of the handheld tool within the region of three-dimensional space to the map; and
determining a difference between an applied position of the handheld tool and the identification of the location of the corresponding fastener or fastener collar.

### Claim 18 (depends on 16)

The method of claim 16, wherein providing the indication of the relative position of the handheld tool with respect to the corresponding fastener or fastener collar further comprises:
generating a graphical user interface (GUI) including a digital representation of the map of the three-dimensional space, with the location of the corresponding fastener or fastener collar, and the position of the handheld tool with respect to the corresponding fastener or fastener collar.

### Claim 19 (depends on 16)

The method of claim 16, wherein providing the indication of the relative position of the handheld tool with respect to the corresponding fastener or fastener collar, comprises the computer being configured to transmit to the communication device, via the base station, the indication of the relative position of the handheld tool with respect to the corresponding fastener or fastener collar; and
wherein the communication device is further configured to provide visual, audio, or haptic feedback to a user based on the indication of the relative position of the handheld tool with respect to the corresponding fastener or fastener collar.

### Claim 20 (depends on 19)

The method of claim 19, wherein a characteristic of the visual, audio or haptic feedback is based at least in part of the relative position of the handheld tool with respect to the corresponding fastener of fastener collar.

### Claim 21 (depends on 16)

The method of claim 16, wherein the map further includes an identification of a location of a second corresponding fastener or fastener collar.

### Claim 22 (depends on 21)

The method of claim 21, further comprising:
determining, based on the position of the handheld tool and the identification of the location of the second corresponding fastener or fastener collar, a relative position of the handheld tool with respect to the second corresponding fastener or fastener collar; and
providing an indication of the relative position of the handheld tool with respect to the second corresponding fastener or fastener collar.
