# US 2020/0394779 A1 · Management of tubular storage in a fingerboard using a vision system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394779 A1
- Document ID: US20200394779A1
- Status: Published application
- Kind: A1
- Application number: 16/897,650
- Filed: 2020-06-10
- Published: 2020-12-17
- Earliest claimed: 2019-06-11
- Assignees: Nabors Drilling Technologies USA, Inc.
- Applicants: Nabors Drilling Technologies USA, Inc.
- Inventors: Arima AYANAMBAKKAM; Pradeep ANNAIYAPPA
- Web page: https://patentgrep.com/patent/US20200394779A1

## CPC

- G06T 7/62: of area, perimeter, diameter or volume
- E21B 19/06: Elevators, i.e. rod- or tube-gripping devices
- E21B 19/14: Racks, ramps, troughs or bins, for holding the lengths of rod singly or connected; Handling between storage place and borehole (E21B19/20, E21B19/22 take precedence)
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 7/001: {using an image reference approach}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N 7/185: {from a mobile camera, e.g. for remote control}
- E21B 19/155: {Handling between horizontal and vertical position}
- G06T 2207/30164: Workpiece; Machine component
- G06T 2207/30204: Marker
- G06T 2207/30242: Counting objects in image
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- E21B: EARTH OR ROCK DRILLING; OBTAINING OIL, GAS, WATER, SOLUBLE OR MELTABLE MATERIALS OR A SLURRY OF MINERALS FROM WELLS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A fingerboard management system that can determine via a rig controller, a selection of a location in the fingerboard for one or more tubulars and placement of the one or more tubulars in the fingerboard based on available locations in the fingerboard and historical information of each of the one or more tubulars and a method that can include importing a 3D model into a rig controller, collecting images of at least a portion of the rig, comparing the images to the 3D model or a visible marker, calibrating the vision system by determining a position, an orientation, and a field of view for each of the sensors based on the comparing, collecting additional images of the fingerboard, and determining an available total tubular storage space of the fingerboard based on the additional images of the fingerboard and the calibrated vision system.

## Claims

### Claim 1

A method of managing tubular storage in a fingerboard on a rig, the method comprising:
determining, via a rig controller, a selection of a location in the fingerboard for one or more tubulars and placement of the one or more tubulars in the fingerboard based on available locations in the fingerboard and historical information of each of the one or more tubulars.

### Claim 2 (depends on 1)

The method of claim 1, wherein the historical information for each one of the one or more tubulars comprises:
one or more positions in one or more tubular strings in which each one of the one or more tubulars have been utilized;
duration of a prior utilization of each one of the one or more tubulars in the one or more tubular strings;
torque applied to the one or more tubular strings during the prior utilization;
weight on bit during the prior utilization;
environmental conditions during the prior utilization; or
combinations thereof.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
collecting, via sensors of a vision system, images of at least a portion of the fingerboard;
identifying each one of the one or more tubulars being stored in or retrieved from the fingerboard based on the images; and
logging a storage location for each of the one or more tubulars being stored in the fingerboard or retrieved from the fingerboard.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
storing, via a pipe handler, the one or more tubulars at a respective storage location in the fingerboard; or
retrieving, via the pipe handler, the one or more tubulars from a respective storage location in the fingerboard, wherein the pipe handler is a manually operated pipe handler or an automated pipe handler.

### Claim 5 (depends on 2)

The method of claim 2, further comprising:
collecting, via sensors of a vision system, one or more first images of at least a portion of the rig;
comparing the first images, via the rig controller, to a reference, wherein the reference is a 3D model, a visible distance indicator, or a combination thereof;
calibrating the vision system by determining, via the rig controller, a position, an orientation, a field of view, or combinations thereof for each of the sensors based on the comparing;
collecting, via the sensors, one or more second images of at least a portion of the rig, with the second images comprising images of the fingerboard; and
determining, via the rig controller, an available total tubular storage space of the fingerboard based on the images of the fingerboard and the calibrated vision system.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
determining, via the rig controller, a width and a length of each one of multiple slots of the fingerboard based on the images of the fingerboard and the calibrated vision system; and
determining, via the rig controller, a slot tubular storage space of each one of the multiple slots based on the width and the length of each one of the multiple slots.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
collecting, via the sensors, one or more third images of at least a portion of the rig, with the third images comprising images of a first tubular;
calculating, via the rig controller, parameters of the first tubular based on the third images and the calibrated vision system; and
determining a type of the first tubular based on the parameters of the first tubular.

### Claim 8 (depends on 7)

The method of claim 7, wherein the parameters of the first tubular comprise:
a presence or absence of a box;
a length of the box, when the box is present;
a diameter of the box, when the box is present;
a diameter of a body portion;
a presence or absence of a sub attached at an end of the first tubular;
a length of the sub, when the sub is present;
a diameter of each end of the sub, when the sub is present;
a diameter of a body of the sub, when the sub is present;
a height, including the sub, above the fingerboard when the sub is present;
the height of the first tubular, without the sub, above the fingerboard when the sub is absent;
an overall length;
cleanliness;
visible markings; or
combinations thereof.

### Claim 9 (depends on 7)

The method of claim 7, further comprising:
determining, via the rig controller, which one of multiple slots of the fingerboard to store the first tubular based on the type of the first tubular;
storing the first tubular in the one of the multiple slots; and
determining, via the rig controller, a remaining portion of the total tubular storage space available to store additional tubulars when the first tubular is stored in the one of the multiple slots.

### Claim 10 (depends on 5)

The method of claim 5, further comprising:
receiving multiple tubulars into the fingerboard;
collecting, via the sensors, one or more fourth images of each one of the multiple tubulars;
calculating, via the rig controller, parameters of each one of the multiple tubulars based on the fourth images and the calibrated vision system;
determining a type of each one of the multiple tubulars based on the parameters of each one of the multiple tubulars; and
logging the parameters and the type of each one of the multiple tubulars.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
determining, via the rig controller, a respective storage location in the fingerboard for each one of the multiple tubulars based on the type of each one of the multiple tubulars;
storing each one of the multiple tubulars in the respective storage location; and
logging, via the rig controller, the respective storage location for each one of the multiple tubulars; and
determining, via the rig controller, a remaining portion of the total tubular storage space of the fingerboard when the multiple tubulars are stored in the fingerboard.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
logging historical information for each one of the multiple tubulars; and
planning, via the rig controller, future utilization of the multiple tubulars based on the historical information of each one of the multiple tubulars.

### Claim 13

A method of managing tubular storage in a fingerboard, the method comprising:
importing a 3D model of a rig into a rig controller;
collecting, via sensors of a vision system, one or more first images of at least a portion of the rig;
comparing the first images, via the rig controller, to the 3D model;
calibrating the vision system by determining, via the rig controller, a position, an orientation, a field of view, or combinations thereof for each of the sensors based on the comparing;
collecting, via the sensors, one or more second images of at least a portion of the rig, with the second images comprising images of the fingerboard; and
determining, via the rig controller, an available total tubular storage space of the fingerboard based on the images of the fingerboard and the calibrated vision system.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
determining, via the rig controller, a width and a length of each one of multiple slots of the fingerboard based on the images of the fingerboard and the calibrated vision system; and
determining, via the rig controller, a slot tubular storage space of each one of the multiple slots based on the width and the length of each one of the multiple slots.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
collecting, via the sensors, one or more third images of at least a portion of the rig, with the third images comprising images of a first tubular;
calculating, via the rig controller, parameters of the first tubular based on the third images and the calibrated vision system; and
determining a type of the first tubular based on the parameters of the first tubular.

### Claim 16 (depends on 15)

The method of claim 15, wherein the parameters of the first tubular comprise:
a presence or absence of a box;
a length of the box, when the box is present;
a diameter of the box, when the box is present;
a diameter of a body portion;
a presence or absence of a sub attached at an end of the first tubular;
a length of the sub, when the sub is present;
a diameter of each end of the sub, when the sub is present;
a diameter of a body of the sub, when the sub is present;
a height, including the sub, above the fingerboard when the sub is present;
the height of the first tubular, without the sub, above the fingerboard when the sub is absent;
an overall length;
cleanliness;
visible markings; or
combinations thereof.

### Claim 17 (depends on 15)

The method of claim 15, further comprising:
determining, via the rig controller, which one of multiple slots of the fingerboard to store the first tubular based on the type of the first tubular;
storing the first tubular in the one of the multiple slots; and
determining, via the rig controller, a remaining portion of the total tubular storage space available to store additional tubulars when the first tubular is stored in the one of the multiple slots.

### Claim 18 (depends on 13)

The method of claim 13, further comprising:
receiving multiple tubulars into the fingerboard;
collecting, via the sensors, one or more fourth images of each one of the multiple tubulars;
calculating, via the rig controller, parameters of each one of the multiple tubulars based on the fourth images and the calibrated vision system;
determining a type of each one of the multiple tubulars based on the parameters of each one of the multiple tubulars; and
logging the parameters and the type of each one of the multiple tubulars.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
determining, via the rig controller, a respective storage location in the fingerboard for each one of the multiple tubulars based on the type of each one of the multiple tubulars;
storing each one of the multiple tubulars in the respective storage location; and
logging, via the rig controller, the respective storage location for each one of the multiple tubulars; and
determining, via the rig controller, a remaining portion of the total tubular storage space of the fingerboard when the multiple tubulars are stored in the fingerboard.

### Claim 20 (depends on 19)

The method of claim 19, further comprising:
logging historical information for each one of the multiple tubulars; and
planning, via the rig controller, future utilization of the multiple tubulars based on the historical information of each one of the multiple tubulars, wherein the historical information for each one of the multiple tubulars comprises:
one or more positions in one or more tubular strings in which each one of the multiple tubulars have been utilized prior to the planning;
duration of prior utilization of each one of the multiple tubulars in the one or more tubular strings;
torque applied to the one or more tubular strings during the prior utilization;
weight on bit during the prior utilization;
environmental conditions during the prior utilization; or
combinations thereof.
