# US 2023/0273618 A1 · Systems and methods for out of aisle localization and vehicle position calibration using rack leg identification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0273618 A1
- Document ID: US20230273618A1
- Status: Published application
- Kind: A1
- Application number: 18/312,685
- Filed: 2023-05-05
- Published: 2023-08-31
- Earliest claimed: 2019-02-06
- Assignees: Crown Equipment Corporation
- Applicants: Crown Equipment Corporation
- Inventors: Justin Forbes Thode; Grace Da-In Lee
- Web page: https://patentgrep.com/patent/US20230273618A1

## CPC

- G05D 1/0253: {extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow (determining position or orientation from images G06T7/70)}
- B60R 11/04: Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
- B65G 1/0492: {with cars adapted to travel in storage aisles}
- B66F 9/063: {Automatically guided}
- B66F 9/0755: {Position control; Position detectors}
- B66F 9/08: Masts; Guides; Chains
- B66F 9/20: Means for actuating or controlling masts, platforms, or forks (power take-off from vehicle transmissions, combined with vehicle propulsion systems B60K)
- G05D 1/0212: {with means for defining a desired trajectory (involving a plurality of land vehicles G05D1/0287)}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/0272: {comprising means for registering the travel distance, e.g. revolutions of wheels (measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00)}
- G05D 1/2247: {providing the operator with simple or augmented images from one or more cameras}
- G05D 1/646: Following a predefined trajectory, e.g. a line marked on the floor or a flight path
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D)
- B66F: HOISTING, LIFTING, HAULING OR PUSHING, NOT OTHERWISE PROVIDED FOR, e.g. DEVICES WHICH APPLY A LIFTING OR PUSHING FORCE DIRECTLY TO THE SURFACE OF A LOAD ({invalid lifting devices A61G7/10; } mounting artificial islands on piles or like supports E02B17/00; scaffolding combined with lifting devices E04G1/22; lifting devices for sliding forms E04G11/24; lifting of buildings E04G23/06; shores or struts E04G25/00; lifts or other hoisting devices on ladders E06C7/12; props for mining E21D15/00)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A materials handling vehicle includes a camera, odometry module, processor, and drive mechanism. The camera captures images of an identifier for a racking system aisle and a rack leg portion in the aisle. The processor uses the identifier to generate information indicative of an initial rack leg position and rack leg spacing in the aisle, generate an initial vehicle position using the initial rack leg position, generate a vehicle odometry-based position using odometry data and the initial vehicle position, detect a subsequent rack leg using a captured image, correlate the detected subsequent rack leg with an expected vehicle position using rack leg spacing, generate an odometry error signal based on a difference between the positions, and update the vehicle odometry-based position using the odometry error signal and/or generated mast sway compensation to use for end of aisle protection and/or in/out of aisle localization.

## Claims

### Claim 1

A materials handling vehicle comprising a camera, a vehicle position processor, and a drive mechanism configured to move the materials handling vehicle along an inventory transit surface, wherein the vehicle position processor is configured to:
generate end-of-aisle data and aisle-specific rack leg spacing data from an image of an aisle entry identifier associated with at least a portion of a rack leg positioned in a racking system aisle, as captured by the camera,
generate an initial position of the materials handling vehicle along the inventory transit surface using the end-of-aisle data,
generate an expected position of the materials handling vehicle as the vehicle travels down a racking system aisle corresponding to the aisle entry identifier from the end-of-aisle data and the aisle-specific rack leg spacing data,
navigate the materials handling vehicle to exit the racking system aisle, and
update the expected position of the materials handling vehicle using data received from the materials handling vehicle when the materials handling vehicle is outside of the racking system aisle, wherein the data comprises at least odometry data, steer wheel angle, steer wheel speed, or combinations thereof.

### Claim 2 (depends on 1)

The materials handling vehicle of claim 1, wherein:
the camera is configured to capture images comprising an aisle exit identifier for a racking system aisle of a multilevel warehouse racking system; and
the vehicle position processor is configured to
generate end-of-aisle data from an image of an aisle exit identifier, as captured by the camera,
generate a vehicle pose of the materials handling vehicle relative to the aisle exit identifier using the end-of-aisle data,
generate an inventory transit surface position of the materials handling vehicle along the inventory transit surface using the end-of-aisle data, the vehicle pose, and the expected position,
navigate the materials handing vehicle to exit the racking system aisle, and
update the inventory transit surface position of the materials handling vehicle using a tag identifier component of the materials handling vehicle when the materials handling vehicle is outside of the racking system aisle.

### Claim 3 (depends on 2)

The materials handling vehicle of claim 2, wherein the vehicle position processor is configured to
generate out-of-aisle data from a tag identifier, as received by the tag identifier component, and
update the vehicle pose of the materials handling vehicle relative to the tag identifier using the out-of-aisle data.

### Claim 4 (depends on 1)

The materials handling vehicle of claim 1, wherein:
the materials handling vehicle further comprises a mast assembly, a mast assembly control unit, and a fork carriage assembly;
the mast assembly and the mast assembly control unit are configured to move the fork carriage assembly along a vertical lift axis; and
the camera is secured to the fork carriage assembly and is configured to capture
(i) a forks down image of at least a portion of a rack leg positioned in a racking system aisle of a multilevel warehouse racking system, and
(ii) a forks-up image of at least a portion of a rack leg positioned in a racking system aisle of the multilevel warehouse racking system; and
the vehicle position processor is configured to
generate a forks-down coordinate X1 of the camera along a horizontal axis from the forks-down image of the rack leg,
generate a forks-up coordinate X2 of the camera along a horizontal axis from a forks-up image of the rack leg captured with the fork carriage assembly at a lift height H1,
determine a mast sway offset as a difference between the forks-down coordinate X1 and the forks-up coordinate X2,
determine a horizontally advanced position of the materials handling vehicle with the fork carriage assembly at the lift height H1 using the mast sway offset and a subsequently captured forks-up image of at least a portion of a rack leg positioned in the racking system aisle, and
update the expected position using the horizontally advanced position.

### Claim 5 (depends on 1)

The materials handling vehicle of claim 1, wherein the vehicle position processor is configured to
generate end-of-aisle limit data comprising an end-of-aisle limit from the image of the aisle entry identifier, as captured by the camera,
determine whether the materials handling vehicle exceeds the end-of-aisle limit,
transmit a limit-based command to the drive mechanism of the materials handling vehicle when an operating state of the materials handling vehicle exceeds the end-of-aisle limit,
initiate a vehicle action based on the limit-based command, the vehicle action configured to modify operation of the materials handling vehicle to a modified operating state within the end-of-aisle limit, and
navigate the materials handling vehicle to exit the racking system aisle in the modified operating state.

### Claim 6 (depends on 5)

The materials handling vehicle of claim 5, wherein the end-of-aisle limit comprises a speed limit, a lift height limit, an acceleration limit, a deceleration limit, a lift acceleration limit, a lift deceleration limit, or combinations thereof.

### Claim 7 (depends on 5)

The materials handling vehicle of claim 5, wherein the vehicle position processor is configured to navigate the materials handling vehicle outside of the racking system aisle in the modified operating state, in an additional operating state that is within or outside of the end-of-aisle limit, or in a combination of operating states selected therefrom.

### Claim 8 (depends on 5)

The materials handling vehicle of claim 5, wherein the vehicle position processor is configured to
determine whether the operating state of the materials handling vehicle exceeds the end-of-aisle limit when the expected position of the materials handling vehicle corresponds to an end of the racking system aisle.

### Claim 9

A materials handling vehicle comprising a camera, a vehicle position processor, and a drive mechanism configured to move the materials handling vehicle along an inventory transit surface, wherein the vehicle position processor is configured to:
generate end-of-aisle positional data from an image of an aisle exit identifier for a racking system aisle, as captured by the camera,
generate a vehicle pose of the materials handling vehicle relative to the aisle exit identifier using the end-of-aisle positional data,
generate an inventory transit surface position of the materials handling vehicle along the inventory transit surface using the end-of-aisle positional data and the vehicle pose, and
navigate the materials handing vehicle to exit the racking system aisle based on the inventory transit surface position.

### Claim 10 (depends on 9)

The materials handling vehicle of claim 9, wherein the vehicle position processor is configured to
update the inventory transit surface position of the materials handling vehicle using a tag identifier component of the materials handling vehicle when the materials handling vehicle is outside of the racking system aisle,
generate out-of-aisle positional data from a tag identifier, as received by the tag identifier component, and
update the vehicle pose of the materials handling vehicle relative to the tag identifier using the out-of-aisle positional data.

### Claim 11 (depends on 9)

The materials handling vehicle of claim 9, wherein the vehicle position processor is further configured to generate the inventory transit surface position of the materials handling vehicle from an image of at least a portion of a rack leg positioned in the racking system aisle.

### Claim 12 (depends on 9)

The materials handling vehicle of claim 9, wherein:
the camera is configured to capture images comprising (i) an aisle entry identifier for a racking system aisle of a multilevel warehouse racking system, and (ii) at least a portion of a rack leg positioned in the racking system aisle; and
the vehicle position processor is configured to
generate end-of-aisle positional data and aisle-specific rack leg spacing data from an image of an aisle entry identifier, as captured by the camera,
generate an initial position of the materials handling vehicle along the inventory transit surface in a racking system aisle using the end-of-aisle positional data,
generate an expected position of the materials handling vehicle as the vehicle travels down the racking system aisle corresponding to the aisle entry identifier from the end-of-aisle positional data and the aisle-specific rack leg spacing data, and
update the inventory transit surface position using the expected position.

### Claim 13 (depends on 12)

The materials handling vehicle of claim 12, wherein the vehicle position processor is further configured to update the inventory transit surface position of the materials handling vehicle using a dead reckoning component of the materials handling vehicle when the materials handling vehicle is outside of the racking system aisle.

### Claim 14 (depends on 9)

The materials handling vehicle of claim 9, wherein:
the materials handling vehicle further comprises a mast assembly, a mast assembly control unit, and a fork carriage assembly;
the mast assembly and the mast assembly control unit are configured to move the fork carriage assembly along a vertical lift axis; and
the camera is secured to the fork carriage assembly and is configured to capture
(i) a forks-down image of at least a portion of a rack leg positioned in a racking system aisle of a multilevel warehouse racking system, and
(ii) a forks-up image of at least a portion of a rack leg positioned in a racking system aisle of the multilevel warehouse racking system; and
the vehicle position processor is configured to
generate a forks-down coordinate X1 of the camera along a horizontal axis from the forks-down image of the rack leg,
generate a forks-up coordinate X2 of the camera along a horizontal axis from a forks-up image of the rack leg captured with the fork carriage assembly at a lift height H1,
determine a mast sway offset as a difference between the forks-down coordinate X1 and the forks-up coordinate X2,
determine a horizontally advanced position of the materials handling vehicle with the fork carriage assembly at the lift height H1 using the mast sway offset and a subsequently captured forks-up image of at least a portion of a rack leg positioned in the racking system aisle, and
update the inventory transit surface position using the horizontally advanced position.

### Claim 15 (depends on 9)

The materials handling vehicle of claim 9, wherein the vehicle position processor is configured to
generate end-of-aisle limit data comprising an end-of-aisle limit from the image of the aisle exit identifier, as captured by the camera,
determine whether the materials handling vehicle exceeds the end-of-aisle limit,
transmit a limit-based command to the drive mechanism of the materials handling vehicle when an operating state of the materials handling vehicle exceeds the end-of-aisle limit,
initiate a vehicle action based on the limit-based command, the vehicle action configured to modify operation of the materials handling vehicle to a modified operating state within the end-of-aisle limit, and
navigate the materials handling vehicle to exit the racking system aisle in the modified operating state.

### Claim 16 (depends on 15)

The materials handling vehicle of claim 15, wherein the end-of-aisle limit comprises a speed limit, a lift height limit, an acceleration limit, a deceleration limit, a lift acceleration limit, a lift deceleration limit, or combinations thereof.

### Claim 17 (depends on 15)

The materials handling vehicle of claim 15, wherein the vehicle position processor is configured to navigate the materials handling vehicle outside of the racking system aisle in the modified operating state, in an additional operating state that is within or outside of the end-of-aisle limit, or in a combination of operating states selected therefrom.

### Claim 18 (depends on 9)

The materials handling vehicle of claim 9, wherein:
the camera is configured to capture an image comprising an aisle entry identifier for a racking system aisle of a multilevel warehouse racking system; and
the vehicle position processor is configured to
generate end-of-aisle limit data comprising an end-of-aisle limit from the image of the aisle entry identifier, as captured by the camera,
determine whether an operating state of the materials handling vehicle exceeds the end-of-aisle limit when the inventory transit surface position of the materials handling vehicle corresponds to an end of the racking system aisle,
transmit a limit-based command to the drive mechanism of the materials handling vehicle when the operating state of the materials handling vehicle exceeds the end-of-aisle limit,
initiate a vehicle action based on the limit-based command, the vehicle action configured to modify operation of the materials handling vehicle to a modified operating state within the end-of-aisle limit, and
navigate the materials handling vehicle to exit the racking system aisle in the modified operating state.

### Claim 19

A materials handling vehicle comprising a camera, a vehicle position processor, and a drive mechanism configured to move the materials handling vehicle along an inventory transit surface, wherein the vehicle position processor is configured to:
generate end-of-aisle data and aisle-specific rack leg spacing data from an image of an aisle entry identifier associated with at least a portion of a rack leg positioned in a racking system aisle associated with at least a portion of a rack leg positioned in a racking system aisle, as captured by the camera,
generate an initial position of the materials handling vehicle along the inventory transit surface using the end-of-aisle data,
generate an expected position of the materials handling vehicle as the vehicle travels down a racking system aisle corresponding to the aisle entry identifier from the end-of-aisle data and the aisle-specific rack leg spacing data,
navigate the materials handling vehicle to exit the racking system aisle, and
update the expected position of the materials handling vehicle using data received from the materials handling vehicle when the materials handling vehicle is outside of the racking system aisle, wherein the data comprises at least data from a tag identifier component, odometry data, steer wheel angle, steer wheel speed, or combinations thereof.

### Claim 20 (depends on 19)

The materials handling vehicle of claim 19, wherein the vehicle position processor is configured to
generate out-of-aisle positional data from a tag identifier, as received by the tag identifier component, and
update the vehicle pose of the materials handling vehicle relative to the tag identifier using the out-of-aisle positional data.