# US 12,704,828 B2 · Real-time predictor of human movement in shared workspaces

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,704,828 B2
- Document ID: US12704828B2
- Status: Granted
- Kind: B2
- Application number: 17/953,344
- Filed: 2022-09-27
- Published: 2023-02-16
- Granted: 2026-08-11
- Earliest claimed: 2022-09-27
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Leobardo Emmanuel Campos Macias; David Gomez Gutierrez; Javier Felip Leon; Jose Ignacio Parra Vilchis
- Web page: https://patentgrep.com/patent/US12704828B2

## CPC

- B25J 9/1676: {Avoiding collision or forbidden zones}
- G05B 19/4155: characterised by program execution, i.e. part program or machine function execution, e.g. selection of a program
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1666: {Avoiding collision or forbidden zones}
- B25J 9/1669: {characterised by special application, e.g. multi-arm co-operation, assembly, grasping}
- G05B 2219/39091: Avoid collision with moving obstacles
- G05B 2219/40333: Singularity, at least one movement not possible, kinematic redundancy
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

Disclosed herein are systems, devices, and methods for real-time determinations of likelihoods for possible trajectories of a collaborator in a workspace with a robot. The system determines a current kinematic state of the collaborator and determines a goal of the collaborator based on occupancy information about objects in the workspace. The system also determines a possible trajectory for the collaborator based on the goal and the current kinematic state and determines a short-horizon trajectory for the collaborator based on previously observed kinematic states of the collaborator towards the goal. The system also determines a likelihood that the collaborator will follow the possible trajectory based on the short-horizon trajectory, the goal, and the current kinematic state. The system also generates a movement instruction to control movement of the robot based on the likelihood that the collaborator will follow the possible trajectory.

## Claims

### Claim 1

A device comprising a processor configured to:
determine a current kinematic state of a collaborator in a workspace shared with a robot;
determine a goal of the collaborator based on occupancy information about objects in the workspace;
generate a sequence of expected kinematic positions based on a k-best paths algorithm between the current kinematic state and the goal;
determine a possible trajectory for the collaborator based on the goal, the current kinematic state, and the sequence of expected kinematic positions;
determine a short-horizon trajectory for the collaborator based on previously observed kinematic states of the collaborator towards the goal;
determine a likelihood that the collaborator will follow the possible trajectory based on the short-horizon trajectory, the goal, and the current kinematic state; and
generate a movement instruction to control movement of the robot based on the likelihood that the collaborator will follow the possible trajectory.

### Claim 2 (depends on 1)

The device of claim 1, the device further comprising a transmitter configured to transmit the movement instruction to the robot.

### Claim 3 (depends on 1)

The device of claim 1, the device further comprising a receiver configured to receive collaborator information about the current kinematic state of the collaborator in the workspace or to receive occupancy information about the objects in the workspace.

### Claim 4 (depends on 1)

The device of claim 1, wherein the occupancy information comprises at least one of a location of the object in the workspace, a volume of space occupied by the object in the workspace, and a movement trajectory of the object within the workspace.

### Claim 5 (depends on 1)

The device of claim 1, wherein the processor configured to determine the possible trajectory comprises the processor configured to assign a semantic cost to a region of the workspace based on a historical database of kinematic states of collaborators in the workspace and based on a comparison of the current kinematic state to the historical database of kinematic states.

### Claim 6 (depends on 1)

The device of claim 1, wherein the processor configured to determine the possible trajectory comprises the processor configured to assign a semantic cost to a region of the workspace based on a historical database of object information and based on a comparison of the objects to the historical database of object information.

### Claim 7 (depends on 6)

The device of claim 6, wherein the object information comprises at least one of a type of the objects, a position within the workspace of the objects, a size of the objects, a weight of the objects, and a shape of the objects.

### Claim 8 (depends on 1)

The device of claim 1, wherein the processor is further configured to generate the sequence of expected kinematic states between the current kinematic state and the goal further based on the semantic cost.

### Claim 9 (depends on 1)

The device of claim 1, wherein the current kinematic state comprises at least one of a position of an arm segment of the collaborator in the workspace, a velocity of the arm segment, an acceleration of the arm segment, and a jerk of the arm segment.

### Claim 10 (depends on 9)

The device of claim 9, wherein the possible trajectory comprises a geometric path of the arm segment from the current kinematic state to the expected goal.

### Claim 11 (depends on 10)

The device of claim 10, wherein the possible trajectory is based on a path cost associated with the geometric path, wherein the path cost is based on distances of subsegments along the geometric path.

### Claim 12 (depends on 11)

The device of claim 11, wherein the path cost is based on a semantic cost associated with each corresponding region traversed by each subsegment of the subsegments along the geometric path.

### Claim 13 (depends on 9)

The device of claim 9, wherein the geometric path is based on a set of collision free paths defined by human-like movements of the arm segment.

### Claim 14

An apparatus for predicting arm movements of a human collaborating with a robot in a collaboration workspace, the apparatus comprising
an observation circuit configured to determine a current kinematic state of the human, determine occupancy information about objects in the collaboration workspace, and determine a goal of the human based on the occupancy information and the current kinematic state;
a geometric planner circuit configured to determine a possible trajectory for the human based on the goal and the current kinematic state;
a short-horizon circuit configured to:
model a next kinematic state of the human at a next time interval based on a matrix of coefficients that define how much each one of a set of previously observed kinematic states contributes to the next kinematic state; and
determine a short-horizon trajectory for the human based on the set of previously observed kinematic states of the human towards the goal;
a controller circuit configured to:
determine a likelihood that the human will follow the possible trajectory, wherein the likelihood is based on the short-horizon trajectory, the goal, and the current kinematic state; and
generate a movement instruction to control movement of the robot based on the likelihood.

### Claim 15

A non-transitory computer readable medium, comprising instructions which, if executed, cause one or more processors to:
determine a current kinematic state of a collaborator in a workspace shared with a robot;
determine a goal of the collaborator based on occupancy information about objects in the workspace;
determine a possible trajectory for the collaborator based on the goal and the current kinematic state;
determine a short-horizon trajectory for the collaborator based on previously observed kinematic states of the collaborator towards the goal;
evaluate a multivariate normal distribution centered on each trajectory point of the possible trajectory;
determine a likelihood that the collaborator will follow the possible trajectory based on the short-horizon trajectory, the goal, the current kinematic state, the multivariate normal distribution; and
generate a movement instruction to control movement of the robot based on the likelihood that the collaborator will follow the possible trajectory.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the current kinematic state is associated with a current time and the previously observed kinematic states are associated with previous times before the current time.

### Claim 17 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the instructions are further configured to cause the processor to iteratively redetermine the likelihood at a time interval, to redetermine the current kinematic state at each time interval, to redetermine the estimated goal at each time interval, to redetermine the possible trajectory at each time interval, and to redetermine the short-horizon trajectory at each time interval.