# US 2024/0231304 A1 · Operation management device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0231304 A1
- Document ID: US20240231304A1
- Status: Published application
- Kind: A1
- Application number: 18/286,280
- Filed: 2021-05-07
- Published: 2024-07-11
- Earliest claimed: 2021-05-07
- Assignees: Fanuc Corporation
- Applicants: Fanuc Corporation
- Inventors: Tomofumi UWANO
- Web page: https://patentgrep.com/patent/US20240231304A1

## CPC

- G05B 19/402: characterised by control arrangements for positioning, e.g. centring a tool relative to a hole in the workpiece, additional detection means to correct position (G05B19/19 takes precedence)
- G06Q 50/04: Manufacturing
- G06Q 10/06: Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G05B 2219/50047: Positioning, indexing
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

An operation management device includes a machine tool detection unit which detects industrial machines from videos from cameras based on registered feature points of the industrial machines, a varying point detection unit which detects, from the videos, time-varying feature points of the detected industrial machines and change times, and an individual machine recognition unit which recognizes the individual industrial machines based on the time-varying feature points and change times. The operation management device also includes a camera parameter calculation unit which calculates camera parameters of each camera, a machine distance calculation unit which calculates the distances between the feature points and the cameras based on the camera parameters, and a layout diagram creation unit which creates a layout diagram of icons of the industrial machines from the calculated distances and locations and three-dimensional coordinates of the center of the lens of each camera.

## Claims

### Claim 1

An operation management device that displays an icon representing at least one industrial machine as arranged on a layout diagram representing an outline of a facility where the industrial machine is installed, the operation management device comprising:
a machine tool detection unit configured to detect the industrial machine installed in the facility, from a video from each of a plurality of cameras installed in the facility, based on a plurality of feature points of each industrial machine registered in advance in a machine shape storage unit;
a point-of-variation detection unit configured to detect, from among the plurality of feature points of the industrial machine detected, a time-varying feature point and a time when the feature point varied over time, from a video from each of the plurality of cameras;
an identical machine recognition unit configured to recognize one or more industrial machines in a video from each of the plurality of cameras, based on the time-varying feature point and the time when the feature point varied over time, detected by the point-of-variation detection unit;
a camera parameter calculation unit configured to calculate a camera parameter for each of the plurality of cameras, based on a distance between at least one pair of feature points registered in advance in the machine shape storage unit, and a distance between the feature points in a video of the industrial machines recognized by the identical machine recognition unit;
a machine distance calculation unit configured to calculate a distance between each feature point of the industrial machines recognized by the identical machine recognition unit and a center of a lens of each of the plurality of cameras, based on the camera parameter for each of the plurality of cameras calculated by the camera parameter calculation unit; and
a layout diagram creation unit configured to create the layout diagram arranged with icons of the industrial machines recognized, based on the distance between each feature point of the industrial machines and the center of the lens of each of the plurality of cameras calculated by the machine distance calculation unit, and three-dimensional coordinates of the center of the lens of each of the plurality of cameras registered in advance.

### Claim 2 (depends on 1)

The operation management device according to claim 1, further comprising:
an operation information acquisition unit configured to acquire operation information on the industrial machine from a control device that controls the industrial machine; and
an operation information machine identification unit configured to identify which one of the industrial machines recognized by the identical machine recognition unit the operation information acquired by the operation information acquisition unit pertains to, based on a variation over time indicated by the operation information acquired by the operation information acquisition unit, and the time-varying feature point and the time when the feature point varied over time, detected by the point-of-variation detection unit.

### Claim 3 (depends on 2)

The operation management device according to claim 2, further comprising:
a machine front zone coordinate calculation unit configured to calculate a front zone of the industrial machine in coordinates in an image coordinate system of a video captured by each of the plurality of cameras, based on a distance between the feature points of the industrial machine and each of the plurality of cameras calculated by the machine distance calculation unit, and the camera parameter for each of the plurality of cameras calculated by the camera parameter calculation unit; and
a machine front zone person detection unit configured to detect whether a person is included in the front zone of the industrial machine calculated by the machine front zone coordinate calculation unit, based on the video from the cameras that are in a positional relationship allowing for detection with a predetermined degree of accuracy, among the plurality of cameras,
wherein the layout diagram creation unit arranges, on the layout diagram, the operation information of the industrial machines identified by the operation information machine identification unit and presence or absence of a person in the front zone of the industrial machine detected by the machine front zone person detection unit, along with the icons of the industrial machines, in real time, based on a distance between each feature point of the industrial machines and the center of the lens of each of the plurality of cameras, calculated by the machine distance calculation unit.

### Claim 4 (depends on 3)

The operation management device according to claim 3, further comprising:
an augmented reality display request gesture detection unit configured to detect a gesture of a person requesting display of augmented reality which overlays virtual on real, based on a video of the person detected by the machine front zone person detection unit; and
an augmented reality information transmission unit configured to transmit augmented reality image data, which includes the operation information of the industrial machine having the front zone where the person performing the gesture detected by the augmented reality display request gesture detection unit exists, to an external device.
