# US 2024/0202877 A1 · Data processing apparatus, data processing method, and data processing system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202877 A1
- Document ID: US20240202877A1
- Status: Published application
- Kind: A1
- Application number: 18/286,912
- Filed: 2022-04-26
- Published: 2024-06-20
- Earliest claimed: 2021-04-28
- Assignees: Telexistence Inc.
- Applicants: Telexistence Inc.
- Inventors: Jade FOUNTAIN; Yusuke MIZUSHINA
- Web page: https://patentgrep.com/patent/US20240202877A1

## CPC

- G09G 5/377: for mixing or overlaying two or more graphic patterns (G09G5/02, G09G5/397 take precedence)
- B25J 13/06: Control stands, e.g. consoles, switchboards
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 11/60: Creating or editing images; Combining images with text
- H04N 21/4131: {home appliance, e.g. lighting, air conditioning system, metering devices (home automation data switching networks exchanging configuration information on appliance services H04L12/2807)}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/4316: {for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window}
- G06T 2207/20221: Image fusion; Image merging
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A data processing apparatus is disclosed which processes data for operating a movable segment of a machine in correspondence with a position of a controller. The data processing apparatus includes a control unit or processor configured to obtain controller information including the position, when an operator operates the movable segment using the controller, obtain capture image data generated by image capturing of the movable segment while the movable segment is operated in correspondence with the position, generate composite image data by superimposing a controller image indicating the position included in the controller information, on an area corresponding to the movable segment on the capture image data, and display the composite image data on a display device viewable by the operator.

## Claims

### Claim 1

A data processing apparatus for processing data for operating a movable segment of a machine in correspondence with a position of a controller, the data processing apparatus comprising a processor configured to:
obtain controller information including the position, when an operator operates the movable segment using the controller;
obtain capture image data generated by image capturing of the movable segment while the movable segment is operated depending on the position;
generate composite image data by superimposing a controller image indicating the position included in the controller information, on an area corresponding to the movable segment on the capture image data; and
display the composite image data on a display device viewable by the operator.

### Claim 2 (depends on 1)

The data processing apparatus according to claim 1, wherein the processor is configured to obtain the controller information including orientation of the controller in addition to the position; and
generate the composite image data by superimposing the controller image indicating the position and orientation included in the controller information.

### Claim 3 (depends on 2)

The data processing apparatus according to claim 2, wherein the controller image is an image indicating a fingertip or a wrist of the operator holding the controller, corresponding to the orientation included in the controller information.

### Claim 4 (depends on 1)

The data processing apparatus according to claim 1, wherein, in the case where the movable segment moved, the control unit is configured to superimpose the controller image on an area corresponding to the movable segment after movement, on the capture image data.

### Claim 5 (depends on 1)

The data processing apparatus according to claim 1, wherein the processor is configured to associate an image indicating the state where the controller is operated with the controller image, and superimpose the image indicating a state where the controller is operated with the controller image on the capture image data.

### Claim 6 (depends on 1)

The data processing apparatus according to claim 1, wherein the processor is configured to superimpose an image indicating operation to be performed next, for the controller, according to task content scheduled to be performed next by the operator, on the capture image data.

### Claim 7 (depends on 1)

The data processing apparatus according to claim 1, wherein the processor is further configured to identify a difference between an actual position of the movable segment and a computational position of the movable segment calculated based on the position included in the controller information.

### Claim 8 (depends on 7)

The data processing apparatus according to claim 7, wherein the processor is configured to:
obtain the controller information including the orientation of the controller in addition to the position; and
identify a difference between actual position and orientation of the movable segment and computational position and orientation of the movable segment based on the position and the orientation included in the controller information.

### Claim 9 (depends on 7)

The data processing apparatus according to claim 7, wherein the processor is configured to associate the difference with the controller image, and display the difference on the display device.

### Claim 10 (depends on 7)

The data processing apparatus according to claim 7, wherein the processor is configured to further perform operation of the movable segment in correspondence with the position included in the controller information, after making correction corresponding to the difference for the position included in the controller information.

### Claim 11 (depends on 10)

The data processing apparatus according to claim 10, wherein, on condition that the difference is not more than a predetermine value, the processor is configured to operate the movable segment in correspondence with the position included in the controller information.

### Claim 12

A data processing method for processing data for operating a movable segment of a machine in correspondence with a position of a controller, the method comprising:
obtaining controller information including the position, when an operator operates the movable segment using the controller;
obtaining capture image data generated by image capturing of the movable segment while the movable segment is operated depending on the position;
generating composite image data by superimposing a controller image indicating the position included in the controller information, on an area corresponding to the movable segment on the capture image data; and
displaying the composite image data on a display device viewable by the operator.

### Claim 13

A non-transitory computer program product having computing program instructions thereon wherein the computing program instructions when executed by a computer, configure the computer to:
process data for operating a movable segment of a machine in correspondence with a position of a controller;
obtain controller information including the position, when an operator operates the movable segment using the controller;
obtain capture image data generated by image capturing of the movable position while the movable position is operated in correspondence with the position;
generate composite image data by superimposing a controller image indicating the position included in the controller information, on an area corresponding to the movable segment on the capture image data; and
display the composite image data on a display device viewable by the operator.

### Claim 14

A data processing system comprising:
an operation device associated with a controller for operating a machine; and
a data processing apparatus for processing data for controlling a movable segment of the apparatus in correspondence with the position of the controller,
wherein the data processing apparatus has a processor configured to:
obtain controller information including the position, when an operator operates the movable segment using the controller;
obtain capture image data generated by image capturing of the movable segment while the movable segment is operated in correspondence with the position;
generate composite image data by superimposing a controller image indicating the position included in the controller information, on an area corresponding to the movable segment on the capture image data; and
display the composite image data on a display device viewable by the operator,
and wherein the operation controller comprises:
a communication unit configured to transmit the controller information based on contents of operation by the operator, to the data processing apparatus; and
a display unit viewable to the operator, and configured to display the composite image data transmitted by the data processing apparatus.
