# US 12,001,018 B2 · Device, method and program for improving cooperation between tele-existence and head-mounted display

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,001,018 B2
- Document ID: US12001018B2
- Status: Granted
- Kind: B2
- Application number: 18/259,093
- Filed: 2021-12-24
- Published: 2024-02-15
- Granted: 2024-06-04
- Earliest claimed: 2021-01-05
- Assignees: Sony Group Corporation; Sony Interactive Entertainment Inc.
- Applicants: Sony Group Corporation; Sony Interactive Entertainment Inc.
- Inventors: Yasushi Okumura; Yukari Konishi; Fumihiko Iida; Ryuichi Suzuki; Kotaro Imamura
- Web page: https://patentgrep.com/patent/US12001018B2

## CPC

- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- B25J 5/007: {mounted on wheels}
- B25J 13/00: Controls for manipulators (program controls B25J9/16)
- B25J 19/023: {including video camera means}
- G02B 27/017: {Head mounted}
- G02B 27/0179: {Display position adjusting means not related to the information to be displayed}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/2245: {providing the operator with a purely computer-generated representation of the environment of the vehicle, e.g. virtual reality}
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 5/66: Transforming electric information into light information (scanning details H04N3/00)
- H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G05D 2105/32: {for amusement, e.g. toys}
- G05D 2109/10: Land vehicles
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A control device generates a first image in which, together with an object in the real world appearing in an image captured by a camera of a robot, a virtual object appears and which presents an exterior appearance of the virtual object while the position of the camera of the robot is set as a view-point. An HMD displays the first image generated by the control device. In a case in which the position of the HMD in the height direction changes, the control device generates a second image from which the object in the real world is deleted and which presents the exterior appearance of the virtual object viewed from a new view-point according to the change in the position of the HMD in the height direction. The HMD displays the second image in place of the first image.

## Claims

### Claim 1

An information processing device comprising:
an acquisition unit that acquires an image in which an object in a real world captured by an imaging device appears;
an image generation unit that generates a first image in which a virtual object appears together with the object in the real world appearing in the image acquired by the acquisition unit and which presents an exterior appearance of the virtual object while a position of the imaging device is set as a view-point;
a display control unit that causes a head-mounted display to display the first image generated by the image generation unit; and
a detection unit that detects a change in a position of the head-mounted display in a height direction,
wherein, in a case in which the position of the head-mounted display in the height direction changes,
the image generation unit generates a second image from which the object in the real world is deleted and which presents the exterior appearance of the virtual object viewed from a new view-point according to the change in the position of the head-mounted display in the height direction, and
the display control unit causes the head-mounted display to display the second image in place of the first image.

### Claim 2 (depends on 1)

The information processing device according to claim 1, wherein the image generation unit generates, in a case in which the position of the head-mounted display in the height direction changes from down to up, a second image from which the object in the real world is deleted and which presents the exterior appearance of the virtual object viewed from a view-point at a position higher than the view-point in the first image.

### Claim 3 (depends on 1)

The information processing device according to claim 1, wherein
the imaging device is mounted to the robot, and
the robot is configured such that the position of the imaging device in the height direction does not change beyond a predetermined range.

### Claim 4

An information processing method executed by a computer, comprising:
acquiring an image in which an object in a real world captured by an imaging device appears;
generating a first image in which a virtual object appears together with the object in the real world appearing in the image acquired in the acquisition and which presents an exterior appearance of the virtual object while a position of the imaging device is set as a view-point;
causing a head-mounted display to display the first image generated by the generation;
detecting a change in a position of the head-mounted display in a height direction;
generating, in a case in which the position of the head-mounted display in the height direction changes, a second image from which the object in the real world is deleted and which presents the exterior appearance of the virtual object viewed from a new view-point according to the change in the position of the head-mounted display in the height direction; and
causing the head-mounted display to display the second image in place of the first image.

### Claim 5

A non-transitory, computer readable storage medium containing a computer program, which when executed by a computer, causes the computer to perform information processing method by carrying out actions, comprising:
acquiring an image in which an object in a real world captured by an imaging device appears;
generating a first image in which a virtual object appears together with the object in the real world appearing in the image acquired by the acquisition and which presents an exterior appearance of the virtual object while a position of the imaging device is set as a view-point;
causing a head-mounted display to display the first image generated by the generation; and
detecting a change in a position of the head-mounted display in a height direction,
wherein, in a case in which the position of the head-mounted display in the height direction changes,
the generating includes generating a second image from which the object in the real world is deleted and which presents the exterior appearance of the virtual object viewed from a new view-point according to the change in the position of the head-mounted display in the height direction, and
the causing includes causing the head-mounted display to display the second image in place of the first image.