# US 2024/0205513 A1 · Video display system, information processing device, information processing method, and recording medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0205513 A1
- Document ID: US20240205513A1
- Status: Published application
- Kind: A1
- Application number: 18/286,354
- Filed: 2022-04-18
- Published: 2024-06-20
- Earliest claimed: 2021-04-16
- Assignees: Panasonic Intellectual Property Management Co., Lid.
- Applicants: Panasonic Intellectual Property Management Co., Lid.
- Inventors: Masayuki KOZUKA; Yoshihiro MORI; Junya SUZUKI; Yoshilchiro KASHIWAGI; Yasutoshi YAMAMOTO; Toshiro NISHIO
- Web page: https://patentgrep.com/patent/US20240205513A1

## CPC

- G09G 5/377: for mixing or overlaying two or more graphic patterns (G09G5/02, G09G5/397 take precedence)
- G02B 27/0172: {characterised by optical features}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06Q 10/40: Business processes related to social networking or social networking services
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06Q 50/10: Services
- G06Q 90/20: {Destination assistance within a business structure or complex}
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 15/00: Three-dimensional [3D] image rendering
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G09G 5/36: characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
- H04N 7/183: {for receiving images from a single remote source}
- H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras}
- H04N 21/2353: {specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata}
- H04N 21/4312: {involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations}
- H04N 21/4316: {for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window}
- H04N 21/439: Processing of audio elementary streams
- H04N 21/472: End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content {(end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612)}
- H04N 21/4728: for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
- H04N 21/4788: communicating with other users, e.g. chatting {(arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104)}
- H04N 21/8106: {involving special audio data, e.g. different tracks for different languages}
- H04N 21/816: {involving special video data, e.g 3D video}
- G06T 2207/20221: Image fusion; Image merging
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A video display system includes: an observation device including a capturing unit that generates a wide-angle video, and a data obtainment unit that obtains data related to a position and/or direction of a target in the wide-angle video to be gazed at by a user of a display device and cue information for informing of a change in the state of the observation system; and a VR device including a reception unit that receives the data and the cue information, and a differential calculation that calculates a relative position and/or direction of the gazing target relative to the position and/or direction of the display device in the wide-angle video, based on a difference between the position and/or direction of the display device in the wide-angle video and the position and/or direction of the gazing target in metadata.

## Claims

### Claim 1

A video display system for displaying a display video via a display device, the video display system comprising:
an observation device including:
a capturing unit configured to generate a wide-angle video;
a data obtainment unit configured to obtain (i) data related to at least one of a position or a direction of a gazing target in the wide-angle video to be gazed at by a user of the display device and (ii) cue information for informing the user of a change in a state of an observation system;
a metadata composition unit configured to compose metadata from the data from the data obtainment unit and other information; and
a transmission unit configured to transmit the wide-angle video together with the metadata; and
a VR device including:
a reception unit configured to receive the wide-angle video, the data, and the cue information;
a display state estimation unit configured to estimate at least one of a position or a direction of the display device in the wide-angle video;
a differential calculation unit configured to calculate, based on a difference between (i) at least one of the position or the direction of the display device in the wide-angle video as estimated and (ii) at least one of the position or the direction of the gazing target in the metadata, at least one of a relative position or a relative direction of the gazing target, the relative position being relative to the position of the display device in the wide-angle video, the relative direction being relative to the direction of the display device in the wide-angle video;
a presentation unit configured to present, to the user of the display device, (i) information on at least one of the relative position calculated or the relative direction calculated, (ii) an instruction in accordance with the cue information, and (iii) a state of the observation system;
a video generation unit configured to generate, from the wide-angle video received, the display video including a portion corresponding to a viewing area corresponding to (i) information on at least one of the position or the direction of the display device in the wide-angle video as estimated from the wide-angle video by the display state estimation unit, (ii) the instruction in accordance with the cue information, and (iii) the state of the observation system; and
the display device that displays the display video.

### Claim 2 (depends on 1)

The video display system according to claim 1, further comprising:
a camera that captures a video or an image generation unit configured to generate an image by calculation, wherein
the wide-angle video is a video captured by the camera or an image calculated by the image generation unit.

### Claim 3 (depends on 1)

The video display system according to claim 1, wherein
the presentation unit is configured to:
generate and output a graphic indicating information that is based on the cue information and at least one of the relative position calculated or the relative direction calculated; and
cause the video generation unit to superimpose the graphic onto the portion to present at least one of the relative position or the relative direction.

### Claim 4 (depends on 3)

The video display system according to claim 3, wherein
the data obtainment unit is configured to receive an input of data related to the direction of the gazing target,
the display state estimation unit is configured to estimate the direction of the display device in the wide-angle video, and
the graphic displays an arrow indicating the relative direction on the display video.

### Claim 5 (depends on 3)

The video display system according to claim 3, wherein
the data obtainment unit is configured to receive an input of data related to the direction of the gazing target,
the display state estimation unit is configured to estimate the direction of the display device in the wide-angle video, and
the graphic displays, as a mask, an image for covering at least part of the display video other than a part corresponding to the relative direction.

### Claim 6 (depends on 3)

The video display system according to claim 3, wherein
the data obtainment unit is configured to receive an input of data related to the position of the gazing target,
the display state estimation unit is configured to estimate the position of the display device in the wide-angle video, and
the graphic displays a map indicating the relative position on the display video.

### Claim 7 (depends on 1)

The video display system according to claim 1, further comprising:
an input interface for inputting the data, wherein
the data obtainment unit is configured to obtain the data input via the input interface.

### Claim 8 (depends on 3)

The video display system according to claim 3, further comprising:
an input interface for specifying a timing of at least one of a start or an end of movement by the user in the wide-angle video, wherein
the data obtainment unit is configured to obtain the timing of at least one of the start or the end of the movement that has been input via the input interface.

### Claim 9 (depends on 7)

The video display system according to claim 7, wherein
images included in the wide-angle video are images output by a capturing unit configured to capture a real space, and
the input interface includes: an indication marker that is held by an operator of the input interface in the real space and indicates at least one of the position or the direction of the gazing target according to the movement of the indication marker; and an image analysis unit configured to receive at least one of the position or the direction of the gazing target indicated by the indication marker by analyzing an image including the indication marker output by the capturing unit.

### Claim 10 (depends on 1)

The video display system according to claim 1, further comprising:
an information processing device that includes at least one of functions included in the observation device and the VR device, is connected to the observation device and the VR device over a network, and performs one or more processes of the observation device or the VR device.

### Claim 11 (depends on 10)

The video display system according to claim 10, wherein
the information processing device includes:
a reception unit configured to receive, as the metadata from the observation device, the wide-angle video, the data, and the cue information;
a presentation unit configured to generate information for presenting, to the user of the display device, information in accordance with the cue information and at least one of the position or the direction of the gazing target in the metadata;
a video generation unit configured to generate the display video from the wide-angle video received by adding the information generated by the presentation unit to a portion of the wide-angle video that corresponds to a viewing area corresponding to information on at least one of the position or the direction of the display device in the wide-angle video estimated by the display state estimation unit; and
a transmission unit configured to transmit: the wide-angle video; a portion of the wide-angle video that corresponds to a viewing area corresponding to information on at least one of the relative position or the relative direction; and the metadata.

### Claim 12 (depends on 10)

The video display system according to claim 10, wherein
the information processing device includes:
a reception unit configured to receive, as the metadata from the observation device, the wide-angle video, the data, and the cue information;
a presentation unit configured to generate information for presenting, to the user of the display device, information in accordance with the cue information and at least one of the position or the direction of the gazing target in the metadata;
a metadata composition unit configured to generate the metadata from the information generated by the presentation unit; and
a transmission unit configured to transmit, to the VR device, the metadata generated by the metadata composition unit, the wide-angle video received by the reception unit, and other information.

### Claim 13 (depends on 10)

The video display system according to claim 10, wherein
the information processing device includes:
a reception unit configured to receive, as the metadata from the observation device, the wide-angle video, the data, and the cue information, and receive, from the display device, data related to an orientation of the display device;
a differential calculation unit configured to calculate a relative movement direction based on the cue information and a difference between the orientation of the display device and movement information related to movement of the capturing unit, the relative movement direction being a movement direction of the capturing unit relative to the orientation of the display device;
a presentation unit configured to generate and output a graphic that indicates the relative movement direction calculated and presents, to the user of the display device, information in accordance with the cue information and the relative movement direction by being superimposed on a portion of the wide-angle video that corresponds to a viewing area corresponding to the orientation of the display device estimated;
a video generation unit configured to generate the display video by correcting the graphic based on data related to the orientation of the display device and superimposing the graphic corrected on the wide-angle video; and
a transmission unit configured to transmit the display video and other information.

### Claim 14 (depends on 10)

The video display system according to claim 10, wherein
the information processing device is provided in a cloud connected to a wide-area network, and is connected to the observation device and the VR device over the wide-area network.

### Claim 15 (depends on 1)

The video display system according to claim 1, wherein
the cue information is information indicating that a movement direction of the observation device or at least one of the position or the direction of the gazing target to be gazed at by the user of the display device changes.

### Claim 16

An information processing device used in a video display system for displaying a display video on a display device, the display video being at least part of a wide-angle video, the information processing device comprising:
a reception unit configured to receive metadata that is based on input data related to at least one of a position or a direction of a gazing target to be gazed at by a user of the display device in the wide-angle video; and
a differential calculation unit configured to calculate and output at least one of a relative position or a relative direction of the gazing target, based on a difference pertaining to at least one of the position or the direction of the gazing target in the metadata, the relative position being relative to a position of the display device in the wide-angle video, the relative direction being relative to a direction of the display device in the wide-angle video.

### Claim 17 (depends on 16)

The information processing device according to claim 16, further comprising:
a presentation unit configured to generate and output a graphic that indicates at least one of the relative position or the relative direction calculated and presents, to the user of the display device, at least one of the relative position or the relative direction by being superimposed on a portion of the wide-angle video that corresponds to a viewing area corresponding to at least one of the position or the direction of the display device in the wide-angle video as estimated.

### Claim 18

An information processing method for displaying a display video on a display device, the display video being at least part of a wide-angle video, the method comprising:
receiving metadata based on input data related to at least one of a position or a direction of a gazing target to be gazed at by a user of the display device in the wide-angle video; and
calculating and outputting, based on a difference between (i) at least one of a position or a direction of the display device in the wide-angle video as estimated and (ii) at least one of the position or the direction of the gazing target in the metadata, at least one of a relative position or a relative direction of the gazing target relative to an orientation of the display device.

### Claim 19 (depends on 18)

A non-transitory computer-readable recording medium having recorded thereon a computer program for causing a computer to execute the information processing method according to claim 18.
