# US 10,371,953 B2 · Image display system and information processing apparatus and control methods thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,371,953 B2
- Document ID: US10371953B2
- Status: Granted
- Kind: B2
- Application number: 15/692,063
- Filed: 2017-08-31
- Published: 2018-03-29
- Granted: 2019-08-06
- Earliest claimed: 2016-09-26
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Norihito Aoki
- Web page: https://patentgrep.com/patent/US10371953B2

## CPC

- G02B 27/0172: {characterised by optical features}
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/017: {Head mounted}
- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06F 3/1407: {General aspects irrespective of display type, e.g. determination of decimal point position, display with fixed or driving decimal point, suppression of non-significant zeros}
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G09G 2330/021: Power management, e.g. power saving
- G09G 2340/0407: Resolution change, inclusive of the use of different resolutions for different screen areas
- G09G 2354/00: Aspects of interface with display user
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

This invention suppresses the narrowing of the communication band between a device and an information processing apparatus that form an image display system. Hence, an HMD of a user includes a sensor that detects a position and orientation, a communication interface that transmits, to the information processing apparatus, position and orientation information representing the detected position and orientation and receives a CG command from the information processing apparatus, a rendering unit that renders a CG based on the received CG command, and a display control unit that displays the rendered CG on a display unit.

## Claims

### Claim 1

An image display system comprising:
a computer graphics (CG) displaying device including a display device that displays an image to a user; and
an information processing apparatus that transmits information for displaying CG on the display device,
wherein the displaying device comprises:
an image capturing device configured to capture an image of a physical space; and
a position and orientation detecting sensor that detects a position and orientation of the displaying device;
a first communication interface that communicates with the information processing apparatus;
at least one first processor or circuit configured to implement first instructions to execute a first plurality of tasks, including:
a first transmitting task that transmits, via the first communication interface, to the information processing apparatus, position and orientation information related to the position and orientation detected by the position and orientation detecting sensor and specific information for specifying a frame captured by the image capturing device;
a first receiving task that receives a CG command from the information processing apparatus;
a rendering task that renders CG based on the CG command received by the first receiving task;
a synthesizing task that synthesizes the CG on to the captured image of the physical space in a case where the specific information of the CG rendered by the rendering task and the specific information of a synthesizing target image match; and
a display control task that controls to the display device to display the CG rendered by the rendering task and an image synthesized by the synthesizing task, and
wherein the information processing apparatus comprises:
a second communication interface for communicating with the displaying device;
at least one second processor or circuit configured to implement second instructions to execute a second plurality of tasks, including:
a second receiving task that receives the position and orientation information and the specific information from the displaying device;
a CG command generating task that generates the CG command based on the position and orientation information and the specific information received from the displaying device; and
a second transmitting task that transmits the generated CG command to the displaying device via the second communication interface.

### Claim 2 (depends on 1)

The system according to claim 1, wherein:
the image capturing device captures the image of the physical space by a predetermined frame rate,
the specific information for specifying the frame captured by the image capturing device specifies an image indicated thereby, and
the first receiving task receives, from the information processing apparatus, the CG command together with the specific information.

### Claim 3 (depends on 1)

The system according to claim 1, wherein:
the displaying device further comprises a sight-line detector that detects a sight-line direction of the user, and
the first plurality of tasks include a modifying task that modifies, based on the sight-line direction detected by the sight-line detector, the synthesized image synthesized by the synthesizing task.

### Claim 4 (depends on 3)

The system according to claim 3, wherein the modifying task blurs a region that exceeds a predetermined range from a position indicated by the sight-line direction in the synthesized image.

### Claim 5 (depends on 4)

The system according to claim 4, wherein the display control task controls the display device to display an image modified by the modifying task.

### Claim 6 (depends on 3)

The system according to claim 3, wherein:
the first transmitting task transmits, to the information processing apparatus, sight-line information representing the sight-line direction detected by the sight-line detector, and
the CG command generating task generates the CG command that is:
a first CG command in a case where a region is within a predetermined range from a position indicated by the sight-line information; and
a second CG command in which rendering is coarser than that of the first CG command in a case where the region is outside the predetermined range.

### Claim 7 (depends on 3)

The system according to claim 3, wherein:
the first plurality of tasks include an encoding task that encodes the modified synthesized image that has been modified by the modification task,
the first transmitting task transmits encoded data obtained by the encoding task to the information processing apparatus,
the second receiving task receives the encoded data,
the second plurality of tasks include a decoding task that decodes the encoded data received by the second receiving task.

### Claim 8

A control method for an image display system comprising a computer graphics (CG) displaying device that includes an image capturing device that captures an image of a physical space and a display device that displays an image to a user, and an information processing apparatus that transmits information for displaying CG on the display device, the method comprising:
a detecting step of detecting, with the displaying device, via a sensor provided in the displaying device, a position and orientation of the displaying device;
a first transmitting step of transmitting, to the information processing apparatus, with the displaying device, via a first communication interface provided in the displaying device, position and orientation information related to the position and orientation detected in the detecting step and specific information for specifying a frame captured by the image capturing device;
a first receiving step of receiving, with the displaying device, via the first communication interface, a CG command from the information processing apparatus;
a rendering step of rendering, with the displaying device, CG based on the CG command received in the receiving step;
a synthesizing step of synthesizing the CG on to the captured image of the physical space in a case where the specific information of the CG rendered in the rendering step and the specific information of a synthesizing target image match;
a displaying step of displaying, with the displaying device, the rendered CG rendered in the rendering step and an image synthesized in the synthesizing step;
a second receiving step of receiving, from the displaying device, with the information processing apparatus, via a second communication interface provided in the image processing apparatus, the position and orientation information and the specific information;
a CG command generating step of generating, with the information processing apparatus, the CG command based on the position and orientation information and the specific information received in the second receiving step; and
a second transmitting step of transmitting, with the information processing apparatus, via the second communication interface, the generated CG command to the displaying device.

### Claim 9

A non-transitory computer-readable storage medium storing a program executable by at least one processor, of an image display system comprising a computer graphics (CG) displaying device that includes an image capturing device that captures an image of a physical space and a display device that displays an image to a user, and an information processing apparatus that transmits information for displaying CG on the display device, to execute a method comprising:
a detecting step of detecting, with the displaying device, via a sensor provided in the displaying device, a position and orientation of the displaying device;
a first transmitting step of transmitting, to the information processing apparatus, with the displaying device, via a first communication interface provided in the displaying device, position and orientation information related to the position and orientation detected in the detecting step and specific information for specifying a frame captured by the image capturing device;
a first receiving step of receiving, with the displaying device, via the first communication interface, a CG command from the information processing apparatus;
a rendering step of rendering, with the displaying device, CG based on the CG command received in the receiving step;
a synthesizing step of synthesizing the CG on to the captured image of the physical space in a case where the specific information of the CG rendered in the rendering step and the specific information of a synthesizing target image match;
a displaying step of displaying, with the displaying device, the rendered CG rendered in the rendering step and an image synthesized in the synthesizing step;
a second receiving step of receiving, from the displaying device, with the information processing apparatus, via a second communication interface provided in the image processing apparatus, the position and orientation information and the specific information;
a CG command generating step of generating, with the information processing apparatus, the CG command based on the position and orientation information and the specific information received in the receiving step; and
a second transmitting step of transmitting, with the information processing apparatus, via the second communication interface, the generated CG command to the displaying device via the communication interface.