# US 12,217,649 B2 · Display device, display control system, and method for controlling display device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,649 B2 - Document ID: US12217649B2 - Status: Granted - Kind: B2 - Application number: 17/700,580 - Filed: 2022-03-22 - Published: 2022-09-29 - Granted: 2025-02-04 - Earliest claimed: 2021-03-24 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Masato Uchihara - Web page: https://patentgrep.com/patent/US12217649B2 ## CPC - G09G 3/2092: {Details of a display terminals using a flat panel, the details relating to the control arrangement of the display terminal and to the interfaces thereto (suitable for both CRT and flat panel G09G5/003; specific for a CRT G09G1/165)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay} - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06F 3/147: using display panels - G09G 2340/0407: Resolution change, inclusive of the use of different resolutions for different screen areas - G09G 2340/045: Zooming at least part of an image, i.e. enlarging it or shrinking it - G09G 2340/0464: Positioning - G09G 2354/00: Aspects of interface with display user - G09G 2360/04: Display device controller operating with a plurality of display units - G09G 2370/022: Centralised management of display operation, e.g. in a server instead of locally - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A head-mounted display includes: an imaging device; and at least one processor and/or at least one circuit which function as: a detecting unit configured to detect a target subject in images captured by the imaging device; an acquiring unit configured to acquire an image capturing the target subject from an external apparatus; and a display control unit configured to display an image acquired from the external apparatus based on the detection result of the detecting unit. ## Claims ### Claim 1 A display device comprising: an imaging device; and at least one processor and/or at least one circuit, the at least one processor and/or at least one circuit being configured to act as a plurality of units comprising: (1) a detecting unit that detects a target subject in images captured by the imaging device; (2) an acquiring unit that acquires, from an external apparatus, an image capturing the target subject; and (3) a display control unit that controls display, on an image of the images captured by the imaging device, of the image acquired from the external apparatus, wherein in a case where the target subject is detected but the target subject facing a predetermined direction is not detected by the detecting unit, (1) the acquiring unit acquires the image capturing the target subject facing the predetermined direction and (2) the display control unit controls display of the image capturing the target subject facing the predetermined direction, and wherein the display control unit (1) controls display of the image acquired by the acquiring unit in a case where a size of the target subject in the image captured by the imaging device becomes smaller than a predetermined size, (2) determines a size of an area to display the image acquired by the acquiring unit, based on the size of the target subject in the image captured by the imaging device, so that the image acquired by the acquiring unit does not overlap with the target subject, and (3) controls the image captured by the imaging device and the image acquired by the acquiring unit to be displayed together. ### Claim 2 (depends on 1) The display device according to claim 1, wherein the display control unit controls display of an image capturing a larger front face of the target subject with priority out of the images capturing the target subject. ### Claim 3 (depends on 1) The display device according to claim 1, wherein in a case where the target subject is not captured in the images captured by the imaging device, the display control unit controls display of the image acquired by the acquiring unit. ### Claim 4 (depends on 1) The display device according to claim 1, wherein in a case where the front face of the target subject is no longer detected by the detecting unit, the display control unit controls display of the image acquired by the acquiring unit. ### Claim 5 (depends on 1) The display device according to claim 1, wherein in a case where the size of the target subject in the image captured by the imaging device becomes smaller than a predetermined size, the display control unit either (a) controls display of the image acquired by the acquiring unit or (b) magnifies an image of a region capturing the target subject cut out from the image captured by the imaging device, and controls display of the magnified image. ### Claim 6 (depends on 1) The display device according to claim 1, wherein the display control unit (1) controls display of the image acquired from the external apparatus in a case where a size of the target subject in the image captured by the imaging device becomes smaller than a predetermined size, (2) determines the display position of the image acquired by the acquiring unit, based on the position of the target subject in the image captured by the imaging device, so that the image acquired by the acquiring unit does not overlap with the target subject, and (3) controls the image captured by the imaging device and the image acquired by the acquiring unit to be displayed together. ### Claim 7 (depends on 1) The display device according to claim 1, wherein the target subject is set by a user in advance. ### Claim 8 (depends on 1) The display device according to claim 1, wherein the detecting unit changes the target subject based on the line-of-sight position of a user. ### Claim 9 (depends on 1) The display device according to claim 1, wherein the display control unit controls display of a virtual monitor within an image capturing range of the imaging device, and controls display of the image acquired by the acquiring unit on the virtual monitor, and wherein the image displayed on the virtual monitor includes the target subject. ### Claim 10 (depends on 1) The display device according to claim 1, wherein the at least one processor and/or the at least one circuit further act as an estimating unit that estimates a missed period when a user missed the target subject based on the detection result of the target subject and line-of-sight moving information of the user, and wherein the display control unit generates a reproduced video of the target subject during a period including the missed period, using images captured by the imaging device and an image captured during the missed period among the images acquired from the external apparatus. ### Claim 11 (depends on 1) The display device according to claim 1, wherein the at least one processor and/or the at least one circuit further act as a transmitting control unit that performs control to transmit an image captured by the imaging device to the external apparatus. ### Claim 12 (depends on 1) The display device according to claim 1, wherein the at least one processor and/or the at least one circuit are further configured to act as an estimating unit that estimates a missed period when a user missed the target subject, and wherein the display control unit generates a reproduced video of the target subject during a period including the missed period, using images captured by the imaging device while the target subject is detected and an image captured during the missed period among images acquired from the external apparatus. ### Claim 13 (depends on 1) The display device according to claim 1, wherein the acquiring unit acquires the image capturing the target subject from the external apparatus in real-time, and wherein the display control unit controls display of the image acquired from the external apparatus in real-time. ### Claim 14 (depends on 1) The display device according to claim 1, wherein the predetermined direction is a front direction. ### Claim 15 (depends on 1) The display device according to claim 1, wherein the display control unit controls the image captured by the imaging device and the image acquired from the external apparatus to be displayed together. ### Claim 16 (depends on 1) A display control system including: an information processing apparatus; and the display device according to claim 1, wherein the information processing apparatus comprises at least one processor and/or at least one circuit, the at least one processor and/or at least one circuit being configured to act as a plurality of units comprising: (1) a receiving control unit that performs control to receive images captured by the imaging device from a plurality of the display devices; (2) an acquiring unit that detects a subject from an image received by the receiving control unit and acquires information on the target subject including orientation and size of the target subject; and (3) a transmitting control unit that performs control to transmit, to the display device, an image selected based on the information on the target subject, out of the images in which the target subject is detected in the display device. ### Claim 17 A method for controlling a display device, the method comprising: capturing an image; detecting a target subject in images captured in the capturing; acquiring an image capturing the target subject from an external apparatus; and displaying, on the image captured in the capturing, the image acquired from the external apparatus, wherein in a case where the target subject is detected but the target subject facing a predetermined direction is not detected in the detecting, (1) the image capturing the target subject facing the predetermined direction is acquired in the acquiring, and (2) display of the image capturing the target subject facing the predetermined direction is controlled in the displaying, and wherein, in the displaying, (1) display of the image acquired in the acquiring is controlled in a case where a size of the target subject in the image captured by the imaging device becomes smaller than a predetermined size, (2) a size of an area to display the image acquired in the acquiring is determined based on the size of the target subject in the image captured by the imaging device, so that the image acquired in the acquiring does not overlap with the target subject, and (3) the image captured by the imaging device and the image acquired in the acquiring are controlled to be displayed together. ### Claim 18 A non-transitory computer-readable medium that stores a program for causing a computer to execute a method for controlling a display device, the method comprising: capturing an image; detecting a target subject in images captured in the capturing; acquiring an image capturing the target subject from an external apparatus; and displaying, on the image captured in the capturing, the image acquired from the external apparatus, wherein in a case where the target subject is detected but the target subject facing a predetermined direction is not detected in the detecting, (1) the image capturing the target subject facing the predetermined direction is acquired in the acquiring, and (2) display of the image capturing the target subject facing the predetermined direction is controlled in the displaying, wherein, in the displaying, (1) display of the image acquired in the acquiring is controlled in a case where a size of the target subject in the image captured by the imaging device becomes smaller than a predetermined size, (2) a size of an area to display the image acquired in the acquiring is determined based on the size of the target subject in the image captured by the imaging device, so that the image acquired in the acquiring does not overlap with the target subject, and (3) the image captured by the imaging device and the image acquired in the acquiring are controlled to be displayed together.