# US 2022/0044482 A1 · Augmented reality display device and augmented reality display method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0044482 A1 - Document ID: US20220044482A1 - Status: Published application - Kind: A1 - Application number: 17/298,688 - Filed: 2018-12-03 - Published: 2022-02-10 - Earliest claimed: 2018-12-03 - Assignees: Maxell, Ltd. - Applicants: Maxell, Ltd. - Inventors: Osamu KAWAMAE; Masuo OKU - Web page: https://patentgrep.com/patent/US20220044482A1 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06V 10/147: Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19) - G06V 20/20: in augmented reality scenes - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An augmented reality (AR) display device includes a camera that captures a background image; a distance measuring sensor that measures a distance to a real object in the background image; a position and orientation sensor that detects the position of the camera and the shooting direction of the camera; a controller that recognizes the real object from the background image captured by the camera and associates the predetermined AR object with the recognized real object; a display displaying an image of the associated AR object; and a memory that stores data of the real object and the AR object associated with each other. The controller determines whether or not the real object is movable from the measurement result of the distance measuring sensor, and arranges the AR object according to the current position of the real object when the position of the real object associated with the AR object moves. ## Claims ### Claim 1 An augmented reality display device that is worn on a head of a user and displays an augmented reality object (hereinafter, referred to as an AR object) so as to overlap a background image visually recognized by the user, comprising: a camera that captures the background image; a distance measuring sensor that measures a distance to a real object in the background image; a position and orientation sensor that detects a position and a shooting direction of the camera; a controller that recognizes the real object from the background image captured by the camera and associates the predetermined AR object with the recognized real object; a display that displays an image of the associated AR object; and a memory that stores data of the real object and the AR object associated with each other, wherein the controller determines whether or not the real object is movable from a measurement result of the distance measuring sensor, and arranges the AR object according to a current position of the real object when a position of the real object associated with the AR object moves. ### Claim 2 (depends on 1) The augmented reality display device according to claim 1, wherein the controller obtains a movement history of the augmented reality display device based on a measurement result of the position and orientation sensor and assigns a movement flag, which indicates whether or not the real object is movable, to the real object in consideration of the movement history. ### Claim 3 (depends on 1) The augmented reality display device according to claim 1, wherein the controller sets a display position, a size, and a direction of the AR object with respect to the real object as display parameters of the AR object arranged on the real object. ### Claim 4 (depends on 1) The augmented reality display device according to claim 1, wherein, when the AR object to be displayed is arranged behind the real object close to the AR object, the controller causes a hidden portion of the AR object not to be displayed. ### Claim 5 (depends on 1) The augmented reality display device according to claim 1, wherein the controller groups a plurality of the real objects and stores the grouped real objects in the memory so as to be associated with the common AR object, and when one of the grouped real objects is not present in the current background image, the AR object is arranged on another real object that has been grouped. ### Claim 6 (depends on 1) The augmented reality display device according to claim 1, wherein, when recognizing the real object from the background image, the controller stores, in the memory, a region having a unique object shape as a captured object and a region that does not have a unique object shape other than a plane and has a farthest distance or a region whose distance is not measurable as a background object. ### Claim 7 (depends on 1) The augmented reality display device according to claim 1, wherein the display is a 3D projector that projects an image of the AR object onto a transmissive screen, and the user is able to visually recognize the background image through the transmissive screen and at the same time is able to visually recognize the AR object on the transmissive screen. ### Claim 8 (depends on 1) The augmented reality display device according to claim 1, wherein the camera and the distance measuring sensor are 3D cameras capable of capturing the background image and measuring a distance to the real object in the background image, and the display is a flat display that performs display so that an image of the AR object is combined with the background image captured by the 3D camera. ### Claim 9 An augmented reality display method for displaying an augmented reality object (hereinafter, referred to as an AR object) so as to overlap a background image visually recognized by a user, comprising: a step of capturing the background image and recognizing a real object from the captured background image; a step of measuring a distance to the real object in the background image; a step of detecting a shooting position and a shooting direction of the background image; a step of associating the predetermined AR object with the recognized real object; a step of recognizing that the object is a movable object by detecting that the real object has moved; and a step of displaying an image of the associated AR object, wherein, in the step of displaying the image of the AR object, when a position of the real object associated with the AR object moves, the AR object is arranged according to a current position of the real object. ### Claim 10 (depends on 9) The augmented reality display method according to claim 9, wherein the step of associating the AR object with the real object includes a first step in which the real object is selected first and then the AR object to be associated with the real object is selected and a second step in which the AR object is selected first and then the real object to be associated with the AR object is selected, and the first step and the second step are selectable by the user.