# US 10,373,389 B2 · Mobile terminal and method of operating thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,373,389 B2 - Document ID: US10373389B2 - Status: Granted - Kind: B2 - Application number: 15/689,764 - Filed: 2017-08-29 - Published: 2018-03-01 - Granted: 2019-08-06 - Earliest claimed: 2016-08-30 - Assignees: LG Electronics Inc. - Applicants: LG Electronics Inc. - Inventors: Jaeduck Jung - Web page: https://patentgrep.com/patent/US10373389B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - A47L 9/009: {Carrying-vehicles; Arrangements of trollies or wheels; Means for avoiding mechanical obstacles} - A47L 9/2805: {Parameters or conditions being sensed} - A47L 9/2852: {Elements for displacement of the vacuum cleaner or the accessories therefor, e.g. wheels, casters or nozzles} - A47L 9/2857: {User input or output elements for control, e.g. buttons, switches or displays} - A47L 9/2894: {Details related to signal transmission in suction cleaners} - G05D 1/0044: {by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps (maps used for automatic navigation G05D1/0274; flight directors G01C23/005)} - G06V 20/20: in augmented reality scenes - A47L 2201/04: Automatic control of the travelling movement; Automatic obstacle detection - G06T 2210/04: Architectural design, interior design - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A47L: DOMESTIC WASHING OR CLEANING (brushes A46B; cleaning quantities of bottles or of other hollow articles of one kind B08B9/00; laundry D06F); SUCTION CLEANERS IN GENERAL (cleaning in general B08) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A mobile terminal includes a camera, a display configured to display a real image projected by the camera, and a controller configured to recognize the real image to request a traveling path corresponding to the real image from a robot cleaner, to generate an augmented reality (AR) image of the traveling path based on the traveling path received from the robot cleaner, and to display the AR image to be superimposed on the real image. ## Claims ### Claim 1 A mobile terminal comprising: a camera; a display configured to display a real image projected by the camera; and a controller configured to recognize the real image to request a traveling path corresponding to the real image from a robot cleaner, to generate an augmented reality (AR) image of the traveling path based on the traveling path received from the robot cleaner, and to display the AR image to be superimposed on the real image, wherein the controller generates and displays the AR image to be superimposed on the real image when the robot cleaner is projected onto the real image. ### Claim 2 (depends on 1) The mobile terminal according to claim 1, wherein the AR image of the traveling path includes at least one of a cleaned region and an uncleaned region. ### Claim 3 (depends on 2) The mobile terminal according to claim 2, wherein, if the cleaned region and the uncleaned region are included in the AR image of the traveling path, the controller distinguishably displays the cleaned region and the uncleaned region. ### Claim 4 (depends on 2) The mobile terminal according to claim 2, wherein the controller denotes the cleaned region by a solid line and denotes the uncleaned region by a dotted line. ### Claim 5 (depends on 2) The mobile terminal according to claim 2, wherein, if the uncleaned region is included in the AR image of the traveling path, the controller controls the display to display a notification message of the uncleaned region. ### Claim 6 (depends on 1) The mobile terminal according to claim 1, wherein the controller displays the AR image of the traveling path in the form of at least one of a stereoscopic image, a holographic image and a moving image. ### Claim 7 (depends on 1) The mobile terminal according to claim 1, wherein the real image includes a region where cleaning is performed by the robot cleaner. ### Claim 8 (depends on 1) The mobile terminal according to claim 1, wherein the controller generates and displays the AR image to be superimposed on the real image, when the robot cleaner in a docking state is projected onto the real image. ### Claim 9 An augmented reality system comprising: a robot cleaner configured to perform cleaning while automatically traveling; and a mobile terminal including a display configured to display a real image projected by a built-in camera and a controller configured to recognize the real image to request a traveling path corresponding to the real image from a robot cleaner, to generate an augmented reality (AR) image of the traveling path based on the traveling path received from the robot cleaner, and to display the AR image to be superimposed on the real image, wherein the controller generates and displays the AR image to be superimposed on the real image when the robot cleaner is projected on the real image. ### Claim 10 (depends on 9) The AR system according to claim 9, wherein the AR image of the traveling path includes at least one of a cleaned region and an uncleaned region. ### Claim 11 (depends on 10) The AR system according to claim 10, wherein, if the cleaned region and the uncleaned region are included in the AR image of the traveling path, the controller distinguishably displays the cleaned region and the uncleaned region. ### Claim 12 (depends on 10) The AR system according to claim 10, wherein the controller denotes the cleaned region by a solid line and denotes the uncleaned region by a dotted line. ### Claim 13 (depends on 9) The AR system according to claim 9, wherein the controller displays the AR image of the traveling path in the form of at least one of a stereoscopic image, a holographic image and a moving image. ### Claim 14 (depends on 9) The AR system according to claim 9, wherein the controller generates and displays the AR image to be superimposed on the real image, when the robot cleaner in a docking state is projected on the real image. ### Claim 15 A non-transitory recording medium having recorded thereon a computer-readable program for performing a method of operating a mobile terminal, the method comprising displaying a real image projected by a built-in camera; recognizing the real image to request a traveling path corresponding to the real image from a robot cleaner; generating an augmented reality (AR) image of the traveling path based on the traveling path received from the robot cleaner; and displaying the AR image to be superimposed on the real image, wherein the method further includes generating and displaying the AR image to be superimposed on the real image when the robot cleaner is projected onto the real image. ### Claim 16 (depends on 15) The recording medium according to claim 15, wherein the AR image of the traveling path includes at least one of a cleaned region and an uncleaned region. ### Claim 17 (depends on 16) The recording medium according to claim 16, wherein the displaying of the AR image to be superimposed on the real image includes distinguishably displaying the cleaned region and the uncleaned region if the cleaned region and the uncleaned region are included in the AR image of the traveling path. ### Claim 18 (depends on 15) The recording medium according to claim 15, wherein the generating of the AR image includes: mapping the traveling path received from the robot cleaner to a traveling path of a virtual space; and generating the AR image of the traveling path in the mapped virtual space. ### Claim 19 (depends on 15) The recording medium according to claim 15, wherein the real image includes a region where cleaning is performed by the robot cleaner. ### Claim 20 (depends on 15) The recording medium according to claim 15, wherein the method further includes generating and displaying the AR image to be superimposed on the real image, when the robot cleaner in a docking state is projected onto the real image.