# US 2019/0377948 A1 · METHOD FOR PROVIDING eXtended Reality CONTENT BY USING SMART DEVICE > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0377948 A1 - Document ID: US20190377948A1 - Status: Published application - Kind: A1 - Application number: 16/545,582 - Filed: 2019-08-20 - Published: 2019-12-12 - Earliest claimed: 2019-06-24 - Assignees: LG Electronics Inc. - Applicants: LG Electronics Inc. - Inventors: Eunkoo LEE; Kihyung LEE - Web page: https://patentgrep.com/patent/US20190377948A1 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06V 20/20: in augmented reality scenes - G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera} - G01C 21/365: {Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 40/18: Eye characteristics, e.g. of the iris - H04W 4/48: for in-vehicle communication - G06T 2207/20081: Training; Learning - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract Disclosed is a method of providing XR content using a smart device, the method including receiving state information of a vehicle driver from a sensor located on a front side of the smart device, receiving a first image of an external environment of a vehicle from a sensor located on a rear side of the smart device, generating an extended reality (XR) item based on the state information of the vehicle driver, and outputting XR content including a second image in which the generated XR item is displayed on the first image. One or more of an autonomous vehicle, a first terminal, a second terminal, and a server disclosed here is connectable to, for example, an artificial intelligence module, an unmanned aerial vehicle (UAV), a robot, an augmented reality (AR) device, a virtual reality (VR) device, or a 5G service device. ## Claims ### Claim 1 A method of providing XR content using a smart device, the method comprising: receiving state information of a vehicle driver from a sensor located on a front side of the smart device; receiving a first image of an external environment of a vehicle from a sensor located on a rear side of the smart device; generating an extended reality (XR) item based on the state information of the vehicle driver; and outputting XR content including a second image in which the generated XR item is displayed on the first image. ### Claim 2 (depends on 1) The method of claim 1, wherein the generating includes generating the XR item based on an estimated driving route of the vehicle when the estimated driving route is input, and wherein the estimated driving route of the vehicle is generated by the smart device or is received from the vehicle. ### Claim 3 (depends on 1) The method of claim 1, wherein the receiving the first image includes selecting a sensor that acquires the first image from among a plurality of sensors based on the state information of the vehicle driver when the sensor located on the rear side of the smart device is provided in a plural number. ### Claim 4 (depends on 3) The method of claim 3, wherein the selecting includes: determining whether or not a gaze of the vehicle driver is directed to a first side based on the state information of the vehicle driver; and selecting a second lens as the sensor that acquires the first image from among the plurality of sensors located on the rear side of the smart device based on a result of the determining. ### Claim 5 (depends on 1) The method of claim 1, wherein the generating includes: receiving information on an internal environment of the vehicle from the vehicle; and generating the XR item based on the received information on the internal environment of the vehicle. ### Claim 6 (depends on 1) The method of claim 1, wherein the outputting includes outputting the second image to at least one of a display of the smart device and a display of the vehicle. ### Claim 7 (depends on 6) The method of claim 6, further comprising: updating the XR item based on an input by the vehicle driver when the input by the vehicle driver is received from the smart device or the vehicle; and outputting a third image including the updated XR item to one of the smart device and the vehicle from which no input is received. ### Claim 8 (depends on 1) The method of claim 1, wherein the state information of the vehicle driver is information acquired by tracing pupils of the vehicle driver. ### Claim 9 (depends on 1) The method of claim 1, further comprising: detecting information on an eye focus of the vehicle driver based on the state information of the vehicle driver; and changing an attribute of the XR item or additionally generating warning content based on the detected information on the eye focus of the vehicle driver. ### Claim 10 (depends on 9) The method of claim 9, wherein the attribute of the XR item includes at least one of a size, a color, a transparency, and a position of the XR item and whether or not the XR item flickers. ### Claim 11 A smart device comprising: a processor configured to receive state information of a vehicle driver from a sensor located on a front side of the smart device, to receive a first image of an external environment of a vehicle from a sensor located on a rear side of the smart device, to generate an XR item based on the state information of the vehicle driver, and to output XR content including a second image in which the generated XR item is displayed on the first image; a memory configured to store at least one of the state information of the vehicle driver, the first image, the XR item, the second image, and the XR content; and a communication unit connected to the processor to perform transmission and reception of a signal between the vehicle and the processor. ### Claim 12 (depends on 11) The smart device of claim 11, wherein the processor generates the XR item based on an estimated driving route of the vehicle when the estimated driving route is input, and wherein the estimated driving route of the vehicle is generated by the smart device or is received from the vehicle. ### Claim 13 (depends on 11) The smart device of claim 11, wherein the processor selects a sensor that acquires the first image from among a plurality of sensors based on the state information of the vehicle driver before receiving the first image when the sensor located on the rear side of the smart device is provided in a plural number. ### Claim 14 (depends on 13) The smart device of claim 13, wherein the processor determines whether or not a gaze of the vehicle driver is directed to a first side based on the state information of the vehicle driver when selecting the sensor that acquires the first image, and selects a second lens as the sensor that acquires the first image from among the plurality of sensors located on the rear side of the smart device based on a result of the determination. ### Claim 15 (depends on 11) The smart device of claim 11, wherein the processor receives information on an internal environment of the vehicle from the vehicle, and generates the XR item based on the received information on the internal environment of the vehicle. ### Claim 16 (depends on 11) The smart device of claim 11, wherein the processor outputs the second image to at least one of a display of the smart device and a display of the vehicle. ### Claim 17 (depends on 16) The smart device of claim 16, wherein the processor updates the XR item based on an input by the vehicle driver when the input by the vehicle driver is received from the smart device or the vehicle, and outputs a third image including the updated XR item to one of the smart device and the vehicle from which no input is received. ### Claim 18 (depends on 11) The smart device of claim 11, wherein the state information of the vehicle driver is information acquired by tracing pupils of the vehicle driver. ### Claim 19 (depends on 11) The smart device of claim 11, wherein the processor detects information on an eye focus of the vehicle driver based on the state information of the vehicle driver, and changes an attribute of the XR item or additionally generates warning content based on the detected information on the eye focus of the vehicle driver. ### Claim 20 (depends on 1) A computer readable recording medium in which a method for executing the method of claim 1 in a computer is recorded.