# US 11,442,274 B2 · Electronic device and method for displaying object associated with external electronic device on basis of position and movement of external electronic device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,274 B2 - Document ID: US11442274B2 - Status: Granted - Kind: B2 - Application number: 17/059,374 - Filed: 2019-03-22 - Published: 2021-08-05 - Granted: 2022-09-13 - Earliest claimed: 2018-05-29 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Hyunji Song; Wootaek Song; Taekyung Lee - Web page: https://patentgrep.com/patent/US11442274B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/53: involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game - A63F 13/537: using indicators, e.g. showing the condition of a game character on screen - A63F 13/98: Accessories, i.e. detachable arrangements optional for the use of the video game device, e.g. grip supports of game controllers - G02B 27/0172: {characterised by optical features} - G06F 3/012: {Head tracking input arrangements} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06V 10/40: Extraction of image or video features - A63F 13/21: characterised by their sensors, purposes or types - A63F 13/525: Changing parameters of virtual cameras - A63F 2300/105: using inertial sensors, e.g. accelerometers, gyroscopes - A63F 2300/1093: using visible light - A63F 2300/8082: Virtual reality - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An electronic device according to various embodiments of the present disclosure includes one or more cameras having a designated field of view, a display, a communication circuitry and a processor wherein the processor is configured to identify an external electronic device among one or more external objects included in the designated field of view using the camera, display a graphic object corresponding to the external electronic device on the display based on first location information of the external electronic device identified based at least on the image information obtained using the camera, and when the external electronic device is out of the designated field of view, display the graphic object on the display based on second location information of the external electronic device identified using the camera before the external electronic device is out of the designated field of view and information related to the movement of the external electronic device received from the external electronic via the communication circuitry after the external electronic device is out of the designated field of view. ## Claims ### Claim 1 An electronic device, comprising: one or more cameras having a designated field of view; a display; a communication circuitry; and a processor, wherein the processor is configured to: identify an external electronic device among one or more external objects included in the designated field of view using the one or more cameras, based on first location information of the external electronic device identified based at least on image information obtained using the one or more cameras, display a graphic object corresponding to the external electronic device on the display, and when the external electronic device is out of the designated field of view, display the graphic object on the display based on second location information of the external electronic device identified using the one or more cameras before the external electronic device is out of the designated field of view and information related to a movement of the external electronic device received from the external electronic device via the communication circuitry after the external electronic device is out of the designated field of view. ### Claim 2 (depends on 1) The electronic device according to claim 1, wherein: the external electronic device further comprises a first sensor comprising a gyro sensor; and a second sensor including an acceleration sensor, and the information related to the movement of the external electronic device comprise at least one of data related to a tilt of the external electronic device obtained by the first sensor or data related to a speed change of the external electronic device obtained by the second sensor. ### Claim 3 (depends on 1) The electronic device according to claim 1, wherein: the external electronic device further comprises a light emitting part including one or more light emitting diode (LED), and the processor is configured to acquire an image including at least a portion of the light emitting part using the one or more cameras, estimate a location related to the light emitting part based at least on an image calculation for the image, and determine the first location information related to the external electronic device based at least on the estimated location. ### Claim 4 (depends on 2) The electronic device according to claim 2, wherein the processor is further configured to: when the external electronic device is located within the designated field of view, identify a display location of the graphic object based at least on the first location information and the data related to the tilt of the external electronic device, when the external electronic device is out of the designated field of view, identify the display location of the graphic object based at least on the second location information and third location information, and display the graphic object based on the identified display location, and wherein the third location information includes vector information obtained based on the data related to the tilt of the external electronic device and the data related to the speed change of the external electronic device. ### Claim 5 (depends on 1) The electronic device according to claim 1, wherein the processor is configured to display the graphic object based on multimedia content related to virtual reality, the first location information, and the second location information provided by the electronic device. ### Claim 6 (depends on 1) The electronic device according to claim 1, wherein the processor is configured to: if the external electronic device is not identified within the designated field of view, receive, from the external electronic device, one or more third location information acquired during a designated time range before the external electronic device enters a low power mode, and determine a display location of the graphic object to be displayed on the display based at least on the one or more third location information when a display request of the graphic object is confirmed. ### Claim 7 (depends on 1) The electronic device according to claim 1, wherein the processor is configured to: identify that the external electronic device has entered a sleep mode, receive third location information of the external electronic device corresponding to an entry time point of the sleep mode and information related to tilting of the external electronic device corresponding to the entry time point, identify that the sleep mode is released, receive information related to the tilting of the external electronic device corresponding to a release time point of the sleep mode, and display a second graphic object for indicating a location of the external electronic device based at least on the third location information of the external electronic device and information related to the tilting of the external electronic device corresponding to the entry time point, and information related to the tilting of the external electronic device corresponding to the release time point. ### Claim 8 (depends on 7) The electronic device according to claim 7, wherein the second graphic object includes an image of a same or similar shape with the external electronic device or an image of an arrow shape. ### Claim 9 A method of an electronic device, comprising: identifying an external electronic device among one or more external objects included in a designated field of view using one or more cameras having the designated field of view; displaying a graphic object corresponding to the external electronic device on a display based on first location information of the external electronic device identified based at least on image information obtained through the one or more cameras; and when the external electronic device is out of the designated field of view, displaying the graphic object on the display based on second location information of the external electronic device identified using the one or more cameras before the external electronic device is out of the designated field of view and information related to a movement of the external electronic device received from the external electronic device via a communication circuitry after the external electronic device is out of the designated field of view. ### Claim 10 (depends on 9) The method according to claim 9, wherein the information related to the movement of the external electronic device includes at least one of data related to a tilt of the external electronic device obtained by a first sensor included in the external electronic device, or data related with a change in speed of the external electronic device obtained by a second sensor included in the external electronic device. ### Claim 11 (depends on 10) The method according to claim 10, further comprising: when the external electronic device is located within the designated field of view, identifying a display location of the graphic object based at least on the first location information and the data related to the tilt of the external electronic device; when the external electronic device is out of the designated field of view, identifying the display location of the graphic object based at least on the second location information and third location information; and based on the identified display location, displaying the graphic object, wherein the third location information includes vector information obtained based on the data related to the tilt of the external electronic device and the data related to the change in speed of the external electronic device. ### Claim 12 (depends on 9) The method according to claim 9, further comprising: obtaining an image including at least a portion of a light emitting part included in the external electronic device using the one or more cameras; estimating a location related to the light emitting part based at least on an image operation on the image; and determining the first location information related to the external electronic device based at least on the estimated location. ### Claim 13 (depends on 9) The method according to claim 9, wherein the displaying the graphic object on the display includes displaying the graphic object based on multimedia content related to virtual reality, the first location information, and the second location information. ### Claim 14 (depends on 9) The method according to claim 9, further comprising: when the external electronic device is not identified within the designated field of view, receiving one or more third location information, from the external electronic device, acquired during a designated time range before the external electronic device enters a low power mode; and when a display request of the graphic object is confirmed, determining a display location of the graphic object to be displayed on the display based at least on the one or more third location information. ### Claim 15 (depends on 9) The method according to claim 9, further comprising: identifying that the external electronic device has entered a sleep mode; receiving third location information of the external electronic device corresponding to an entry time point of the sleep mode and information related to tilting of the external electronic device corresponding to the entry time point; identifying that the sleep mode has been released; receiving information related to tilting of the external electronic device corresponding to a release time point of the sleep mode; and displaying a second graphic object for indicating a location of the external electronic device at least based on the third location information of the external electronic device and information related to the tilting of the external electronic device corresponding to the entry time point, and information related to the tilting of the external electronic device corresponding to the release time point. ### Claim 16 An electronic device, comprising: a housing; a memory; a camera having a designated field of view and visually exposed through at least a portion of a first surface of the housing; a display exposed through at least a portion of a second surface, of the housing, different from the first surface; a communication module; and a processor, wherein the processor is configured to: obtain information related to a location of an external electronic device interlocked with the electronic device using the camera, receive information related to tilting of the external electronic device from the external electronic device using the communication module, identify a graphic object based on at least one of the information related to the location or the information related to tilting, provide the graphic object for representing the external electronic device with multimedia content related to virtual reality using the display, identify that the external electronic device deviates from the designated field of view of the camera using the camera, in response to identifying a deviation, transmit a signal for requesting the external electronic device to transmit information related to a change in the location of the external electronic device, and change a displaying of the graphic object based on at least one of the information related to the location of the external electronic device at a deviation time obtained using the camera, the information related to tilting of the external electronic device, or the information related to the change in the location of the external electronic device. ### Claim 17 (depends on 16) The electronic device according to claim 16, wherein: the information related to the change in the location of the external electronic device is determined based on data obtained by a sensor included in the external electronic device, and the sensor includes at least one of an acceleration sensor or a gyro sensor. ### Claim 18 (depends on 16) The electronic device according to claim 16, wherein the processor is configured to activate an acceleration sensor in response to identifying that the external electronic device deviates from the designated field of view of the camera. ### Claim 19 (depends on 16) The electronic device according to claim 16, wherein the processor is configured to obtain a location to which the external electronic device is predicted to have moved based at least on the information related to the location of the external electronic device at the deviation time, the information related to tilting of the external electronic device, and the information related to the change in the location of the external electronic device. ### Claim 20 (depends on 16) The electronic device according to claim 16, wherein: the graphic object is displayed superimposed on the multimedia content related to the virtual reality, and the multimedia content includes content for providing virtual reality service.