# US 2018/0322707 A1 · Method and system for performing interaction based on augmented reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0322707 A1 - Document ID: US20180322707A1 - Status: Published application - Kind: A1 - Application number: 16/034,130 - Filed: 2018-07-12 - Published: 2018-11-08 - Earliest claimed: 2013-01-25 - Assignees: Tencent Technology (Shenzhen) Company Limited - Applicants: Tencent Technology (Shenzhen) Company Limited - Inventors: Zhenwei Zhang; Ling Wang; Fen Xiao; Zhehui Wu - Web page: https://patentgrep.com/patent/US20180322707A1 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 11/00: Two-dimensional [2D] image generation - G06V 20/20: in augmented reality scenes - G06T 2207/10004: Still image; Photographic image - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method for performing interaction based on augmented reality is performed at a first mobile terminal. While rendering a real-time scene photographed by its camera, the first mobile terminal submits its current location a remote server and then receives, from the remote server, information of multiple mobile terminals. After calculating the distance and orientation relationships between the multiple mobile terminals and itself, the first mobile terminal selects a second mobile terminal whose distance and orientation relationship with the first mobile terminal satisfies a preset requirement and then renders one or more interactive icons of the second mobile terminal near a portion of the real-time scene corresponding to the second location of the second mobile terminal. Finally, the first mobile terminal performs a respective interaction with the second mobile terminal in response to a user selection of a respective one of the one or more interactive icons. ## Claims ### Claim 1 A method for performing interaction based on augmented reality at a first mobile terminal having a camera, a display, one or more processors and memory storing a plurality of programs to be executed by the one or more processors, the method comprising: while rendering, on the display, a real-time scene photographed by the camera: submitting a current location of the first mobile terminal to a remote server; receiving, from the remote server, information of second locations of multiple mobile terminals, the second location of each mobile terminal being within a first predefined distance of the current location of the first mobile terminal; calculating distance and orientation relationships between the multiple mobile terminals and the first mobile terminal; selecting, among the multiple mobile terminals, a second mobile terminal whose distance and orientation relationship with the first mobile terminal satisfies a preset requirement; rendering one or more interactive icons of the second mobile terminal on top of the real-time scene photographed by the camera, wherein the interactive icons are located near a portion of the real-time scene corresponding to the second location of the second mobile terminal; and performing a respective interaction with the second mobile terminal in response to a user selection of a respective one of the one or more interactive icons, the respective interaction including sending by the first mobile terminal text, voice, video, and/or image message to the second mobile terminal. ### Claim 2 (depends on 1) The method of claim 1, wherein the distance and orientation relationship between the second mobile terminal and the first mobile terminal satisfies a preset requirement when both the distance relationship between the first mobile terminal and the second mobile terminal satisfies a second threshold distance that is less than the first threshold distance, and the orientation relationship between the second mobile terminal and the first mobile terminal is within a threshold field angle of the camera of the first mobile terminal. ### Claim 3 (depends on 1) The method of claim 1, wherein the second threshold distance and the threshold field angle are automatically adjusted in accordance with a tilt angle and camera parameters of the first mobile terminal. ### Claim 4 (depends on 1) The method of claim 1, further comprising: updating the distance and orientation relationship between the second mobile terminal and the first mobile terminal; and removing the one or more interactive icons of the second mobile terminal from the display when the updated distance and orientation relationship between the second mobile terminal and the first mobile terminal no longer satisfies the preset requirement. ### Claim 5 (depends on 1) The method of claim 1, wherein there are multiple second mobile terminals whose distance and orientation relationships with the first mobile terminal satisfy the preset requirement and the interactive icons corresponding to each of the multiple second mobile terminals are located near a corresponding portion of the real-time scene corresponding to the second location of the corresponding mobile terminal such that their respective positional relationships on the real-time scene are in correspondence with the corresponding positional relationships of the multiple second terminals in the real world. ### Claim 6 A first mobile terminal for performing interaction based on augmented reality, comprising: a camera; a display; one or more processors; memory coupled to the one or more processors; and a plurality of machine executable computer instructions stored in the memory, wherein the plurality of machine executable computer instructions, when executed by the one or more processors, cause the first terminal to perform the following operations: while rendering, on the display, a real-time scene photographed by the camera: submitting a current location of the first mobile terminal to a remote server; receiving, from the remote server, information of second locations of multiple mobile terminals, the second location of each mobile terminal being within a first predefined distance of the current location of the first mobile terminal; calculating distance and orientation relationships between the multiple mobile terminals and the first mobile terminal; selecting, among the multiple mobile terminals, a second mobile terminal whose distance and orientation relationship with the first mobile terminal satisfies a preset requirement; rendering one or more interactive icons of the second mobile terminal on top of the real-time scene photographed by the camera, wherein the interactive icons are located near a portion of the real-time scene corresponding to the second location of the second mobile terminal; and performing a respective interaction with the second mobile terminal in response to a user selection of a respective one of the one or more interactive icons, the respective interaction including sending by the first mobile terminal text, voice, video, and/or image message to the second mobile terminal. ### Claim 7 (depends on 6) The first mobile terminal of claim 6, wherein the distance and orientation relationship between the second mobile terminal and the first mobile terminal satisfies a preset requirement when both the distance relationship between the first mobile terminal and the second mobile terminal satisfies a second threshold distance that is less than the first threshold distance, and the orientation relationship between the second mobile terminal and the first mobile terminal is within a threshold field angle of the camera of the first mobile terminal. ### Claim 8 (depends on 6) The first mobile terminal of claim 6, wherein the second threshold distance and the threshold field angle are automatically adjusted in accordance with a tilt angle and camera parameters of the first mobile terminal. ### Claim 9 (depends on 6) The first mobile terminal of claim 6, wherein the operations further comprise: updating the distance and orientation relationship between the second mobile terminal and the first mobile terminal; and removing the one or more interactive icons of the second mobile terminal from the display when the updated distance and orientation relationship between the second mobile terminal and the first mobile terminal no longer satisfies the preset requirement. ### Claim 10 (depends on 6) The first mobile terminal of claim 6, wherein there are multiple second mobile terminals whose distance and orientation relationships with the first mobile terminal satisfy the preset requirement and the interactive icons corresponding to each of the multiple second mobile terminals are located near a corresponding portion of the real-time scene corresponding to the second location of the corresponding mobile terminal such that their respective positional relationships on the real-time scene are in correspondence with the corresponding positional relationships of the multiple second terminals in the real world. ### Claim 11 A non-transitory computer readable storage medium storing a plurality of programs for performing interaction based on augmented reality at a first mobile terminal having a camera, a display, and one or more processors, wherein the plurality of programs, when executed by the one or more processors, cause the first mobile terminal to perform the following operations: while rendering, on the display, a real-time scene photographed by the camera: submitting a current location of the first mobile terminal to a remote server; receiving, from the remote server, information of second locations of multiple mobile terminals, the second location of each mobile terminal being within a first predefined distance of the current location of the first mobile terminal; calculating distance and orientation relationships between the multiple mobile terminals and the first mobile terminal; selecting, among the multiple mobile terminals, a second mobile terminal whose distance and orientation relationship with the first mobile terminal satisfies a preset requirement; rendering one or more interactive icons of the second mobile terminal on top of the real-time scene photographed by the camera, wherein the interactive icons are located near a portion of the real-time scene corresponding to the second location of the second mobile terminal; and performing a respective interaction with the second mobile terminal in response to a user selection of a respective one of the one or more interactive icons, the respective interaction including sending by the first mobile terminal text, voice, video, and/or image message to the second mobile terminal. ### Claim 12 (depends on 11) The non-transitory computer readable storage medium of claim 11, wherein the distance and orientation relationship between the second mobile terminal and the first mobile terminal satisfies a preset requirement when both the distance relationship between the first mobile terminal and the second mobile terminal satisfies a second threshold distance that is less than the first threshold distance, and the orientation relationship between the second mobile terminal and the first mobile terminal is within a threshold field angle of the camera of the first mobile terminal. ### Claim 13 (depends on 11) The non-transitory computer readable storage medium of claim 11, wherein the second threshold distance and the threshold field angle are automatically adjusted in accordance with a tilt angle and camera parameters of the first mobile terminal. ### Claim 14 (depends on 11) The non-transitory computer readable storage medium of claim 11, wherein the operations further comprise: updating the distance and orientation relationship between the second mobile terminal and the first mobile terminal; and removing the one or more interactive icons of the second mobile terminal from the display when the updated distance and orientation relationship between the second mobile terminal and the first mobile terminal no longer satisfies the preset requirement. ### Claim 15 (depends on 11) The non-transitory computer readable storage medium of claim 11, wherein there are multiple second mobile terminals whose distance and orientation relationships with the first mobile terminal satisfy the preset requirement and the interactive icons corresponding to each of the multiple second mobile terminals are located near a corresponding portion of the real-time scene corresponding to the second location of the corresponding mobile terminal such that their respective positional relationships on the real-time scene are in correspondence with the corresponding positional relationships of the multiple second terminals in the real world.