# US 12,118,678 B2 · Method and apparatus for displaying augmented reality object

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,678 B2
- Document ID: US12118678B2
- Status: Granted
- Kind: B2
- Application number: 17/883,077
- Filed: 2022-08-08
- Published: 2022-12-01
- Granted: 2024-10-15
- Earliest claimed: 2021-02-10
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Daiwoong Choi; Soowan Kim; Sanghun Lee; Yongjin Cho; Hyunsoo Kim
- Web page: https://patentgrep.com/patent/US12118678B2

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 20/20: in augmented reality scenes
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An electronic device is provided. The electronic device is configured to execute an augmented reality (AR) application, to obtain an image, to obtain sensing information including at least one of posture information or position information of the electronic device that correspond to the image, to produce a virtual object that is superimposed on the image according to a user input, to obtain, based on the image and the sensing information, identification information corresponding to the virtual object, and to output the virtual object to a display, in response to a case in which information, obtained via at least one sensor in the state in which the AR application is executed, corresponds to the sensing information.

## Claims

### Claim 1

An electronic device comprising:
a display;
at least one sensor;
at least one camera; and
memory storing one or more computer programs; and
one or more processors operatively connected to the display, the at least one sensor, the at least one camera, and the memory,
wherein the one or more computer programs include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to:
execute an augmented reality (AR) application that uses the at least one camera,
obtain a first image of surroundings of the electronic device using the at least one camera,
obtain, using the at least one sensor, first sensing information including at least one of first posture information or first position information of the electronic device that correspond to the first image,
while outputting the first image to the display, produce a virtual object that is superimposed on the first image according to a user input,
, based on the first image and the first sensing information, obtain first identification information corresponding to the virtual object,
obtain a second image that at least partially overlaps the first image using the at least one camera,
obtain, using the at least one sensor, second sensing information including at least one of second posture information or second position information of the electronic device that correspond to the second image,
based on a range of overlap between the first image and the second image being less than a threshold value, obtain second identification information corresponding to the virtual object, and
in response to information; obtained via the at least one sensor in a state in which the AR application is executed, corresponding to the second sensing information, output the virtual object to the display.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to:
based on a change in movement of the electronic device being within a designated range, and the range of overlap between the first image and the second image being less than the threshold value, determine that the second identification information satisfies a designated condition, and
in response to determining that the second identification information satisfies the designated condition, obtain the second identification information.

### Claim 3 (depends on 2)

The electronic device of claim 2, wherein the change in movement of the electronic device comprises at least one of a rotation or a translation of the electronic device.

### Claim 4 (depends on 1)

The electronic device of claim 1, wherein the one or more computer programs include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to;
obtain the second identification information in at least one of a first time interval before production of the virtual object or a second time interval after production of the virtual object, in the state in which the AR application is executed.

### Claim 5 (depends on 1)

The electronic device of claim 1, wherein the one or more computer programs include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to;
in response to information; obtained via the at least one sensor in the state in which the AR application is executed, corresponding to the first sensing information, output the virtual object to the display.

### Claim 6 (depends on 1)

The electronic device of claim 1, wherein the one or more computer programs include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to:
output the virtual object to include at least one of three dimensional (3D) text, a 3D image, or a 3D video according to the user input.

### Claim 7 (depends on 1)

The electronic device of claim 1, wherein the one or more computer programs include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to;
obtain, using a point cloud, information associated with a first space included in the first image and information associated with a second space included in the second image.

### Claim 8 (depends on 7)

The electronic device of claim 7, wherein the information associated with the second space included in the second image includes relative position information of the first image.

### Claim 9 (depends on 1)

The electronic device of claim 1, wherein the one or more computer programs include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to:
store the first identification information and the second identification information in the memory, and
in response to information; obtained via the at least one camera and the at least one sensor in the state in which the AR application is executed, corresponding to the second identification information stored in the memory, output the virtual object to the display.

### Claim 10

A method performed by an electronic device, the method comprising:
executing, by the electronic device, an augmented reality (AR) application that uses at least one camera of the electronic device;
obtaining, by the electronic device, a first image of surroundings of the electronic device using the at least one camera;
obtaining, by the electronic device using at least one sensor of the electronic device, first sensing information including at least one of first posture information or first position information of the electronic device that correspond to the first image;
while outputting the first image to a display of the electronic device, producing, by the electronic device, a virtual object that is superimposed on the first image according to a user input;
based on the first image and the first sensing information, obtaining, by the electronic device, first identification information corresponding to the virtual object;
obtaining, by the electronic device using the at least one camera, a second image that at least partially overlaps the first image;
obtaining, by the electronic device using the at least one sensor, second sensing information including at least one of second posture information or -second position information of the electronic device that correspond to the second image;
based on a range of overlap between the first image and the second image being less than a threshold value, obtaining, by the electronic device, second identification information corresponding to the virtual object; and
in response to information obtained via the at least one sensor in a state in which the AR application is executed corresponding to the second sensing information, outputting, by the electronic device, the virtual object to the display.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
based on a change in movement of the electronic device being within a designated range, and the range of overlap between the first image and the second image is being less than the threshold value, determining, by the electronic device, that the second identification information satisfies a designated condition; and
in response to determining that the second identification information satisfies the designated condition, obtaining, by the electronic device, the second identification information.

### Claim 12 (depends on 10)

The method of claim 10, wherein the obtaining of the second identification information comprises obtaining, by the electronic device, the second identification information in at least one a first time interval before production of the virtual object or a second time interval after production of the virtual object, in the state in which the AR application is executed.

### Claim 13 (depends on 10)

The method of claim 10, wherein the outputting of the virtual object comprises, in response to a information obtained via the at least one sensor in the state in which the AR application is executed; corresponds to the first sensing information, outputting, by the electronic device, the virtual object.

### Claim 14 (depends on 10)

The method of claim 10, wherein the outputting of the virtual object comprises outputting, by the electronic device, the virtual object to include at least one of three dimensional (3D) text, a 3D image, or a 3D video according to the user input.

### Claim 15 (depends on 10)

The method of claim 10,
wherein the obtaining of the first image and the second image comprises obtaining, by the electronic device using a point cloud, information associated with a first space included in the first image and information associated with a second space included in the second image, and
wherein the method further comprises:
storing, by the electronic device, the first identification information and the second identification information in a memory of the electronic device; and
in response to information; obtained via the at least one camera and the at least one sensor in the state in which the AR application is executed; corresponding to the second identification information stored in the memory, outputting, by the electronic device, the virtual object to the display.

### Claim 16

An electronic device comprising:
communication circuitry;
a display;
at least one sensor;
at least one camera;
memory storing one or more computer programs; and
one or more processors operatively connected to the communication circuitry, the display, the at least one sensor, the at least one camera, and the memory,
wherein the one or more computer programs include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to:
execute an augmented reality (AR) application that uses the at least one camera,
obtain a first image of surroundings of the electronic device using the at least one camera,
obtain, using the at least one sensor, first sensing information including at least one of first posture information or first position information of the electronic device that correspond to the first image,
while outputting the first image to the display, produce a virtual object that is superimposed on the first image according to a user input,
based on the first image and the first sensing information, obtain first identification information corresponding to the virtual object,
obtain a second image that at least partially overlaps the first image using the at least one camera,
obtain, using the at least one sensor, second sensing information including at least one of second posture information or second position information of the electronic device that correspond to the second image,
based on a range of overlap between the first image and the second image being less than a threshold value, obtain second identification information corresponding to the virtual object,
transmit, to a server via the communication circuitry, the first identification information and the second identification information,
receive, from the server, the first identification information and the second identification information, and
in response to information; obtained via the at least one camera and the at least one sensor in a state in which the AR application is executed, corresponding to the second identification information, output the virtual object to the display.

### Claim 17 (depends on 16)

The electronic device of claim 16, wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to:
based on a change in movement of the electronic device being within a designated range, and the range of overlap between the first image and the second image is being less than the threshold value, determine that the second identification information satisfies a designated condition, and
in response to determining that the second identification information satisfies the designated condition, obtain the second identification information.

### Claim 18 (depends on 16)

The electronic device of claim 16, wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to;
in response to information, obtained via the at least one sensor in the state in which the AR application is executed, corresponding to the first sensing information, output the virtual object to the display.

### Claim 19 (depends on 16)

The electronic device of claim 16, wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to;
output, to the display, the virtual object to include at least one of three dimensional (3D) text, a 3D image, or a 3D video according to the user input.

### Claim 20 (depends on 16)

The electronic device of claim 16, wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to;
obtain, using a point cloud, information associated with a first space included in the first image and information associated with a second space included in the second image.