# US 11,308,653 B2 · Electronic device and method for providing augmented reality service based on a user of electronic device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,653 B2
- Document ID: US11308653B2
- Status: Granted
- Kind: B2
- Application number: 16/761,988
- Filed: 2018-08-29
- Published: 2020-08-20
- Granted: 2022-04-19
- Earliest claimed: 2017-11-23
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Yeo Jun Yoon; Ji Yeon Kwak; Rae Kyoung Lee; Eun Joo Mo; Jae Ik Lee
- Web page: https://patentgrep.com/patent/US11308653B2

## CPC

- G06T 11/00: Two-dimensional [2D] image generation
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/2431: Multiple classes
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06V 10/17: {using hand-held instruments}
- G06V 20/20: in augmented reality scenes
- G06V 20/35: {Categorising the entire scene, e.g. birthday party or wedding scene}
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- H04M 1/725: Cordless telephones (user interfaces specially adapted therefor H04M1/724)
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)

## Abstract

An electronic device is disclosed. The electronic device according to an embodiment disclosed in the disclosure includes a camera, a display, a memory storing a plurality of captured images and preference information for each captured image, and a processor operatively connected to the camera, the display, and the memory, wherein the processor obtains a live view image using the camera, determines at least one image of which a preference is relatively high among the plurality of captured images as additional information of an augmented reality service based on at least one of the preference information or an object included in the live view image, and displays the additional information and the live view image on the display.

## Claims

### Claim 1

An electronic device comprising:
a camera;
a display;
a memory storing a plurality of captured images; and
a processor operatively connected to the camera, the display, and the memory,
wherein the processor is configured to:
obtain a live view image using the camera;
determine a first category to which at least one object included in the live view image belongs;
select, among the plurality of captured images, at least one first captured image corresponding to the first category;
determine preference information indicating a preference of a user for the selected at least one first captured image based on at least one object included in the selected at least one first captured image;
determine at least one image of which the preference is relatively high among the selected at least one first captured image;
display a first additional information of an augmented reality service based on the determined at least one image, the live view image, and a filtering object for determining a second additional information of the augmented reality service on the display;
in response to receiving a first user input selecting the filtering object,
display a list of at least one category to which the at least one object included in the selected at least one first captured image belongs on the display; and
in response to receiving a second user input selecting a second category from the list of at least one category,
select at least one second captured image corresponding to the second category,
display the second additional information of the augmented reality service based on the selected at least one second captured image and the live view image.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the memory stores each captured image and category information corresponding to a category to which an object included in each captured image in a manner of associating each captured image with the category information,
wherein the preference information is preference information for the category information,
wherein the processor is configured to:
select, among the plurality of captured images, the at least one first captured image corresponding to the first category based on the category information; and
determine the at least one image belonging to a category of which the preference is relatively high among the selected at least one first captured image based on the preference information.

### Claim 3 (depends on 2)

The electronic device of claim 2, further comprising:
a communication module set to communicate with a server,
wherein the processor is configured to:
transmit the live view image to the server through the communication module; and
receive the category information about a category to which the at least one object included in the live view image belong from the server.

### Claim 4 (depends on 3)

The electronic device of claim 3, wherein the category information about the category to which the at least one object included in the live view image belongs is determined based on a result of comparison, by the server, between the at least one object included in the live view image and an object stored in a database of the server.

### Claim 5 (depends on 2)

The electronic device of claim 2, wherein the processor is configured to:
identify the category information about categories to which the at least one object included in the at least one first captured image belongs; and
determine preference information for the category information about the categories to which the at least one object included in the plurality of captured images belong using a frequency of appearances of the category information about the categories to which the at least one object included in the plurality of captured images belong.

### Claim 6 (depends on 5)

The electronic device of claim 5, wherein the processor is configured to determine the preference information for the category information about the categories to which the at least one object included in the plurality of captured images belong by applying a weight value on a frequency of appearances of at least one category based on at least one of history information about creation, sharing, or editing of each captured image, or environment information corresponding to a time of creation of each captured image.

### Claim 7 (depends on 6)

The electronic device of claim 6, wherein the history information includes at least one of search history information of each captured image, shared history information of each captured image, tag information of each captured image, and app information about an app created each captured image.

### Claim 8 (depends on 6)

The electronic device of claim 6, wherein the environment information includes at least one of time information, date information, current location information, and weather information.

### Claim 9 (depends on 2)

The electronic device of claim 2, wherein the processor is configured to:
identify at least one third category excluding the first category to which the at least one object included in the live view image belongs among a plurality of different categories of the at least one object included in the at least one first captured image based on category information of the at least one first captured image; and
determine the at least one image of which the preference is relatively high among the at least one third category as the first additional information based on the preference information.

### Claim 10 (depends on 9)

The electronic device of claim 9, wherein the processor is configured to determine at least one image having a shared history, a search history, or a tagging history, or created using a screen capture function among the plurality of captured images as the first additional information.

### Claim 11 (depends on 2)

The electronic device of claim 2, wherein the processor is configured to:
when the at least one first captured image includes a plurality of images, and preferences of the plurality of images are the same,
identify environment information at a time of capturing each of the plurality of images and environment information at a current time of obtaining the live view image, respectively; and
determine the at least one image among the plurality of images based on a relative similarity between the environment information at the time of capturing and the environment information at the current time.

### Claim 12 (depends on 11)

The electronic device of claim 11, wherein the processor is configured to, when there is an image captured at a close location within a specified distance from a current location of the electronic device among the plurality of images, determine at least one image at the close location within the specified distance as the first additional information.

### Claim 13 (depends on 2)

The electronic device of claim 2, wherein the processor is configured to:
when the at least one first captured image includes a plurality of images, and preferences of the plurality of images are the same,
identify current time information and current date information among capturing date and capturing time information and environment information of the plurality of images; and
determine at least one image relatively recently captured among the plurality of images as the first additional information.

### Claim 14 (depends on 2)

The electronic device of claim 2, wherein the processor is configured to:
when the at least one first captured image includes a plurality of images, and preferences of the plurality of images are the same,
identify tones of the live view image and the plurality of images; and
determine at least one image relatively similar in tone to the live view image among the plurality of images.

### Claim 15 (depends on 2)

The electronic device of claim 2, wherein the memory further stores a plurality of promotion information,
wherein the processor is configured to:
identify association between environment information of the plurality of captured images and category information of the plurality of captured images;
identify the user's taste corresponding to environment information and the category information based on the identified association;
determine promotion information corresponding to the user's taste among the plurality of promotion information based on environment information at a current time of obtaining the live view image; and
display the live view image and the promotion information on the display.