# US 10,769,413 B2 · Mobile terminal and control method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,413 B2
- Document ID: US10769413B2
- Status: Granted
- Kind: B2
- Application number: 15/499,820
- Filed: 2017-04-27
- Published: 2017-12-28
- Granted: 2020-09-08
- Earliest claimed: 2016-06-23
- Assignees: LG Electronics Inc.
- Applicants: LG Electronics Inc.
- Inventors: Taehoon Cho; Sanghyun Eim
- Web page: https://patentgrep.com/patent/US10769413B2

## CPC

- G06Q 20/322: {Aspects of commerce using mobile devices [M-devices]}
- G06Q 20/3278: {RFID or NFC payments by means of M-devices}
- G06Q 20/40145: {Biometric identity checks}
- G06Q 30/0201: Market modelling; Market analysis; Collecting market data
- G06Q 30/0217: involving input on products or services in exchange for incentives or rewards
- G06Q 30/0282: Rating or review of business operators or products
- G06Q 30/0631: {Recommending goods or services}
- G06V 10/17: {using hand-held instruments}
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/172: {Classification, e.g. identification}
- G06V 40/174: {Facial expression recognition}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- H04W 4/80: Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

A mobile terminal includes a wireless communication unit communicating with a point-of-sales (POS) terminal positioned nearby, and a controller performing, when a payment event regarding goods occurs, payment regarding the goods together with the POS terminal using an image received from the outside, wherein when a facial image corresponding to the image corresponds to a previously registered user, the controller approves payment regarding the goods and generates evaluation information regarding the goods using the facial image included in the received image.

## Claims

### Claim 1

A mobile terminal comprising:
a display;
a camera capturing images;
a database;
a wireless communication unit communicating with a wireless communication module located in a store or a point-of-sale (POS) terminal in the store; and
a controller configured to:
receive a facial image of a user related to goods from the POS terminal when a payment is made between the mobile terminal and the POS terminal;
receive a facial image of the user related to goods from the wireless communication module located in the store when the user of the mobile terminal leaves the store without purchasing the goods;
analyze the facial image received from the POS terminal or the wireless communication module using a face analysis algorithm to obtain facial expression information corresponding to the facial image received from the POS terminal or the wireless communication module according to the analysis;
generate goods information by associating the obtained facial expression information with other information related to the goods, and storing the goods information in the database;
search for goods information stored in the database by analyzing a facial image of the user received via the camera based on the face analysis algorithm in order to extract facial expression information corresponding to the facial image of the user received via the camera and feeling information, the feeling information extracted based on the facial expression information corresponding to the facial image received via the camera; and
cause the display to output the searched goods information stored in the database corresponding to at least the extracted facial expression information or feeling information,
wherein the analyzing the facial image received via the camera comprises:
causing the display to output an analysis result for the user to confirm whether the facial expression information corresponding to the facial image received via the camera corresponds to the analysis result when the analysis of the facial image received via the camera is complete;
causing the display to output other feeling items according to the user's response with regard to confirmation; and
matching a feeling corresponding to a feeling item selected by the user from the other feeling items to the facial expression information corresponding to the facial image received via the camera based on the facial image received via the camera.

### Claim 2 (depends on 1)

The mobile terminal of claim 1, wherein generating the goods information comprises:
obtaining the user's gesture from an image received from the POS terminal;
combining the facial expression information corresponding to the facial image received from the POS terminal or the wireless communication module with the obtained user's gesture to extract the feeling information related to the goods; and
matching the extracted feeling information to the other information related to the goods.

### Claim 3 (depends on 1)

The mobile terminal of claim 1, wherein the controller is further configured to: sense that the mobile terminal is positioned in the store based on communication between the wireless communication unit and the wireless communication module located in the store; activate at least the camera or a microphone when it is sensed that the mobile terminal is positioned in the store; and generate the goods information by using information received via at least the camera or the microphone.

### Claim 4 (depends on 3)

The mobile terminal of claim 3, wherein the wireless communication unit further: receives other goods information from the goods or a Near Field Communication (NFC) tag located in vicinity of the goods; and generates the goods information by using visual information received via the camera or audio information received via the microphone when a time at which the other goods information was received corresponds to a time at which the visual or audio information was received.

### Claim 5 (depends on 4)

The mobile terminal of claim 4, wherein the controller further utilizes the received visual or audio information to generate the goods information when a time during which the mobile terminal has remained near the goods satisfies a preset condition.

### Claim 6 (depends on 1)

The mobile terminal of claim 1, wherein the received facial image is captured by a camera in the POS terminal and is received from the POS terminal when the payment is made.

### Claim 7

A method for controlling a mobile terminal, the method comprising:
receiving a facial image of a user related to goods, from a point-of-sale (POS) terminal when a payment is made between the mobile terminal and the POS terminal;
receiving a facial image from a wireless communication module located in a store when the user of the mobile terminal leaves the store without purchasing the goods;
analyzing the facial image received from the POS terminal or the wireless communication module using a face analysis algorithm to obtain facial expression information corresponding to the facial image received from the POS terminal or the wireless communication module according to the analysis;
generating goods information by associating the obtained facial expression information with other information related to the goods and storing the goods information in the database;
searching for goods information stored in the database by analyzing a facial image of the user received via a camera of the mobile terminal based on the face analysis algorithm in order to extract facial expression information corresponding to the facial image of the user received via the camera and feeling information, the feeling information extracted based on the facial expression information corresponding to the facial image received via the camera;
and outputting, via a display of the mobile terminal, the searched goods information stored in the database corresponding to at least the extracted facial expression information or feeling information,
wherein the analyzing the facial imago received via the camera comprises:
outputting, via the display, an analysis result for the user to confirm whether the facial expression information corresponding to the facial image received via the camera corresponds to the analysis result when the analysis of the facial image received via the camera is complete;
outputting, via the display, other feeling items according to the user's response with regard to confirmation; and
matching a feeling corresponding to a feeling item selected by the user from the other feeling items to the facial expression information corresponding to the facial image received via the camera based on the facial image received via the camera.

### Claim 8 (depends on 7)

The method of claim 7, wherein generating the goods information comprises:
obtaining a user's gesture from an image received from the POS terminal;
combining the facial expression information with the obtained user's gesture to extract the feeling information related to the goods; and
matching the extracted feeling information to the other information related to the goods.

### Claim 9 (depends on 7)

The method of claim 7, further comprising:
sensing that the mobile terminal is positioned in the store based on communication between a wireless communication unit of the mobile terminal and the wireless communication module located in the store;
activating at least the camera or a microphone when the mobile terminal is positioned in the store; and
generating the goods information by using information received via at least the camera or the microphone.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
receiving other goods information from the goods or a Near Field Communication (NFC) tag located in vicinity of the goods; and
generating the goods information by using visual information received via the camera or audio information received via the microphone when a time at which the other goods information was received corresponds to a time at which the visual or audio information was received.

### Claim 11 (depends on 7)

The method of claim 7, wherein the received facial image is captured by a camera in the POS terminal and is received from the POS terminal when the payment is made.