# US 10,872,168 B1 · Method and system for providing user notification when personal information is used in voice control device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,872,168 B1
- Document ID: US10872168B1
- Status: Granted
- Kind: B1
- Application number: 16/842,834
- Filed: 2020-04-08
- Published: 2020-12-10
- Granted: 2020-12-22
- Earliest claimed: 2019-06-07
- Assignees: Piamond Corp.
- Applicants: Piamond Corp.
- Inventors: Jinhong Yang
- Web page: https://patentgrep.com/patent/US10872168B1

## CPC

- G06F 21/6245: {Protecting personal data, e.g. for financial or medical purposes}
- G06F 16/635: Filtering based on additional data, e.g. user or group profiles
- G06Q 10/107: Computer-aided management of electronic mailing [e-mailing]
- G06Q 50/265: {Personal security, identity or safety}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 17/22: Interactive procedures; Man-machine interfaces
- G10L 17/00: Speaker identification or verification techniques
- G10L 2015/223: {Execution procedure of a spoken command}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Disclosed are a method and system for providing user notification when personal information is used in a speech controller. A method of providing user notification may include receiving information monitored for reference to personal information in a process of processing a user's query in a voice service, storing a personal information utilization history for each user and for each service based on the monitored information, determining the suitability of the reference to the personal information based on the monitored information, determining whether the personal information is included in a response when generating the response to the user's query, generating and providing guide information indicating that the personal information is included in the response if the personal information is included in the response, and providing the response to the user's query based on feedback from the user for the guide information.

## Claims

### Claim 1

A method of providing user notification in a computer device including at least one processor, the method comprising:
receiving, by the at least one processor, information monitored for reference to personal information in a process of processing a user's query in a voice service;
storing, by the at least one processor, a personal information utilization history for each user and for each service based on the monitored information;
determining, by the at least one processor, suitability of the reference to the personal information based on the monitored information;
determining, by the at least one processor, whether the personal information is included in a response when generating the response to the user's query;
generating and providing, by the at least one processor, guide information indicating that the personal information is included in the response if the personal information is included in the response; and
providing, by the at least one processor, the response to the user's query based on feedback from the user for the guide information,
wherein determining the suitability comprises determining whether a service policy has been violated in relation to access to the personal information of the user identified based on the monitored information.

### Claim 2 (depends on 1)

The method of claim 1, wherein the personal information comprises at least one of personally identifiable information (PII) and potential personally identifiable information (PPII).

### Claim 3 (depends on 1)

The method of claim 1, wherein the reference to the personal information is performed at least one reference point of a device that captures a speech corresponding to the user's query, the voice service that processes the captured speech, and an external service invoked by the voice service.

### Claim 4 (depends on 1)

The method of claim 1, wherein determining the suitability comprises determining whether user consent has been performed normally in relation to access to the personal information of the user identified based on the monitored information.

### Claim 5 (depends on 1)

The method of claim 1, wherein providing the response to the user's query comprises providing the response to the user's query through a response channel preset by the user or a response channel configured through the feedback from the user.

### Claim 6 (depends on 5)

The method of claim 5, wherein the response channel comprises two or more of a speech output channel through a speaker included in a device of the user, a push notification channel for an application installed in the device of the user, a message transmission channel using a short message service (SMS) for the device of the user, a message transmission channel using an e-mail account of the user, and an information display channel through a screen of another device related to the user.

### Claim 7

A method of providing user notification in a computer device including at least one processor, the method comprising:
receiving, by the at least one processor, information monitored for reference to personal information in a process of processing a user's query in a voice service;
storing, by the at least one processor, a personal information utilization history for each user and for each service based on the monitored information;
determining, by the at least one processor, suitability of the reference to the personal information based on the monitored information;
determining, by the at least one processor, whether the personal information is included in a response when generating the response to the user's query;
generating and providing, by the at least one processor, guide information indicating that the personal information is included in the response if the personal information is included in the response; and
providing, by the at least one processor, the response to the user's query based on feedback from the user for the guide information,
wherein the personal information of the user is managed based on a grade or score according to at least one of a group for each user, a policy for each service and personal information handling regulations suggested in a corresponding country, and
determining the suitability comprises determining the suitability of the reference to the personal information based on a grade or score calculated for the reference to the personal information and a grade or score for the personal information of the user.

### Claim 8

A method of providing user notification in a computer device including at least one processor, the method comprising:
receiving, by the at least one processor, information monitored for reference to personal information in a process of processing a user's query in a voice service;
storing, by the at least one processor, a personal information utilization history for each user and for each service based on the monitored information;
determining, by the at least one processor, suitability of the reference to the personal information based on the monitored information;
determining, by the at least one processor, whether the personal information is included in a response when generating the response to the user's query;
generating and providing, by the at least one processor, guide information indicating that the personal information is included in the response if the personal information is included in the response; and
providing, by the at least one processor, the response to the user's query based on feedback from the user for the guide information,
wherein generating and providing the guide information comprises generating the guide information based on query words defined by the voice service or a personal information usage policy of the voice service, and
the guide information comprises related service provisions for responding to the user's additional query about the guide information.

### Claim 9

A method of providing user notification in a computer device including at least one processor, the method comprising:
receiving, by the at least one processor, information monitored for reference to personal information in a process of processing a user's query in a voice service;
storing, by the at least one processor, a personal information utilization history for each user and for each service based on the monitored information;
determining, by the at least one processor, suitability of the reference to the personal information based on the monitored information;
determining, by the at least one processor, whether the personal information is included in a response when generating the response to the user's query;
generating and providing, by the at least one processor, guide information indicating that the personal information is included in the response if the personal information is included in the response; and
providing, by the at least one processor, the response to the user's query based on feedback from the user for the guide information,
wherein a server providing the voice service is configured to:
recognize the user based on speech information corresponding to the user's query,
issue a permission key capable of accessing the personal information of the recognized user,
check user consent and a service policy based on the permission key when a command selected based on the speech information accesses the personal information of the user,
issue an access token having a grade based on the user consent and the service policy,
generate a response related to the personal information of the user by confirming the grade of the access token when a service using the access token is invoked.

### Claim 10

A method of providing user notification in a computer device including at least one processor, the method comprising:
receiving, by the at least one processor, information monitored for reference to personal information in a process of processing a user's query in a voice service;
storing, by the at least one processor, a personal information utilization history for each user and for each service based on the monitored information;
determining, by the at least one processor, suitability of the reference to the personal information based on the monitored information;
determining, by the at least one processor, whether the personal information is included in a response when generating the response to the user's query;
generating and providing, by the at least one processor, guide information indicating that the personal information is included in the response if the personal information is included in the response;
providing, by the at least one processor, the response to the user's query based on feedback from the user for the guide information;
outputting, by the at least one processor, a speech for requesting consent for usage and exposure of the personal information;
receiving, by the at least one processor, a response to the output speech; and
determining, by the at least one processor, whether consent for the usage and exposure of the personal information has been given based on contents of the received response.

### Claim 11

A method of providing user notification in a computer device including at least one processor, the method comprising:
receiving, by the at least one processor, information monitored for reference to personal information in a process of processing a user's query in a voice service;
storing, by the at least one processor, a personal information utilization history for each user and for each service based on the monitored information;
determining, by the at least one processor, suitability of the reference to the personal information based on the monitored information;
determining, by the at least one processor, whether the personal information is included in a response when generating the response to the user's query;
generating and providing, by the at least one processor, guide information indicating that the personal information is included in the response if the personal information is included in the response;
providing, by the at least one processor, the response to the user's query based on feedback from the user for the guide information; and
providing, by the at least one processor, the user with information on an issue from a personal information generation viewpoint, which is capable of occurring, in advance or at timing when a third party service is provided by an external service provider, when the at least one processor operates in conjunction with the third party service.

### Claim 12

A method of providing user notification in a computer device including at least one processor, the method comprising:
receiving, by the at least one processor, information monitored for reference to personal information in a process of processing a user's query in a voice service;
storing, by the at least one processor, a personal information utilization history for each user and for each service based on the monitored information;
determining, by the at least one processor, suitability of the reference to the personal information based on the monitored information;
determining, by the at least one processor, whether the personal information is included in a response when generating the response to the user's query;
generating and providing, by the at least one processor, guide information indicating that the personal information is included in the response if the personal information is included in the response;
providing, by the at least one processor, the response to the user's query based on feedback from the user for the guide information;
monitoring, by the at least one processor, a number of times that a response including the personal information is output to the user and sensitivity of the personal information including the output response; and
analyzing, by the at least one processor, the monitored number and sensitivity.

### Claim 13

A method of providing user notification in a computer device including at least one processor, the method comprising:
receiving, by the at least one processor, information monitored for reference to personal information in a process of processing a user's query in a voice service;
storing, by the at least one processor, a personal information utilization history for each user and for each service based on the monitored information;
determining, by the at least one processor, suitability of the reference to the personal information based on the monitored information;
determining, by the at least one processor, whether the personal information is included in a response when generating the response to the user's query;
generating and providing, by the at least one processor, guide information indicating that the personal information is included in the response if the personal information is included in the response;
providing, by the at least one processor, the response to the user's query based on feedback from the user for the guide information; and
calculating, by the at least one processor, a number of times that the personal information is exposed through an application and sensitivity of the exposed personal information through matching between permission information of the application installed in a device of the user and a PII provided by a standard organization.

### Claim 14 (depends on 1)

A non-transitory computer-readable recording medium in which a computer program for executing a method according to claim 1 in a computer device is written.

### Claim 15

A computer device, comprising:
at least one processor configured to execute a computer-readable command,
wherein the at least one processor is configured to:
receive information monitored for reference to personal information in a process of processing a user's query in a voice service;
store a personal information utilization history for each user and for each service based on the monitored information;
determine suitability of the reference to the personal information based on the monitored information;
determine whether the personal information is included in a response when generating the response to the user's query;
generate and provide guide information indicating that the personal information is included in the response; and
provide the response to the user's query based on feedback from the user for the guide information,
wherein the at least one processor is configured to determine whether a service policy has been violated in relation to access to the personal information of the user identified based on the monitored information.

### Claim 16 (depends on 15)

The computer device of claim 15, wherein the reference to the personal information is performed at least one reference point of a device that captures a speech corresponding to the user's query, the voice service that processes the captured speech, and an external service invoked by the voice service.

### Claim 17 (depends on 15)

The computer device of claim 15, wherein the at least one processor is configured to determine whether user consent has been performed normally in relation to access to the personal information of the user identified based on the monitored information.

### Claim 18

A computer device, comprising:
at least one processor configured to execute a computer-readable command,
wherein the at least one processor is configured to:
receive information monitored for reference to personal information in a process of processing a user's query in a voice service;
store a personal information utilization history for each user and for each service based on the monitored information;
determine suitability of the reference to the personal information based on the monitored information;
determine whether the personal information is included in a response when generating the response to the user's query;
generate and provide guide information indicating that the personal information is included in the response; and
provide the response to the user's query based on feedback from the user for the guide information,
wherein the personal information of the user is managed based on a grade or score according to at least one of a group for each user, a policy for each service and personal information handling regulations suggested in a corresponding country, and
the at least one processor is configured to determine the suitability of the reference to the personal information based on a grade or score calculated for the reference to the personal information and a grade or score for the personal information of the user.