# US 2021/0240970 A1 · Terminal apparatus, function suggestion method, storage medium, and server apparatus > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0240970 A1 - Document ID: US20210240970A1 - Status: Published application - Kind: A1 - Application number: 17/164,943 - Filed: 2021-02-02 - Published: 2021-08-05 - Earliest claimed: 2020-02-05 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Atsushi MIKAWA - Web page: https://patentgrep.com/patent/US20210240970A1 ## CPC - G06V 10/82: using neural networks - G06V 30/19173: {Classification techniques} - G06V 30/194: References adjustable by an adaptive method, e.g. learning - G06V 40/165: {using facial parts and geometric relationships} - G06V 40/172: {Classification, e.g. identification} - G06V 40/174: {Facial expression recognition} - G06V 40/176: {Dynamic expression} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A terminal apparatus includes a controller configured to transmit, to a server apparatus, a suggestion request requesting a suggestion of a function to be invoked, receive, from the server apparatus, a suggestion response that specifies a suggested function determined by the server apparatus; and invoke the suggested function. The suggest ion request includes face image data of a user. The suggested function is determined by applying the face image data to a learned model. The controller is configured to; evaluate, based on face images of the user, a degree of change in facial expressions before and after invoking the suggested function; and transmit, to the server apparatus, a learning request requesting re-learning of the learned model when the evaluated degree of change in the facial expressions satisfies a predetermined condition. ## Claims ### Claim 1 A terminal apparatus comprising a communication interface configured to communicate with a server apparatus and a controller, wherein the controller is configured to: transmit, via the communication interface to the server apparatus, a suggestion request requesting a suggestion of a function to be invoked in the terminal apparatus: receive, via the communication interface from the server apparatus, a suggestion response that specifies a suggested function determined by the server apparatus in response to the suggestion request; and invoke the suggested function specified by the suggestion response, wherein the suggestion request includes face image data of a user of the terminal apparatus, and the suggested function is determined by applying the face image data to a learned model at the server apparatus, wherein the controller is configured to: evaluate, based on face images of the user, a degree of change in facial expressions of the user before and after invoking the suggested function; and transmit, via the communication interface to the server apparatus, a learning request requesting re-learning of the learned model when the evaluated degree of change in the facial expressions satisfies a predetermined condition. ### Claim 2 (depends on 1) The terminal apparatus according to claim 1, wherein the learning request includes user identification information that identifies the user, and the re-learning of the learned model is performed per user to acquire a user-specific model. ### Claim 3 (depends on 1) The terminal apparatus according to claim 1, wherein the controller is configured to: obtain a first evaluated value by applying a first face image of the user before invoking the suggested function to an evaluation model for calculating an evaluated value of change in facial expression of the user based on a face image of the user; and obtain a second evaluated value by applying a second face image of the user after invoking the suggested function to the evaluation model; and wherein the degree of change in the facial expressions corresponds to change from the first evaluated value to the second evaluated value. ### Claim 4 (depends on 3) The terminal apparatus according to claim 3, wherein the predetermined condition includes a difference between the first evaluated value and the second evaluated value being above a first threshold. ### Claim 5 (depends on 3) The terminal apparatus according to claim 3, wherein the evaluation model is acquired in advance through machine learning using images for learning which are face images of a plurality of persons labeled with degrees of satisfaction as training data, and the predetermined condition includes that the change from the first evaluated value to the second evaluated value indicates that the degree of satisfaction has improved. ### Claim 6 (depends on 4) The terminal apparatus according to claim 4, wherein the re-learning of the learned model is performed using the first face image as an image for learning with the suggested function as right function to be suggested. ### Claim 7 (depends on 3) The terminal apparatus according to claim 3, wherein the controller is configured to transmit the suggestion request to the server apparatus when the first evaluated value obtained by applying the first face image to the evaluation model is below a second threshold ### Claim 8 (depends on 1) The terminal apparatus according to claim 1, wherein the suggestion request induces application identification information dial identifies an application running in the terminal apparatus, and the suggested function determined by the server apparatus is different depending on the application running in the terminal apparatus ### Claim 9 (depends on 1) The terminal apparatus according to claim 1, further comprising an image capture device configured to capture a face image of the user. ### Claim 10 A function suggestion method performed by a terminal apparatus comprising; transmitting, from the terminal apparatus to a server apparatus, a suggestion request requesting a suggestion of a function to be invoked in the terminal apparatus: receiving, by the terminal apparatus from the server apparatus, a suggestion response that specifies n suggested function determined by the server apparatus in response to the suggestion request; and invoking, by the terminal apparatus, the suggested function specified by the suggestion response. wherein the suggestion request includes face image data of a user of the terminal apparatus, and the suggested function is determined by applying the face image data to a learned model at the server apparatus. wherein the function suggestion method further comprises: evaluating, based on face images of the user, a degree of change in facial expressions of the user before and after invoking the suggested function; and transmitting, from the terminal apparatus to the server apparatus, a learning request requesting re-teaming of the learned model when the evaluated degree of change in the facial expressions satisfies a predetermined condition. ### Claim 11 A non-transitory computer-readable storage medium having stored therein a computer program which, when executed by a processor of a terminal apparatus, causes the terminal apparatus to: transmit, to the server apparatus, a suggestion request requesting a suggestion of a function to be invoked in the terminal apparatus, the suggestion request including face image data of a user of the terminal apparatus; receive, from the server apparatus, a suggestion response that specifies a suggested function determined by the server apparatus in response to the suggestion request, the suggested function being determined by applying the face image data to a learned model at the server apparatus; invoke the suggested function specified by the suggestion response; evaluate, based on face images of the user, a degree of change in facial expressions of the user before and after invoking the suggested function; and transmit, to the server apparatus, a learning request requesting re-learning of the learned model when the evaluated degree of change in the facial expressions satisfies a predetermined condition. ### Claim 12 A server apparatus comprising a communication interlace configured to communicate with a terminal apparatus and a controller, wherein the controller is configured to: receive, via the communication interface from the terminal apparatus, a suggestion request requesting a suggestion of a function to be invoked in the terminal apparatus, the suggestion request including face image data of a user of the terminal apparatus; determine the function to be invoked in the terminal apparatus by applying the face image data to a learned model; transmit via the communication interface to the terminal apparatus, a suggestion response that specifies the determined function to thereby cause the terminal apparatus to invoke the function; and in a case where a learning request requesting red earning of the learned model is received via the communication interface from the terminal apparatus, perform the re-learning of the learned model using a face image indicated by the face image data as an image for learning with the determined function as right function. ### Claim 13 A terminal apparatus comprising. an image obtaining unit configured to image a user to thereby obtain a face image of the user; a control unit configured to invoke, among functions of the terminal apparatus, a function determined by applying the face image to a learned model; and a learning unit configured to perform re-learning of the learned model when a degree of change in facial expressions of the user before and after invoking the function satisfies a predetermined condition.