# US 11,113,813 B2 · Evaluating a condition of a person > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,813 B2 - Document ID: US11113813B2 - Status: Granted - Kind: B2 - Application number: 16/597,266 - Filed: 2019-10-09 - Published: 2020-04-16 - Granted: 2021-09-07 - Earliest claimed: 2018-10-15 - Assignees: Siemens Healthcare GmbH - Applicants: Siemens Healthcare GmbH - Inventors: Alper Alsan; Elif Kirmizi-Alsan - Web page: https://patentgrep.com/patent/US11113813B2 ## CPC - G06T 7/0016: {involving temporal comparison} - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - A61B 5/4082: {Diagnosing or monitoring movement diseases, e.g. Parkinson, Huntington or Tourette} - A61B 5/7267: {involving training the classification device} - G06V 40/176: {Dynamic expression} - G16H 30/40: for processing medical images, e.g. editing - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T 2207/30004: Biomedical image processing - G06T 2207/30201: Face - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A computer-implemented method is for evaluating a condition of a person. The method includes determining at least one characteristic of a first facial expression of at least a mouth of the person, at a first time, based at least on a first image previously captured; determining at least one characteristic of a second facial expression of at least a mouth of a person, at a second time, based at least on a second image previously captured, the first facial expression and the second facial expression being of a same first type of facial expression; determining at least one difference between the at least one characteristic of the first facial expression determined and the at least one characteristic of the second facial expression determined; and generating an output signal indicating the condition of the person based at least on the at least one difference determined. ## Claims ### Claim 1 A computer-implemented method for evaluating a condition of a person, comprising: acquiring a first image of at least a mouth of the person at a first time at which the person performs a first facial expression; determining at least one characteristic of the first facial expression based at least on the first image acquired; acquiring a second image of at least the mouth of the person at a second time at which the person performs a second facial expression, the first facial expression and the second facial expression being of a same first type of facial expression; determining at least one characteristic of the second facial expression based at least on the second image acquired; determining at least one difference between the at least one characteristic of the first facial expression determined and the at least one characteristic of the second facial expression determined; and generating an output signal indicating the condition of the person based at least on the at least one difference determined, wherein at least one of the acquiring of the first image and the acquiring of the second image includes acquiring a video of at least the mouth of the person and determining a characterizing frame of the video acquired as at least one of the first image acquired and the second image acquired, and wherein one of the characterizing frame of the video acquired is automatically determined based on a criterion, or a user or the person is prompted to select one of the frames of the video acquired as the characterizing frame. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein of at least one of the determining of the at least one characteristic of the first facial expression and the determining of the at least one characteristic of the second facial expression, is performed using a trained machine learning entity. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein at least one of a first characteristic of the at least one characteristic of the first facial expression and a first characteristic of the at least one characteristic of the second facial expression is a digital representation of a mouth line of the mouth of the person. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein at least one of a second characteristic of the at least one characteristic of the first facial expression and a second characteristic of the at least one characteristic of the second facial expression is a state of at least one facial muscle of the person. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the criterion is that the characterizing frame is a frame in which at least one of the at least one characteristic of the first facial expression and at least one characteristic of the second facial expression has an extremum, an average or a median within the video acquired. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising: acquiring a first comparison image of at least the mouth of the person at the first time at which the person performs a specific second type of facial expression, wherein the determining of the at least one characteristic of the first facial expression is additionally based on the first comparison image acquired. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein the acquiring of the first image and the acquiring of the first comparison image comprises: acquiring a video of at least the mouth of the person; determining a first characterizing frame of the video acquired as the first image to be acquired; and determining a second characterizing frame of the video acquired as the first comparison image to be acquired. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein a third characteristic of the at least one characteristic of the first facial expression and the at least one characteristic of the second facial expression is a time from a starting frame of the video acquired, to the frame determined as the respective first image or second image. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein after the first image and the second image, further images at further times at which the person is performing further facial expressions of the same first type of facial expression are acquired within a respective time interval from one another; wherein, based on each of the further images, the at least one characteristic of each of respective further facial expression in a corresponding respective further image of the further images, is determined; wherein at least one further difference between the determined at least one characteristic of the further facial expressions and at least one respective previously determined at least one characteristic is determined; and wherein the output signal is additionally based on the at least one determined further difference. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein at least one of the first type of facial expression type comprises at least one of a smiling facial expression, a crying facial expression, a screaming facial expression or a frowning facial expression and the second type of facial expression comprises a neutral facial expression of the person. ### Claim 11 (depends on 1) A non-transitory computer-readable data storage medium storing executable program code configured to, when executed by at least one processor, perform the method of claim 1. ### Claim 12 A system for evaluating a condition of a person, comprising: an image capturing device configured to: acquire a first image of at least a mouth of the person at a first time at which the person performs a first facial expression, and acquire a second image of at least the mouth of the person at a second time at which the person performs a second facial expression, the first facial expression and the second facial expression being of a same first type of facial expression, wherein the image capturing device, to acquire at least one of the first image and at least one of the second image, is configured to acquire a video of at least the mouth of the person; and a computing device configured to: determine, based at least on the first image acquired, at least one characteristic of the first facial expression; determine, based on at least the second image acquired, the at least one characteristic of the second facial expression; determine a difference between the at least one characteristic of the first facial expression determined and the at least one characteristic of the second facial expression determined; and generate an output signal, indicating the condition of the person based at least on the difference determined, wherein the computing device is further configured to determine a characterizing frame of the video acquired as at least one of the first image acquired and the second image acquired, and wherein one of the computing device is further configured to determine the characterizing frame of the video acquired, automatically, based on a criterion, or the computing device is further configured to prompt a user or the person to select one of the frames of the video acquired as the characterizing frame. ### Claim 13 (depends on 12) The system of claim 12, further comprising: an interface, to output the output signal. ### Claim 14 (depends on 12) The system of claim 12, wherein the criterion is that the characterizing frame is a frame in which at least one of the at least one characteristic of the first facial expression and at least one characteristic of the second facial expression has an extremum, an average or a median within the video acquired. ### Claim 15 (depends on 12) The system of claim 12, wherein a third characteristic of the at least one characteristic of the first facial expression and the at least one characteristic of the second facial expression is a time from a starting frame of the video acquired, to the frame determined as the respective first image or second image. ### Claim 16 A computer-implemented method for evaluating a condition of a person, comprising: determining at least one characteristic of a first facial expression of at least a mouth of the person, at a first time at which the person performs the first facial expression, based at least on a first image previously captured; determining at least one characteristic of a second facial expression of at least a mouth of a person, at a second time at which the person performs the second facial expression, based at least on a second image previously captured, the first facial expression and the second facial expression being of a same first type of facial expression; determining at least one difference between the at least one characteristic of the first facial expression determined and the at least one characteristic of the second facial expression determined; and generating an output signal indicating the condition of the person based at least on the at least one difference determined, wherein at least one of previously capturing of the first image and previously capturing of the second image includes previously capturing a video of at least the mouth of the person and determining a characterizing frame of the video acquired as at least one of the first image acquired and the second image acquired, and wherein one of the characterizing frame of the video acquired is automatically determined based on a criterion, or a user or the person is prompted to select one of the frames of the video acquired as the characterizing frame. ### Claim 17 (depends on 16) The computer-implemented method of claim 16, wherein of at least one of the determining of the at least one characteristic of the first facial expression and the determining of the at least one characteristic of the second facial expression, is performed using a trained machine learning entity. ### Claim 18 (depends on 16) The computer-implemented method of claim 16, wherein the criterion is that the characterizing frame is a frame in which at least one of the at least one characteristic of the first facial expression and at least one characteristic of the second facial expression has an extremum, an average or a median within the video acquired. ### Claim 19 (depends on 16) The computer-implemented method of claim 16, wherein the criterion is that the characterizing frame is a frame in which at least one of the at least one characteristic of the first facial expression and at least one characteristic of the second facial expression has an extremum, an average or a median within the video acquired. ### Claim 20 (depends on 16) The computer-implemented method of claim 16, wherein a third characteristic of the at least one characteristic of the first facial expression and the at least one characteristic of the second facial expression is a time from a starting frame of the video acquired, to the frame determined as the respective first image or second image. ### Claim 21 A non-transitory computer-readable data storage medium storing executable program code configured to, when executed by at least one processor, perform at least: determining at least one characteristic of a first facial expression of at least a mouth of a person, at a first time at which the person performs the first facial expression, based at least on a first image previously captured; determining at least one characteristic of a second facial expression of at least a mouth of a person, at a second time at which the person performs the second facial expression, based at least on a second image previously captured, the first facial expression and the second facial expression being of a same first type of facial expression; determining at least one difference between the at least one characteristic of the first facial expression determined and the at least one characteristic of the second facial expression determined; and generating an output signal indicating a condition of the person based at least on the at least one difference determined, wherein at least one of previously capturing of the first image and previously capturing of the second image includes previously capturing a video of at least the mouth of the person and determining a characterizing frame of the video acquired as at least one of the first image acquired and the second image acquired, and wherein one of the characterizing frame of the video acquired is automatically determined based on a criterion, or a user or the person is prompted to select one of the frames of the video acquired as the characterizing frame. ### Claim 22 (depends on 21) The non-transitory computer-readable data storage medium of claim 21, wherein the criterion is that the characterizing frame is a frame in which at least one of the at least one characteristic of the first facial expression and at least one characteristic of the second facial expression has an extremum, an average or a median within the video acquired. ### Claim 23 (depends on 21) The non-transitory computer-readable data storage medium of claim 21, wherein a third characteristic of the at least one characteristic of the first facial expression and the at least one characteristic of the second facial expression is a time from a starting frame of the video acquired, to the frame determined as the respective first image or second image.