Neural network architecture for movement analysis

A computer-implemented system, comprising:

• one or more computers;
• and
• one or more non-tangible, computer-readable storage media interoperably coupled with the one or more computers and storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• obtaining a video, wherein the video is segmented into a plurality of sequences corresponding to different facial states performed by a patient in the video;
• for each sequence,
• tracking displacement of a plurality of groups of landmarks of a face of the patient, to obtain, for each group of the plurality of groups, one or more displacement measures characterizing positions of the landmarks of the group,
• for each group of the plurality of groups,
• providing the one or more displacement measures corresponding to the group into a corresponding neural network, to obtain a landmark feature corresponding to the group, wherein the neural networks corresponding to each group of the plurality of groups are different from one another, and
• determining a sequence score for the sequence based on a plurality of landmark features corresponding to the plurality of groups;
• providing a plurality of sequence scores corresponding to the plurality of sequences into a machine learning component, to obtain a patient score;
• and
• identifying a disease state of the patient based on the patient score.

One or more non-transitory, computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

• obtaining a video, wherein the video is segmented into a plurality of sequences corresponding to different facial states performed by a patient in the video;
• for each sequence,
• tracking displacement of a plurality of groups of landmarks of a face of the patient, to obtain, for each group of the plurality of groups, one or more displacement measures characterizing positions of the landmarks of the group,
• for each group of the plurality of groups,
• providing the one or more displacement measures corresponding to the group into a corresponding neural network, to obtain a landmark feature corresponding to the group, wherein the neural networks corresponding to each group of the plurality of groups are different from one another, and
• determining a sequence score for the sequence based on a plurality of landmark features corresponding to the plurality of groups;
• providing a plurality of sequence scores corresponding to the plurality of sequences into a machine learning component, to obtain a patient score;
• and
• identifying a disease state of the patient based on the patient score.