Telehealth suite for psychiatry digital phenotyping

A system for determining a digital phenotype of a target individual, the system comprising:

• a data processing handler configured to receive a data stream, wherein the data stream comprises at least one of:
• audio data comprising a vocal feature of the target individual,
• visual data comprising at least one of:
• an image of a face of the target individual, an image of a body of the target individual,
• contact sensing data of the target individual,
• physiological data of the target individual, and
• text data comprising a health record of the target individual;
• a first machine learning model is configured to:
• receive as input, the visual data from the data processing handler;
• and
• extract the face of the target individual;
• wherein the first machine learning model comprises at least one of:
• a neural network configured to perform a neural network method, a deep neural network, or a transformer model;
• a second machine learning model configured to:
• receive as input the visual data from the data processing handler;
• determine a first intermediate representation of at least one of:
• an emotional affect of the target individual using the visual data or the body of the target individual using the visual data;
• and
• determine an output estimation comprising at least one of:
• a facial action unit intensity, or a valence and arousal estimation pair;
• wherein the second machine learning model comprises at least one of:
• a neural network configured to perform a neural network method, a deep neural network, or a transformer model;
• a third machine learning model configured to:
• receive as input the audio data from the data processing handler;
• determine a second intermediate representation of a voice of the target individual using the audio data;
• and
• determine an output estimation comprising at least one of:
• an emotion classification or a voice prosody feature,
• wherein the third machine learning model comprises at least one of:
• a neural network configured to perform a neural network method, a deep neural network, or a transformer model;
• a fourth machine learning model configured to:
• receive as input the visual data from the data processing handler;
• and
• determine a third intermediate representation of at least one of:
• the face of the target individual using the visual data or the body of the target individual using the visual data;
• determine an output estimation comprising at least one of:
• a heart rate of the target individual, a raw blood volume pulse signal of the target individual, or a heart rate variability of the target individual,
• wherein the fourth machine learning model comprises at least one of:
• a neural network configured to perform a neural network method, a deep neural network, or a transformer model;
• a fifth machine learning model configured to:
• receive as input at least one of:
• the visual data from the data processing handler or the text data from the data processing handler;
• and
• determine a fourth intermediate representation of least one of:
• the health record of the target individual using the text data or biometric data using the text data, or
• an image and prompt correlation of at least one of:
• the health record of the target individual using the visual and text data or biometric data using the visual and text data,
• wherein the fifth machine learning model is configured to execute a zero-shot contrastive pre-training method;
• a sixth machine learning model configured to:
• receive as input at least one of:
• the first intermediate representation, the second intermediate representation, the third intermediate representation, or the fourth intermediate representation;
• and
• determine, based on the received input and by applying a weight or transform logic, at least one of:
• an output time series forecast or a data imputation, and
• wherein the sixth machine learning model comprises at least one of:
• a neural network configured to perform a neural network method, a recurrent neural network-based model, a transformer model configured to employ a probsparse method or a self-attention method, or a large language model;
• a seventh machine learning model configured to:
• receive as input at least one of:
• the first intermediate representation, the second intermediate representation, the third intermediate representation, the fourth intermediate representation, the output time series forecast, or the data imputation;
• and
• determine a digital phenotype;
• an output handler configured to:
• receive as input at least one of:
• the first intermediate representation, the second intermediate representation, the third intermediate representation, the fourth intermediate representation, the output time series, the data imputation, the contact sensing data, the biometric data, or the digital phenotype;
• and
• perform at least one of:
• transmitting the digital phenotype to a computing device or a remote display platform;
• displaying the digital phenotype as a visual notification;
• or
• storing the digital phenotype in a memory location.

A method for determining a digital phenotype of a target individual, the method comprising:

• receiving, by a data processing handler, a data stream,
• wherein the data stream comprises at least one of:
• audio data of the target individual, visual data depicting the target individual, and text data comprising at least one of:
• a health record of the target individual, an audio recording transcription, and a textual note;
• receiving as input, by a machine learning model, at least one of the audio data, the visual data, and the text data;
• determining, dependent on receipt of the visual data and by the machine learning model, a first output estimation comprising at least one of:
• a facial action unit intensity, or a valence and arousal estimation pair;
• determining, dependent on receipt of the audio data and by the machine learning model, a second output estimation comprising at least one of:
• an emotion classification or a voice prosody feature;
• determining, dependent on receipt of the visual data and by the machine learning model, a third output estimation comprising at least one of:
• a heart rate of the target individual, a raw blood volume pulse signal of the target individual, or a heart rate variability of the target individual;
• determining, dependent on receipt of the text data and by the machine learning model, a fourth output estimation comprising at least one of:
• an image and prompt correlation utilizing the health record of the target individual or the textual note, and a sentiment analysis of the audio recording transcription;
• determining, by the machine learning model, a digital phenotype based on at least one of:
• the first output estimation, the second output estimation, the third output estimation, and the fourth output estimation;
• and
• outputting, by an output handler, the digital phenotype.