# US 12,119,115 B2 · Systems and methods for self-supervised learning based on naturally-occurring patterns of missing data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,119,115 B2 - Document ID: US12119115B2 - Status: Granted - Kind: B2 - Application number: 18/156,010 - Filed: 2023-01-18 - Published: 2023-08-03 - Granted: 2024-10-15 - Earliest claimed: 2022-02-03 - Assignees: Evidation Health, Inc. - Applicants: Evidation Health, Inc. - Inventors: Luca Foschini; Filip Jankovic; Raghunandan Melkote Kainkaryam; Juan Ignacio Oguiza Mendez; Arinbjörn Kolbeinsson - Web page: https://patentgrep.com/patent/US12119115B2 ## CPC - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - G16H 50/30: for calculating health indices; for individual health risk assessment - G16H 40/67: for remote operation - G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract Disclosed is a method comprising accessing, by a machine learning system, a set of data records for a plurality of users, the data records representative of physical statistics measured for each of the plurality of users over a time period. At least a subset of the data records comprises patterns of missing data for at least a portion of the time period. The method also comprises generating a set of masked data records by masking a subset of the data records in accordance with a pattern of natural missingness from a data record. The method also comprises generating, by the machine learning system, a set of learned representations from at least the set of masked data records. Finally, the method comprises fine tuning, by the machine learning system, a machine learning model using the set of learned representations, the machine learning model configured to perform a downstream machine learning task. ## Claims ### Claim 1 A method comprising: (a) identifying, based at least in part on one or more demographics of a target user, a population from a larger group of users, wherein the population comprises one or more digital twins of the target user, and wherein the population is characterized by having a common demographic; (b) accessing, by a machine learning system, a set of data records for a plurality of users of the population, the set of data records representative of physical statistics measured for each user of the plurality of users of the population over a time period, wherein the physical statistics for each user of the plurality of users of the population are measured using a wearable device associated with each user of the plurality of users of the population; (c) generating a set of masked data records by masking at least a subset of the set of data records in accordance with a pattern of missingness from the set of data records, wherein the pattern of missingness from the set of data records corresponds to periods of disuse or deactivation of the wearable device associated with each user of the plurality of users of the population; (d) generating, by the machine learning system, a plurality of learned representations from at least the set of masked data records; and (e) fine tuning, by the machine learning system, a machine learning model using the plurality of learned representations, the machine learning model configured to perform a downstream machine learning task that comprises imputing missing data from a wearable device associated with the target user, thereby generating complete data for the target user. ### Claim 2 (depends on 1) The method of claim 1, wherein a data record of the subset of the set of data records comprises missing data different from the pattern of missingness. ### Claim 3 (depends on 2) The method of claim 2, further comprising using a learned representation of the plurality of learned representations to identify another subset of the set of data records using one or more clustering or segmentation techniques to one or more of: (i) perform event detection, (ii) detect or predict onset of an acute health condition, (iii) monitor a chronic health condition, (iv) detect trends, (v) detect outliers, or (vi) identify users that closely resemble one another in terms of health, behavior, or activity. ### Claim 4 (depends on 1) The method of claim 1, wherein generating the set of masked data records comprises determining a level of similarity between a data record of the set of data records and a data record of the subset of the set of data records. ### Claim 5 (depends on 1) The method of claim 1, wherein generating the set of masked data records comprises dividing the subset of the set of data records into a plurality of groups using one or more segmentation or clustering techniques, wherein missingness of each data record of the subset of the set of data records is used to mask another data record of the subset of the set of data records that is within a common group of the plurality of groups when generating a training dataset. ### Claim 6 (depends on 1) The method of claim 1, wherein the physical statistics comprise physiological data, wherein the physiological data comprise one or more of: resting heart rate, current heart rate, heart rate variability, respiration rate, galvanic skin response, skin temperature, or blood oxygen level. ### Claim 7 (depends on 1) The method of claim 1, wherein the physical statistics comprise behavioral data, and wherein the behavioral data comprise one or more of: daily number of steps, distance walked, time active, exercise amount, exercise type, time slept, number of times sleep was interrupted, sleep start times, sleep end times, napping, or resting. ### Claim 8 (depends on 1) The method of claim 1, wherein the set of data records comprise time series data. ### Claim 9 (depends on 1) The method of claim 1, wherein the wearable device associated with each user of the plurality of users of the population comprises a personal health sensor device. ### Claim 10 A system comprising a computing device comprising at least one processor and instructions executable by the at least one processor to cause the at least one processor to perform operations comprising: (a) identifying, based at least in part on one or more demographics of a target user, a population from a larger group of users, wherein the population comprises one or more digital twins of the target user, and wherein the population is characterized by having a common demographic; (b) accessing, by a machine learning system, a set of data records for a plurality of users of the population, the set of data records representative of physical statistics measured for each user of the plurality of users of the population over a time period, wherein the physical statistics for each user of the plurality of users of the population are measured using a wearable device associated with each user of the plurality of users of the population; (c) for each data record of a subset of the set of data records: (i) identifying, by the machine learning system, a portion of the time period associated with a pattern of missing data from the set of data records, wherein the pattern of missing data corresponds to periods of disuse or deactivation of the wearable device associated with each user of the plurality of users of the population, and (ii) generating, by the machine learning system, a masked data record by masking a portion of an additional data record of the set of data records corresponding to the identified portion of the time period, wherein the masking of the additional data record of the subset of the set of data records causes the additional data record to resemble the pattern of missing data; (d) generating, by the machine learning system, a training dataset comprising both of at least the portion of the additional data record and the corresponding generated masked data record for each data record of the subset of the set of data records; (e) training, by the machine learning system, a machine learning model using the generated training dataset, the machine learning model configured to predict, for a received data record comprising masked data, imputed data corresponding to data of the received data record obscured based on the masked data; (e) generating, by the machine learning model, a plurality of learned representations, wherein the plurality of learned representations are associated with the prediction of the imputed data; and (g) fine-tuning, by the machine learning system, a learned representation of the plurality of learned representations to a downstream machine learning task, wherein the downstream machine learning task comprises processing a set of data records from a wearable device associated with the target user, thereby generating complete data for the target user. ### Claim 11 (depends on 10) The system of claim 10, wherein the additional data record comprises missing data different from the identified portion of the time period of the corresponding data record. ### Claim 12 (depends on 10) The system of claim 10, wherein the instructions are executable by the at least one processor to cause the at least one processor to perform operations further comprising: determining a level of similarity between a current data record and the additional data record. ### Claim 13 (depends on 10) The system of claim 10, wherein the instructions are executable by the at least one processor to cause the at least one processor to perform operations further comprising: dividing the subset of the set of data records into a plurality of groups using one or more segmentation or clustering techniques, wherein the pattern of missing data from the set of data records is used to mask other data records within a common group of the plurality of groups when generating the training dataset. ### Claim 14 (depends on 10) The system of claim 10, wherein the missing data is a result of missingness arising from user behavioral patterns. ### Claim 15 (depends on 10) The system of claim 10, wherein the physical statistics comprise physiological data, wherein the physiological data comprise one or more of: resting heart rate, current heart rate, heart rate variability, respiration rate, galvanic skin response, skin temperature, or blood oxygen level. ### Claim 16 (depends on 10) The system of claim 10, wherein the physical statistics comprise behavioral data, wherein the behavioral data comprise one or more of daily number of steps, distance walked, time active, exercise amount, exercise type, time slept, number of times sleep was interrupted, sleep start times, sleep end times, napping, or resting. ### Claim 17 (depends on 10) The system of claim 10, wherein the set of data records comprise time series data. ### Claim 18 (depends on 10) The system of claim 10, wherein the set of data records are generated by personal health sensor devices. ### Claim 19 (depends on 18) The system of claim 18, wherein the wearable device associated with each user of the plurality of users of the population comprises a personal health sensor device. ### Claim 20 (depends on 10) The system of claim 10, wherein the instructions are executable by the at least one processor to cause the at least one processor to perform operations further comprising: generating multiple training datasets over a plurality of iterations. ### Claim 21 A non-transitory computer-readable storage media encoded with instructions executable by one or more processors to cause the at least one processor to perform operations comprising: (a) identifying, based at least in part on one or more demographics of a target user, a population from a larger group of users, wherein the population comprises one or more digital twins of the target user, and wherein the population is characterized by having a common demographic; (b) accessing, by a machine learning system, a set of data records for a plurality of users of the population, the data records representative of physical statistics measured for each user of the plurality of users of the population over a time period, wherein the physical statistics for each user of the plurality of users of the population are measured using a wearable device associated with each user of the plurality of users of the population; (c) for each data record of a subset of data records: (i) identifying, by the machine learning system, a portion of the time period corresponding to missing data, and (ii) generating, by the machine learning system, a masked data record by masking a portion of an additional data record of the set of data records corresponding to the identified portion of the time period, to resemble a pattern of missing data from the set of data records, wherein the pattern of missing data corresponds to periods of disuse or deactivation of the wearable device associated with each user of the plurality of users of the population; (d) generating, by the machine learning system, a training dataset comprising both of at least the portion of the additional data record and the corresponding generated masked data record for each data record of the subset of the set of data records; (e) training, by the machine learning system, a machine learning model using the generated training dataset, the machine learning model configured to predict, for a received data record comprising masked data, imputed data corresponding to data of the received data record obscured based on the masked data; (f) generating, by the machine learning system, a plurality of learned representations, wherein the plurality of learned representations are associated with the prediction of the imputed data as a result of the imputation of the masked data in the pattern of missing data; and (g) fine-tuning, by the machine learning machine learning system, a learned representation of the plurality of learned representations to a downstream machine learning task, wherein the downstream machine learning task comprises processing a set of data records from a wearable device associated with the target user, thereby generating complete data for the target user. ### Claim 22 A computer-implemented method of training a machine learning model to generate inferences from wearable sensor data, comprising: (a) identifying, based at least in part on one or more demographics of a target user, a population from a larger group of users, wherein the population comprises one or more digital twins of the target user, and wherein the population is characterized by having a common demographic; (b) retrieving a first set of wearable sensor data for each subject of a plurality of subjects of the population, wherein the first set of wearable sensor data for each subject of the plurality of subjects of the population is measured using a wearable device associated with each subject of the plurality of subjects of the population; (c) selectively masking portions of at least a subset of the first set of wearable sensor data, wherein the masked portions of at least the subset of the first set of wearable sensor data are associated with periods of missing data and wherein the masking of the portions of at least the subset of the first set of the wearable sensor data causes the portions of at least the subset of the first set of the wearable sensor data to resemble a pattern of missing data from the set of data records, wherein the pattern of missing data corresponds to periods of disuse or deactivation of the wearable device associated with each user of the plurality of users of the population; (d) creating a training set comprising at least the subset of the first set of wearable sensor data; (e) training the machine learning model to impute data to the masked portions of at least the subset of the first set of wearable sensor data, thereby producing at least one learned representation from the training; and (f) fine-tuning the at least one learned representation by using the machine learning model to a downstream machine learning task, wherein the downstream machine learning task comprises processing a second set of wearable sensor data from a wearable device associated with the target user, thereby generating complete data for the target user. ### Claim 23 (depends on 22) The method of claim 22, wherein the downstream machine learning task is imputation, regression, segmentation, or classification. ### Claim 24 (depends on 22) The method of claim 22, wherein the machine learning model comprises an attention mechanism. ### Claim 25 (depends on 24) The method of claim 24, wherein the attention mechanism is a multi-head attention mechanism. ### Claim 26 (depends on 22) The method of claim 22, wherein at least a portion of the first wearable sensor data is synthetically generated, wherein synthetically generating the at least the portion of the first wearable sensor data comprises: (i) providing a set of time series wearable sensor data; (ii) generating a plurality of embeddings from the time series wearable data, wherein an embedding of the plurality of embeddings comprises a sequence of values, wherein a value of the sequence of values is associated with a position of a set of positions; and (iii) predicting a value for a position of the set of positions not associated with a value of the sequence of values by processing the plurality of embeddings with a second machine learning model, wherein the second machine learning model comprises an attention mechanism, wherein at least a portion of an attention weight matrix generated from processing the plurality of embeddings is masked. ### Claim 27 (depends on 26) The method of claim 26, wherein the set of positions comprises a set of positions in time, and wherein a position in time of the set of positions in time corresponds to a future position in time. ### Claim 28 (depends on 26) The method of claim 26, wherein the set of positions comprises a set of positions in time, and wherein a position in time of the set of positions in time corresponds to a masked position in time. ### Claim 29 (depends on 1) The method of claim 1, wherein the common demographic is a common health condition. ### Claim 30 (depends on 1) The method of claim 1, wherein the downstream machine learning task further comprises: predicting onset of a health condition for the target user based at least in part on the complete data for the target user.