Personalized and curated transcription of auditory experiences to improve user engagement

One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform steps of:

• computing a first metric for a word, the metric indicating an attentional state of a user perceiving the word during an auditory experience, wherein the metric is computed based on biomarkers associated with the user that are computed using signals from one or more sensors on a near-eye display system operating in an artificial reality environment;
• computing a second metric corresponding to an intelligibility of the word by estimating an audibility of the word using a signal-to-noise (SNR) ratio associated with the word during the auditory experience;
• computing a third metric corresponding to an importance of the word by using a machine learning model to determine the importance of the word in a context of a conversation during the auditory experience;
• and
• based on a weighting assigned to each of the first metric, the second metric and the third metric, computing a fourth metric;
• and
• determining whether to transcribe the word for display based on the fourth metric.

A wearable device, comprising:

• one or more memories that store instructions, and
• one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to:
• compute a first metric for a word, the metric indicating an attentional state of a user perceiving the word during an auditory experience, wherein the metric is computed based on biomarkers associated with the user that are computed using signals from one or more sensors on a near-eye display system operating in an artificial reality environment;
• compute a second metric corresponding to an intelligibility of the word by estimating an audibility of the word using a signal-to-noise (SNR) ratio associated with the word during the auditory experience;
• compute a third metric corresponding to an importance of the word by using a machine learning model to determine the importance of the word in a context of a conversation during the auditory experience;
• and
• based on a weighting assigned to each of the first metric, the second metric and the third metric, compute a fourth metric;
• and
• determine whether to transcribe the word for display based on the fourth metric.