Spatially-aware voice note system for augmented reality

A mobile device to create an enhanced voice note for transmission to an augmented reality (AR) device, the mobile device comprising:

• a display;
• at least one camera;
• at least one microphone;
• at least one processor;
• and
• at least one memory storage device storing instructions thereon, which, when executed by the at least one processor, cause the mobile device to perform operations comprising:
• receiving a user input to initiate recording of a voice note;
• in response to receiving the user input:
• activating a front-facing camera and at least one microphone to capture video of the face of a user and corresponding audio;
• processing the video by applying a computer vision segmentation process to isolate a region around the mouth of the user;
• generating a visual representation of the voice note by extracting the isolated mouth region from the video;
• receiving a user selection of a spatial destination for playing back the enhanced voice note via the AR device of a message recipient;
• applying one or more enhancements to the recorded audio, wherein at least one of the one or more enhancements comprises processing the audio to reflect the selection of the spatial destination for playback;
• packaging the enhanced audio, the visual representation, and information reflecting the spatial destination to generate the enhanced voice note, wherein the information reflecting the spatial destination is for use in positioning the visual representation of the enhanced voice note in AR space relative to an orientation of the AR device of the message recipient;
• and
• transmitting the voice note message for communication to the AR device of the message recipient.

A system for creating an enhanced voice note for transmission to an augmented reality (AR) device, the system comprising:

• means for receiving a user input to initiate recording of a voice note;
• means for activating a front-facing camera and at least one microphone to capture video of the face of a user and corresponding audio in response to receiving the user input;
• means for processing the video by applying a computer vision segmentation process to isolate a region around the mouth of the user;
• means for generating a visual representation of the voice note by extracting the isolated mouth region from the video;
• means for receiving a user selection of a spatial destination for playing back the enhanced voice note via the AR device of a message recipient;
• means for applying one or more enhancements to the recorded audio, wherein at least one of the one or more enhancements comprises processing the audio to reflect the selection of the spatial destination for playback;
• means for packaging the enhanced audio, the visual representation, and information reflecting the spatial destination to generate the enhanced voice note, wherein the information reflecting the spatial destination is for use in positioning the visual representation of the enhanced voice note in AR space relative to an orientation of the AR device of the message recipient;
• and
• means for transmitting the voice note message for communication to the AR device of the message recipient.