Digital assistant and a corresponding method for voice-based interactive communication based on detected user gaze indicating attention

A digital assistant arranged for voice-based interactive communication, wherein the digital assistant is configured to:

• detect a user attention and as a result be set into a listening mode, wherein the digital assistant is further configured to:
• detect a gaze of the user;
• determine that the gaze is directed towards a physical or virtual object, wherein the physical or virtual object is different from the digital assistant;
• and
• in response to determining that the gaze is directed towards the physical or virtual object, activate the listening mode of the digital assistant;
• detect, during the listening mode, the user as a current speaker;
• detect and record speech uttered by said current speaker;
• detect a lip movement of the current speaker while detecting said speech;
• detect an endpoint after the speech sound detection step, in which the digital assistant detects an endpoint of said speech and as a result exits said listening mode, wherein the endpoint detection step further comprises:
• a user depiction step, in which a face of the user is depicted and stored as a digital image;
• a facial expression detection step, wherein the facial expression detection step includes detecting, based on a set of predetermined parameters, a facial expression that is indicative of the user not having finished speaking;
• and as a result of said facial expression detection, postponing the endpoint detection despite no speech currently being detected;
• parse said recorded speech and extract speech-based verbal informational content from said recorded speech;
• and
• thereafter provide feedback to the user based on said recorded speech.

A non-transitory computer program product configured to, when executed on one or more processors, cause a digital assistant to perform operations comprising:

• an attention detection step, in which the digital assistant detects attention of a user by:
• detecting a gaze of a user;
• determining that the gaze is directed towards a physical or virtual object, wherein the physical or virtual object is different from the digital assistant;
• and
• in response to determining that the gaze is directed towards the physical or virtual object, activating a listening mode of the digital assistant;
• a speaker detection step, in which the digital assistant is caused to detect the user as a current speaker;
• a speech sound detection step, in which the digital assistant is caused to detect and to record speech uttered by said current speaker, which speech sound detection step comprises a lip movement detection step, in which the digital assistant is caused to detect a lip movement of the current speaker;
• an endpoint detection step after the speech sound detection step, in which the digital assistant detects an endpoint of said speech and as a result exits said listening mode, wherein the endpoint detection step further comprises:
• a user depiction step, in which a face of the user is depicted and stored as a digital image;
• a facial expression detection step, wherein the facial expression detection step includes detecting, based on a set of predetermined parameters, a facial expression that is indicative of the user not having finished speaking;
• and
• as a result of said facial expression detection, postponing the endpoint detection despite no speech currently being detected;
• a speech analysis step, in which the digital assistant is caused to parse said recorded speech and to extract speech-based verbal informational content from said recorded speech;
• and
• a subsequent response step, in which the digital assistant is caused to provide feedback to the user based on said recorded speech.