Method and Apparatus for Communication Between Humans and Devices

#This invention relates to attentive user interfaces for improving communication between humans and devices. More particularly, this invention relates to use of eye contact/gaze direction information by technological devices and appliances to more effectively communicate with users, in device or subject initiated communications.

#Interaction with technological devices is becoming an ever-increasing part of everyday life. However, effectiveness and efficiency of such interaction is generally lacking. In particular, when seeking user input, devices such as computers, cellular telephones and personal digital assistants (PDAs) are often disruptive, because such devices cannot assess the user's current interest or focus of attention. More efficient, user-friendly interaction is desirable in interactions with household appliances and electronic equipment, computers, and digital devices.

#One way that human-device interactions can be improved is by employing user input such as voice and/or eye contact, movement, or position to allow users to control the device. Many previous attempts relate to controlling computer functions by tracking eye gaze direction. For example, U.S. Pat. No. 6,152,563 to Hutchinson et al. and U.S. Pat. No. 6,204,828 to Amir et al. teach systems for controlling a cursor on a computer screen based on user eye gaze direction. U.S. Pat. Nos. 4,836,670 and 4,973,49 to Hutchinson, U.S. Pat. No. 4,595,990 to Garwin et al., U.S. Pat. No. 6,437,758 to Nielsen et al., and U.S. Pat. No. 6,421,064 and U.S. Patent Application No. 2002/0105482 to Lemelson et al. relate to controlling information transfer, downloading, and scrolling on a computer based on the direction of a user's eye gaze relative to portions of the computer screen. U.S. Pat. No. 6,456,262 to Bell provides an electronic device with a microdisplay in which a displayed image may be selected by gazing upon it. U.S. Patent Application No. 2002/0141614 to Lin teaches enhancing the perceived video quality of the portion of a computer display corresponding to a user's gaze.

#Embodiments of the invention are described below, by way of example, with reference to the accompanying drawings, wherein:

#FIG. 1 is a schematic diagram of an eye contact sensor;

#FIG. 2 depicts an algorithm for an eye contact sensor in accordance with an embodiment of the invention;

#The present invention is based, at least in part, on the recognition that human-device interaction can be improved by implementing in devices some of the basic social rules that govern human face-to-face conversation. Such social rules are exemplified in the following scenario: Person A is in conversation with person B (or engaged in a task), and person C wishes to gain A's attention. There are a number of ways in which C may do so without interfering with A's activities. Firstly, C may position himself such that A becomes peripherally aware of his presence. Secondly, C may use proximity, movement, gaze or touch to capture A's attention without using verbal interruption. The use of nonverbal visual cues by C allows A to finish his conversation/task before acknowledging C's request for attention, e.g., by making eye contact. If A does not provide acknowledgement, C may choose to withdraw his request by moving out of A's visual field. Indeed, Frolich (1994) found that initiators of conversations often wait for visual cues of attention, in particular, the establishment of eye contact, before launching into their conversation during unplanned face-to-face encounters. Face-to-face interaction is therefore different from the way we typically interact with most technological devices in that it provides a rich selection of both verbal and nonverbal communication channels. This richness is characterized by (i) flexibility in choosing alternate channels of communication to avoid interference or interruption, (ii) a continuous nature of the information conveyed, and (iii) a bi-directionality of communication.

#Electronic devices that require user input or attention do not follow such social rules in communicating with users. As a result they often generate intrusive and annoying interruptions. With the advent of devices such as cell phones and personal digital assistants (PDAs; e.g., Blackberry®, Palm Pilot®), users are regularly interrupted with requests for their attention. The present invention solves this problem by augmenting devices with attentive user interfaces: user interfaces that negotiate the attention they receive from or provide to users by negotiations through peripheral channels of interaction. Attentive user interfaces according to the invention follow social rules of human group communication, where, likewise, many people might simultaneously have an interest in speaking. In human group conversations, eye contact functions as a nonverbal visual signal that peripherally conveys who is attending to whom without interrupting the verbal auditory channel. With it, humans achieve a remarkably efficient process of conversational turn-taking. Without it, turn-taking breaks down. Thus, an attentive user interface according to the invention applies such social rules to device-initiated interactions or communications, by assessing a user's attentive state, and making a determination as to whether, when, and how to interrupt (e.g., notify) the user on the basis of the user's attentive state.

#To facilitate turn-taking between devices and users in a non-intrusive manner, an attentive user interface according to the invention assesses a user's attentive state by sensing one or more parameters of the user. Such parameters are indicative of the user's attentive state, and include, but are not limited to, eye contact, eye movement, eye position, eye gaze direction, voice, body presence, body orientation, head and/or face orientation, activity, and brain activity/arousal. In the case of eye contact, movement, or position, an attentive user interface senses the eyes of the user, or between the user and a subject (e.g., another human), to determine when, whether, and how to interrupt the user. For example, notification by a PDA seeking user input can be modulated on the basis of whether the user is engaged with the PDA, with another device, or a subject. The PDA then can decide whether, when, and how to notify; for example, directly, or indirectly via another device with which the user is engaged. Body presence can be sensed in various ways, such as, for example, a motion detector, a radio frequency (RF) ID tag worn by a user and sensed using, e.g., BlueTooth®, a visual tag, electro-magnetic sensors for sensing presence/location/orientation of a user within a magnetic field, and a global positioning system (GPS).