# US 11,113,672 B2 · Computer support for meetings > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,672 B2 - Document ID: US11113672B2 - Status: Granted - Kind: B2 - Application number: 15/933,249 - Filed: 2018-03-22 - Published: 2019-09-26 - Granted: 2021-09-07 - Earliest claimed: 2018-03-22 - Applicants: Microsoft Technology Licensing, LLC - Inventors: Robert Alexander Sim; Marcello Mendes Hasegawa; Ryen William White; Mudit Jain; Tomer Hermelin; Adi Gerzi Rosenthal; Sagi Hilleli - Web page: https://patentgrep.com/patent/US11113672B2 ## CPC - G06Q 10/1093: Calendar-based scheduling for persons or groups - G06F 40/10: Text processing (natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40) - G06N 20/00: Machine learning - G06Q 10/107: Computer-aided management of electronic mailing [e-mailing] - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units} - G10L 2015/088: {Word spotting} - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A system and method to provide computer support for a meeting of invitees comprises accessing one or more sensory data streams providing digitized sensory data responsive to an activity of one or more of the invitees during the meeting, the one or more sensory data streams including at least one audio stream. The method also comprises subjecting the at least one audio stream to phonetic and situational computer modeling to recognize a sequence of words in the audio stream and to assign each word to an invitee, subjecting the sequence of words to semantic computer modeling to recognize a sequence of directives in the sequence of words, and releasing one or more output data streams based on the sequence of directives, the one or more output data streams including one or more notifications. ## Claims ### Claim 1 A method to provide computer support for a meeting of invitees, the method comprising: accessing one or more sensory data streams providing digitized sensory data responsive to an activity of one or more of the invitees during the meeting, the one or more sensory data streams including at least one audio stream and a video stream; subjecting the at least one audio stream to phonetic and situational computer modeling to recognize a sequence of words in the at least one audio stream; assigning each word of the sequence of words to an invitee based at least in part on an analysis of image data from the video stream; subjecting the sequence of words to semantic computer modeling by a machine-learning model to recognize a sequence of directives in the sequence of words, each directive associated with at least one invitee or other individual; releasing one or more output data streams based on the sequence of directives, the one or more output data streams including one or more notifications comprising a list of directives recognized by the machine-learning model; receiving an edited list of directives actually discussed during the meeting; and refining the machine-learning model based on the edited list of directives. ### Claim 2 (depends on 1) The method of claim 1 wherein releasing the one or more output data streams includes submitting the sequence of directives to a user that organized the meeting for editing and subsequent distribution to one or more of the invitees. ### Claim 3 (depends on 1) The method of claim 1 wherein recognizing the sequence of directives includes extracting features from the sequence of words to recognize one or more of a directive, acceptance of a directive, and revocation of a directive. ### Claim 4 (depends on 1) The method of claim 1 wherein recognizing the sequence of directives includes classification via a trained machine-learning model. ### Claim 5 (depends on 1) The method of claim 1 wherein the sequence of directives includes a personal directive to be enacted by a first invitee. ### Claim 6 (depends on 1) The method of claim 1 wherein the one or more sensory data streams includes data revealing geophysical location of a first invitee. ### Claim 7 (depends on 1) The method of claim 1 wherein the sequence of directives includes a personal directive to a first invitee to remind the first invitee to attend the meeting. ### Claim 8 (depends on 1) The method of claim 1 wherein the one or more output data streams include an email or sms stream through which the one or more notifications are issued. ### Claim 9 An automated meeting-support system comprising: a processor; a network device operatively coupled to the processor; and computer memory operatively coupled to the processor and configured to hold program instructions executable by the processor, the program instructions comprising: instructions to train a machine-learning model on previous meeting minutes to recognize directives; instructions to access one or more sensory data streams providing digitized sensory data responsive to an activity of one or more invitees during a meeting, the one or more sensory data streams including at least one audio stream, instructions to subject the at least one audio stream to phonetic and situational computer modeling to recognize a sequence of words in the at least one audio stream and to assign each word of the sequence of words to an invitee, instructions to subject the sequence of words to semantic computer modeling by the machine-learning model to recognize a sequence of directives in the sequence of words, instructions to release, via the network device, one or more output data streams based on the sequence of directives, the one or more output data streams including one or more notifications comprising a list of directives recognized by the machine-learning model, instructions to receive an edited list of directives actually discussed during the meeting, and instructions to refine the machine-learning model based on the edited list of directives. ### Claim 10 (depends on 9) The system of claim 9 wherein the program instructions further comprise instructions to receive a plurality of agenda items and to determine, based on the sequence of words, which agenda items have been discussed at the meeting. ### Claim 11 (depends on 9) The system of claim 9 wherein the program instructions further comprise instructions to prompt the one or more invitees during the meeting for commitments. ### Claim 12 (depends on 9) The system of claim 9 wherein the program instructions further comprise instructions to track a geophysical location of a first invitee and to issue a notification via the one or more output data streams when the first invitee is late for the meeting. ### Claim 13 (depends on 12) The system of claim 12 wherein the instructions to release the one or more output data streams specify that the notification is directed to the first invitee. ### Claim 14 (depends on 12) The system of claim 12 wherein the instructions to release the one or more output data streams specify that the notification is directed to a user that is an organizer of the meeting. ### Claim 15 (depends on 9) The system of claim 9 wherein the program instructions further comprise instructions to receive a plurality of agenda items and to determine, based on the sequence of words, which agenda items have been discussed at the meeting. ### Claim 16 One or more memory devices having embodied thereon computer-executable instructions which, when executed by one or more processors, performs a method of providing support for a meeting of invitees, the method comprising: accessing one or more sensory data streams providing digitized sensory data responsive to an activity of one or more invitees during the meeting, the one or more sensory data streams including at least one audio stream; subjecting the at least one audio stream to phonetic and situational computer modeling to recognize a sequence of words in the at least one audio stream; accessing depth images obtained by a depth-imaging camera; analyzing the depth images to determine relative postures of the invitees; assigning each word of the sequence of words to an invitee based at least in part on the relative postures of the invitees to associate audio with individual invitees; subjecting the sequence of words to semantic computer modeling by a machine-learning model to recognize a sequence of directives in the sequence of words; releasing, via a network device, one or more output data streams based on the sequence of directives, the one or more output data streams including one or more notifications comprising a list of directives recognized by the machine-learning model; receiving an edited list of directives actually discussed during the meeting; and refining the machine-learning model based on the edited list of directives. ### Claim 17 (depends on 16) The one or more memory devices of claim 16, the method further comprising prompting the one or more invitees during the meeting for commitments. ### Claim 18 (depends on 16) The one or more memory devices of claim 16, the method further comprising tracking a geophysical location of a first invitee and to issue a notification via the one or more output data streams when the first invitee is late for the meeting. ### Claim 19 (depends on 18) The one or more memory devices of claim 18, wherein the notification is directed to the first invitee. ### Claim 20 (depends on 18) The one or more memory devices of claim 18, wherein the notification is directed to a user that is an organizer of the meeting.