# US 2017/0289080 A1 · System for annotation of electronic messages with contextual information > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2017/0289080 A1 - Document ID: US20170289080A1 - Status: Published application - Kind: A1 - Application number: 15/489,198 - Filed: 2017-04-17 - Published: 2017-10-05 - Earliest claimed: 2015-07-29 - Assignees: Mimecast North America, Inc. - Applicants: Mimecast North America, Inc. - Inventors: Nathaniel Borenstein; Marc Amphlett; Clive Jordan; Max Linscott; Niall O'Malley; Jacqueline Osborne; Luke Pentreath; Oliver Scott; Rahul Sharma - Web page: https://patentgrep.com/patent/US20170289080A1 ## CPC - H04L 51/08: Annexed information, e.g. attachments - H04L 51/216: Handling conversation history, e.g. grouping of messages in sessions or threads - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract A system that transforms electronic messages into annotated messages that include contextual information to aid a recipient in utilizing the electronic message, understanding its meaning, and responding to the message. Annotations are additions or modifications to the original message with contextual information that is related to the features and contents of the original message. Message features are extracted and used to search one or more sources of contextual information. Relevant items are retrieved and added to the message, for example as attachments, hyperlinks, or inline notes. Machine learning techniques may be used to generate or refine modules for feature extraction and information selection. Feedback components may be used to track the usage and value of annotations, in order to iteratively improve the annotation system. ## Claims ### Claim 1 A system for annotation of electronic messages with contextual information comprising: a computer system comprising at least one processor, at least one tangible memory device, and at least one network interface; an interface to one or more contextual information sources that communicates over one or more of said at least one network interface; a message input module configured to receive an electronic message on one or more of said at least one network interface, said electronic message comprising one or more message artifacts, said message artifacts comprising: one or more senders, one or more sender addresses, one or more sender organizations, one or more recipients, one or more recipient addresses, one or more recipient organizations, a subject, a contents, one or more message body parts, a message thread, an event, a timestamp, a location, one or more links, an importance indicator, one or more media types, a set of message metadata, and one or more attachments; a feature extraction module coupled to said message input module and configured to execute on said at least one processor, wherein said feature extraction module analyzes said message artifacts of said electronic message, and generates a set of features associated with said message artifacts; an information selection module coupled to said feature extraction module and to said interface to said one or more contextual information sources, and configured to execute on said at least one processor, wherein said information selection module selects one or more contextual information items from said one or more contextual information sources based on said set of features; a message annotation module coupled to said information selection module and to said message input module, and configured to execute on said at least one processor, wherein said message annotation module adds said one or more contextual information items to said electronic message to create an annotated electronic message; and, a message output module coupled to said message annotation module, wherein said message output module transmits said annotated electronic message on one or more of said at least one network interface to at least said one or more recipients. ### Claim 2 (depends on 1) The system for annotation of electronic messages with contextual information of claim 1, wherein said message annotation module attaches said one or more contextual information items to said electronic message as attachments containing said one or more contextual information items or links or references to said one or more contextual information items. ### Claim 3 (depends on 1) The system for annotation of electronic messages with contextual information of claim 1, wherein said message annotation module inserts said one or more contextual information items or links or references to said one or more contextual information items into said contents of said electronic message. ### Claim 4 (depends on 1) The system for annotation of electronic messages with contextual information of claim 1, wherein said set of features comprises a frequency distribution of n-grams appearing in said message artifacts, wherein the length of each of said n-grams is between a minimum n-gram length and a maximum n-gram length selected by said feature extraction module. ### Claim 5 (depends on 1) The system for annotation of electronic messages with contextual information of claim 1, wherein said information selection module is configured to: calculate a relevance score for each of a set of available context information items based on said one or more features associated with said message artifacts; rank said set of available context information items based on said relevance score; and, select a top-ranked subset of said set of available context information items. ### Claim 6 (depends on 5) The system for annotation of electronic messages with contextual information of claim 5, wherein said relevance score is a similarity metric or a distance metric between said set of features associated with said message artifacts and a corresponding set of features associated with or calculated from each of said available context information items. ### Claim 7 (depends on 1) The system for annotation of electronic messages with contextual information of claim 1, wherein said information selection module is configured to: perform one or more queries against one or more external search engines based on said one or more features; retrieve a set of top-ranked results from said one or more queries; and, wherein said message annotation module annotates said electronic message with said set of top-ranked results. ### Claim 8 (depends on 1) The system for annotation of electronic messages with contextual information of claim 1, wherein said set of features comprises said one or more recipients, and other features extracted from said electronic message; said one or more contextual information sources comprise access rules that indicate an accessible subset of said contextual information items that each of said one or more recipients is permitted to access; said information selection module selects a set of contextual information items for each of said one or more recipients from the accessible subset of said contextual information items for that recipient, based on said other features extracted from said electronic message. ### Claim 9 (depends on 1) The system for annotation of electronic messages with contextual information of claim 1, wherein said feature extraction module categorizes said electronic message or an artifact of said electronic message into one or more classes using a classifier. ### Claim 10 (depends on 9) The system for annotation of electronic messages with contextual information of claim 9, wherein said classifier is a probabilistic classifier, and said message annotation module annotates said electronic message only if the probability of a correct classification from said probabilistic classifier exceeds a threshold probability. ### Claim 11 (depends on 1) The system for annotation of electronic messages with contextual information of claim 1, wherein said feature extraction module generates one or more topics associated with said electronic message using a probabilistic topic model. ### Claim 12 (depends on 1) The system for annotation of electronic messages with contextual information of claim 1, further comprising a machine learning module that uses a training set to develop or revise methods for feature extraction and information selection. ### Claim 13 (depends on 1) The system for annotation of electronic messages with contextual information of claim 1, further comprising a feedback module that tracks usage of said one or more contextual information items by said one or more recipients. ### Claim 14 (depends on 13) The system for annotation of electronic messages with contextual information of claim 13, wherein said feedback module accepts input from said one or more recipients that indicates the relevance or usefulness of said one or more contextual information items. ### Claim 15 (depends on 12) The system for annotation of electronic messages with contextual information of claim 12, further comprising a feedback module that tracks usage of said one or more contextual information items by said one or more recipients, wherein data received by said feedback module is used by said machine learning module to revise said methods for feature extraction and information selection. ### Claim 16 (depends on 12) The system for annotation of electronic messages with contextual information of claim 12, further comprising a search monitoring module that monitors the information searches of said one or more recipients after they receive said annotated electronic message; and provides data on said information searches to said machine learning module to revise said methods for feature extraction and information selection. ### Claim 17 (depends on 1) The system for annotation of electronic messages with contextual information of claim 1, wherein said electronic message contains one or more media items, each comprising one or more images, videos, audio clips, or any combinations thereof; said feature extraction module analyzes said one or more media items to extract one or more segments of said one or more media items that represent one or more objects contained in said one or more media items; and said information selection module compares said one or more segments of said one or more media items to said contextual information sources to identify said one or more objects and to select contextual information about said one or more objects. ### Claim 18 (depends on 1) The system for annotation of electronic messages with contextual information of claim 1, wherein said electronic message contains medical information related to a medical or legal case; and, said information selection module selects information on other cases similar to said medical or legal case. ### Claim 19 (depends on 1) The system for annotation of electronic messages with contextual information of claim 1, wherein said feature extraction module extracts or derives one or more locations from said one or more message artifacts; said one or more contextual information sources comprise location-tagged contextual information items; and, said information selection module selects a set of said location-tagged contextual information items with location-tags that are equal to or in proximity to said one or more locations. ### Claim 20 (depends on 19) The system for annotation of electronic messages with contextual information of claim 19, wherein said derives one or more locations from said one or more message artifacts comprises obtain the location of one or more of said one or more senders or of said one or more receivers from one or more of location configuration information describing usual or standard locations for one or more of said one or more senders or of said one or more receivers; current sender location information describing the location of said one or more senders at the time said electronic message is composed or sent; current receiver location information describing the location of said one or more receivers at the time said annotated electronic message is received or read.