Information density of documents

A system comprising:

• one or more processors;
• and
• a memory coupled to the one or more processors, the memory including instructions that, when executed by the one or more processors, cause the one or more processors to:
• receive a document, the document comprising multiple data units arranged in a sequence;
• separate a fragment from the sequence by identifying a token delimiter, wherein the token delimiter comprises one of a start portion or an end portion of the fragment;
• separate a first data unit from the fragment by identifying a second token delimiter;
• determine a fragment rank based on a frequency score for the first data unit within the fragment;
• separate a first data unit from the fragment by identifying a second token delimiter;
• place the fragment in a sorted list based on the fragment rank, the sorted list comprising multiple fragments of the document;
• form a modified document including at least a top fragment from the sorted list, the top fragment having a top fragment rank greater than a user selected rank;
• and
• provide the modified document to the user.

A non-transitory, computer readable medium comprising instructions which, when executed by a processor in a computer, cause the computer to execute a method, the method comprising:

• receiving a document, the document comprising multiple data units arranged in a sequence;
• separating a fragment from the sequence by identifying a token delimiter, wherein the token delimiter comprises one of a start portion or an end portion of the fragment;
• separating a first data unit from the fragment by identifying a second token delimiter;
• determining a frequency score of the first data unit based on a frequency of the first data unit and a frequency of a second data unit within the fragment
• determining a fragment rank based on the frequency score of the first data unit within the fragment;
• placing the fragment in a sorted list based on the fragment rank, the sorted list comprising multiple fragments of the document;
• forming a modified document including at least a top fragment from the sorted list, the top fragment having a top fragment rank greater than a user selected rank;
• and
• providing the modified document to the user.