View playback to enhance collaboration and comments

A device that records and plays back collaborator content selection and manipulation actions and comments during interaction with a document to enhance asynchronous collaboration, comprising:

• an instruction memory that stores instructions;
• a memory that stores a representation of a document as a string of primitive data structures and command operations expressing changes to the primitive data structures;
• and
• one or more processing devices that execute the instructions to perform operations including:
• implementing a document editor that records collaborator content selection and manipulation actions and comments during interaction with the document in a first collaboration viewport at a first time and inserts a recording of collaborator content selection and manipulation actions and comments during interaction with the document by at least one collaborator in the first collaboration viewport at the first time into the document as an interactable element, the interactable element being activatable by a collaborator to playback at least one audio or video recording and a text selection, mouse or pen movements, or ink annotations during interaction with the document by the at least one collaborator in the first collaboration viewport at the first time, and the collaborator content selection and manipulation actions and comments including at least one of the audio or video recording and text selection, mouse or pen movements, or ink annotations formatted as strings of primitive data structures and command operations expressing changes to the primitive data structures;
• and
• implementing a viewer that, upon activation of the interactable element by the collaborator, retrieves selected strings of the primitive data structures and command operations expressing changes to the primitive data structures for the document from the memory for playback of the at least one audio or video recording while the command operations are executed for the text selection, mouse or pen movements, or ink annotations during interaction with the document by the at least one collaborator to illustrate changes made to the document in a second collaboration viewport at a second time.

A system for recording and playing back collaborator content selection and manipulation actions and comments during interaction with a document to enhance asynchronous collaboration, comprising:

• a memory that stores a representation of a document as a string of primitive data structures and command operations expressing changes to the primitive data structures;
• and
• a collaboration server that manages collaboration on the document by at least two collaborators, the collaboration server receiving recorded collaborator content selection and manipulation actions and comments during interaction with the document by at least one collaborator in a first collaboration viewport at a first time and inserting a recording of collaborator content selection and manipulation actions and comments during interaction with the document by the at least one collaborator in the first collaboration viewport at the first time into the document as an interactable element, the interactable element being activatable by a collaborator to playback at least one audio or video recording and a text selection, mouse or pen movements, or ink annotations during interaction with the document by the at least one collaborator in the first collaboration viewport at the first time, and, the collaborator content selection and manipulation actions and comments including at least one of the audio or video recording and text selection, mouse or pen movements, or ink annotations formatted as strings of primitive data structures and command operations expressing changes to the primitive data structures, the collaboration server storing the strings of primitive data structures and command operations expressing changes to the primitive data structures in the memory and, upon activation of the interactable element by the collaborator, retrieving selected strings of the primitive data structures and command operations expressing changes to the primitive data structures for presentation to a viewer that plays back the at least one audio or video recording while the command operations are executed for the text selection, mouse or pen movements, or ink annotations during interaction with the document by the at least one collaborator to illustrate changes made to the document in a second collaboration viewport at a second time.