# US 8,751,921 B2 · Presenting annotations in hierarchical manner > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,751,921 B2 - Document ID: US8751921B2 - Status: Granted - Kind: B2 - Application number: 12/179,574 - Filed: 2008-07-24 - Published: 2010-01-28 - Granted: 2014-06-10 - Earliest claimed: 2008-07-24 - Assignees: Microsoft Corporation - Applicants: Christian Schormann - Inventors: Christian Schormann - Web page: https://patentgrep.com/patent/US8751921B2 ## CPC - G06Q 10/10: Office automation; Time management - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract Ambient displays and electronic whiteboards are commonly used to present information during meeting or other appropriate settings. Many times graphical objects are displayed on ambient displays where users of the ambient display will create annotations to the graphical object to provide further comments, meaning, and/or context. Annotations on the ambient display may be recorded for later review. An effective way to present recorded annotations is through a hierarchical system that provides context to the annotations. Annotations may be defined as sets of annotations based upon a logical grouping (e.g. spatial density, temporal density, etc.). These sets can be displayed to users as annotation branches. One method for displaying annotations branches is with a timeline. Depending on where along the timeline a user is, different annotations and/or annotation branches are displayed in relation to one another (e.g. annotation branches that derive from another annotation branch are displayed in a hierarchical manner). ## Claims ### Claim 1 A method for presenting a hierarchical representation of a plurality of recorded annotations, comprising: logically organizing a first annotation and a second annotation into a first annotation branch based upon determining that the first annotation and the second annotation were created within a threshold physical distance of one another, the first annotation positioned relative to the second annotation within the first annotation branch based upon a spatial proximity between the first annotation and the second annotation on a display within which the first annotation and the second annotation were created; logically organizing a third annotation and a fourth annotation into a second annotation branch; and presenting, on an ambient display, the first annotation branch and the second annotation branch within a hierarchical representation comprising a hierarchical timeline such that the second annotation branch branches off of the first annotation branch based upon a determination that at least one of: the third annotation comments textually on at least one of the first annotation or the second annotation; or the fourth annotation comments textually on at least one of the first annotation or the second annotation, at least one of the first annotation branch or the second annotation branch presented at a particular location within the hierarchical representation based upon predetermined criteria comprising at least one of a spatial density of annotations or a zoom state, where responsive to placement of a timeline marker on a first location on the ambient display associated with a first time, at least one of one or more annotations or one or more objects corresponding to the first time are displayed, at least one of the first annotation, the second annotation, the third annotation or the fourth annotation comprising at least one of electronic ink, audio, text, or video. ### Claim 2 (depends on 1) The method of claim 1, comprising highlighting a portion of the first annotation branch based upon the portion comprising an annotation density above a threshold, the annotation density comprising a temporal density. ### Claim 3 (depends on 1) The method of claim 1, comprising highlighting a portion of the first annotation branch based upon the portion comprising an annotation density above a threshold, the annotation density comprising a spatial density. ### Claim 4 (depends on 1) The method of claim 1, comprising highlighting a portion of the second annotation branch based upon the portion comprising an annotation density above a threshold, the annotation density comprising a temporal density. ### Claim 5 (depends on 1) The method of claim 1, comprising highlighting a portion of the second annotation branch based upon the portion comprising an annotation density above a threshold, the annotation density comprising a spatial density. ### Claim 6 (depends on 1) The method of claim 1, at least one of the first annotation or the second annotation comprising at least one of a deleted annotation or an overlapping annotation. ### Claim 7 (depends on 1) The method of claim 1, the hierarchical representation displayed horizontally. ### Claim 8 (depends on 1) The method of claim 1, the logically organizing a first annotation and a second annotation into a first annotation branch comprising: logically organizing the first annotation and the second annotation into the first annotation branch based upon determining that the first annotation and the second annotation were created within a threshold time span of one another. ### Claim 9 (depends on 1) The method of claim 1, comprising: presenting a thumbnail comprising a snapshot image representing the first annotation within the first annotation branch. ### Claim 10 (depends on 1) The method of claim 1, comprising highlighting a portion of the first annotation branch based upon the portion being designated as being highly relevant by one or more users. ### Claim 11 (depends on 1) The method of claim 1, comprising highlighting a portion of the second annotation branch based upon the portion being designated as being highly relevant by one or more users. ### Claim 12 (depends on 1) The method of claim 1, comprising: merging at least one of the first annotation branch or the second annotation branch with another annotation branch within the hierarchical representation based upon user input. ### Claim 13 (depends on 1) The method of claim 1, comprising: creating a new annotation branch within the hierarchical representation based upon user input associated with the hierarchical representation. ### Claim 14 A system, implemented at least in part via a processing unit, for presenting a hierarchical representation of a plurality of recorded annotations, comprising: a branch component configured to: logically organize a first annotation and a second annotation into a first annotation branch based upon determining that the first annotation and the second annotation were created within a threshold physical distance of one another, the first annotation positioned relative to the second annotation within the first annotation branch based upon a spatial proximity between the first annotation and the second annotation on a display within which the first annotation and the second annotation were created; and logically organize a third annotation and a fourth annotation into a second annotation branch; and a presentation component configured to: present the first annotation branch and the second annotation branch within a hierarchical representation such that the second annotation branch branches off of the first annotation branch based upon a determination that at least one of: the third annotation comments textually on at least one of the first annotation or the second annotation; or the fourth annotation comments textually on at least one of the first annotation or the second annotation, at least one of the first annotation branch or the second annotation branch presented at a particular location within the hierarchical representation based upon predetermined criteria comprising at least one of a spatial density of annotations or a zoom state, where responsive to placement of a timeline marker on a first location associated with a first time, at least one of one or more annotations or one or more objects corresponding to the first time are displayed, at least one of the first annotation, the second annotation, the third annotation or the fourth annotation comprising at least one of electronic ink, audio, text, or video. ### Claim 15 (depends on 14) The system of claim 14, at least one of the first annotation or the second annotation comprising a graphic object, at least one of the third annotation or the fourth annotation comprising a textual comment. ### Claim 16 (depends on 14) The system of claim 14, at least one of the first annotation or the second annotation comprising data recorded from physical markings on the display. ### Claim 17 (depends on 14) The system of claim 14, the branch component configured to: logically organize the first annotation and the second annotation into the first annotation branch based upon determining that the first annotation and the second annotation were created within a threshold time span of one another. ### Claim 18 A computer readable medium, excluding signals, comprising instructions that when executed perform a method for presenting a hierarchical representation of a plurality of recorded annotations from an ambient display, the method comprising: logically organizing one or more annotations into a first annotation branch, the first annotation branch comprising a first annotation positioned relative to a second annotation within the first annotation branch based upon a spatial proximity between the first annotation and the second annotation on an ambient display upon which the first annotation and the second annotation were created; logically organizing a third annotation and a fourth annotation into a second annotation branch; and presenting the first annotation branch and the second annotation branch within a hierarchical representation comprising a hierarchical timeline such that the second annotation branch branches off of the first annotation branch based upon a determination that at least one of: the third annotation comments textually on at least one of the first annotation or the second annotation; or the fourth annotation comments textually on at least one of the first annotation or the second annotation, at least one of the first annotation branch or the second annotation branch presented at a particular location within the hierarchical representation based upon predetermined criteria comprising at least one of a spatial density of annotations or a zoom state, where responsive to placement of a timeline marker on a first location associated with a first time, at least one of the one or more annotations or one or more objects corresponding to the first time are displayed, at least one of the first annotation, the second annotation, the third annotation or the fourth annotation comprising at least one of electronic ink, audio, text, or video. ### Claim 19 (depends on 18) The computer readable medium of claim 18, at least one of the first annotation or the second annotation comprising at least one of a deleted annotation or an overlapping annotation. ### Claim 20 (depends on 18) The computer readable medium of claim 18, the predetermined criteria comprising the zoom state.