Associating strokes with content

A computer-implemented method that displays content on a display surface and allows strokes to be written over the content on the display surface comprising:

• storing contexts for a plurality of strokes in a memory, wherein each context represents an association between at least one stroke and a portion of image data previously displayed, wherein each association comprises a set of one or more feature points extracted from the portion of image data previously displayed;
• receiving, by a processor, an indication of a new stroke to be associated with a first image and a first context, wherein the first context comprises a set of one or more first feature points extracted from the first image;
• determining, by the processor, if the portion of the first context is the same as, overlaps or is independent of the contexts stored in the memory based, at least in part, on the set of one or more first feature points extracted from the first image used to search against sets of feature points of the stored contexts, wherein feature points from a subset of the stored contexts from a context group associated with an application program that renders the first image are searched before feature points from other contexts of the stored contexts, and wherein information indicative of the application is used to retrieve the context group;
• updating, by the processor, the memory to include the new stroke by adding the new stroke to an existing context stored in the memory when the first context is the same as the existing context, adding the new stroke to the existing context when the first context at least partially overlaps the existing context and the stroke exists in the existing context and a non-overlapping portion of the first context, wherein feature points are added to the existing context based on an amount of feature similarity between feature points of the first image and feature points of an image associated with the existing context, and wherein at least one feature point from the first context corresponding to the non-overlapping portion of the first image that does not overlap an image associated with the existing context is added to the existing context to expand the existing context, and storing the new stroke with the first context independent of the contexts already stored in the memory when the first context is independent of the contexts stored in the memory, and wherein the new stroke and at least one other stroke that appeared together in the application are stored in the context group, the new stroke and the at least one other stroke having different feature points that are stored together with feature points of the subset of the stored contexts from the context group;
• determining, by the processor responsive to a second image, if second image data of the second image is associated with one of the contexts stored in the memory and returning information identifying one or more strokes of the context associated with the second image data and transformation information to transform the one or more strokes of the associated context;
• and
• displaying, on the display surface, transformed versions of the one or more identified strokes of the context associated with the second image data.

An article of manufacture having one or more computer readable non-transitory storage media storing instructions thereon which, when executed by a system, cause the system to perform a method comprising:

• storing contexts for a plurality of strokes in a memory, wherein each context represents an association between at least one stroke and a portion of image data previously displayed, wherein each association comprises a set of one or more feature points extracted from the portion of image data previously displayed;
• receiving an indication of a new stroke to be associated with a first image and a first context, wherein the first context comprises a set of one or more first feature points extracted from the first image;
• determining if the portion of the first context is the same as, overlaps or is independent of the contexts stored in the memory based, at least in part, on the set of one or more first feature points extracted from the first image used to search against sets of feature points of the stored contexts, wherein feature points from a subset of the stored contexts from a context group associated with an application program that renders the first image are searched before feature points from other contexts of the stored contexts, and wherein information indicative of the application is used to retrieve the context group;
• updating the memory to include the new stroke by adding the new stroke to an existing context stored in the memory when the first context is the same as the existing context, adding the new stroke to the existing context when the first context at least partially overlaps the existing context and the stroke exists in the existing context and a non-overlapping portion of the first context, wherein feature points are added to the existing context based on an amount of feature similarity between feature points of the first image and feature points of an image associated with the existing context, and wherein at least one feature point from the first context corresponding to the non-overlapping portion of the first image that does not overlap an image associated with the existing context is added to the existing context to expand the existing context, and storing the new stroke with the first context independent of the contexts already stored in the memory when the first context is independent of the contexts stored in the memory, and wherein the new stroke and at least one other stroke that appeared together in the application are stored in the context group, the new stroke and the at least one other stroke having different feature points that are stored together with feature points of the subset of the stored contexts from the context group.