Network crawling prioritization

A system comprising:

• one or more processors;
• and
• a memory that stores instructions for causing, when executed, the one or more processors to:
• receive a set of entries, each entry of the set of entries including a time-stamp value, a path identifier to a data object having content that may be changed, and a freshness value indicating a frequency to retrieve possibly changed content from the data object, wherein a time-stamp value of an entry is updated periodically based on a freshness value of the entry, wherein a plurality of entries in the set of entries are ordered based on time-stamp values of the entries,
• select a first entry to be retrieved from the plurality of entries, in order of the plurality of entries, wherein the selected first entry includes a first time-stamp value identifying a first time being no later than other times identified by time-stamp values of other entries of the set of entries,
• retrieve first content from a first data object using a first path identifier of the selected first entry, and
• after the retrieving the first content, update a first time-stamp value of the selected first entry based on a freshness value of the selected first entry so as to re-order the plurality of entries in the set of entries,
• wherein a first entry and a second entry in the set of the entries include different freshness values, and
• wherein first content of the first entry is retrieved at a higher frequency than second content of the second entry, based on a difference of the freshness values.

A non-transitory computer readable medium comprising executable instructions, the instructions being executable by a data processing device to perform a method, the method comprising:

• receiving a set of entries, each entry of the set of entries including a time-stamp value, a path identifier to a data object having content that may be changed, and a freshness value indicating a frequency to retrieve possibly changed content from the data object, wherein a time-stamp value of an entry is updated periodically based on a freshness value of the entry, wherein a plurality of entries in the set of entries are ordered based on time-stamp values of the entries;
• selecting a first entry to be retrieved from the plurality of entries, in order of the plurality of entries, wherein the selected first entry includes a first time-stamp value identifying a first time being no later than other times identified by time-stamp values of other entries of the set of entries;
• retrieving first content from a first data object using a first path identifier of the selected first entry;
• and
• after the retrieving the first content, updating a first time-stamp value of the selected first entry based on a freshness value of the selected first entry so as to re-order the plurality of entries in the set of entries,
• wherein a first entry and a second entry in the set of the entries include different freshness values, and
• wherein first content of the first entry is retrieved at a higher frequency than second content of the second entry, based on a difference of the freshness values.