System and method for dynamically limiting robot access to server data

In a server computer in a distributed computer network, a method for protecting data objects accessible on said server computer from excessive access by an abusive client computer, said method comprising the steps of:

• a) receiving at said server computer a request from said client computer for one of said data objects accessible on said server computer;
• b) recording a log entry associated with said request in a log file, wherein said log entry comprises a client identifier, at least one data object identifier, and a timestamp for said request, wherein said client identifier is an IP address;
• c) calculating client request values associated with said client identifier from said log entry and from previous log entries associated with said client identifier, wherein said client request values comprise a request frequency and a cumulative data request value;
• d) refusing to send from the server computer said requested data object to said client computer if at least one of said client request values exceeds one of a set of corresponding predefined maximum request values, wherein said predefined maximum request values comprise a maximum request frequency and a data access threshold;
• and
• e) calculating a second cumulative data request value from said log entry and from previous log entries associated with all client identifiers, and refusing to send from the server computer said requested data object to said client computer if the second cumulative data request value exceeds a second data access threshold,
• wherein said distributed computer network is the Internet and said client computer is a webcrawler.

In a server computer in a distributed computer network, a method for protecting data objects accessible on said server computer from excessive access by an abusive client computer, said method comprising the steps of:

• a) receiving at the server computer a request for one of said data objects from said client computer;
• b) recording a log entry associated with said request in a log file, wherein said log entry comprises a client identifier and a timestamp for said request;
• c) comparing said client identifier with a deny list comprising denied client identifiers;
• d) refusing to send from said server computer said requested data object to said client computer if said client identifier is on said deny list;
• e) if said client identifier is not on said deny list, calculating client request values associated with said client identifier from said log entry and from previous log entries associated with said client identifier, wherein said client request values comprise a request frequency and a cumulative data request value;
• and
• f) adding said client identifier to said deny list if at least one of said client request values exceeds one of a set of corresponding predefined maximum request values, wherein said predefined maximum request values comprise a maximum request frequency and a data access threshold,
• wherein said distributed computer network is the Internet and said client computer is a webcrawler.

In a distributed computer network comprising a server computer and client computers, wherein said client computers request data objects from said server computer, a data protection system associated with said server computer, said data protection system comprising:

• a) a log file comprising log entries, wherein each of said log entries corresponds to a client request from one of said client computers for one of said data objects from said server computer and comprises a client identifier, a data object identifier, and a timestamp;
• b) a request analyzer associated with said server computer for calculating client request values from said log entries, wherein said client request values comprise a client request frequency and a cumulative data request value, and for comparing said client request values with a set of corresponding predefined maximum request values comprising a maximum request frequency and a data access threshold, to generate failed client identifiers;
• and
• c) a dynamically-generated deny list comprising said failed client identifiers, wherein said request analyzer compares a new request from a known client computer with said deny list and refuses said new request if said known client computer has a client identifier matching one of said failed client identifiers,
• wherein said computer network is the Internet and said client computer is a webcrawler.