US 8,789,177 B1Granted

Method and system for automatically obtaining web page content in the presence of redirects

AssigneeSymantec Corporation

InventorsNicholas Johnston, Graham Coomer

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method and apparatus for automatically obtaining web page content in the presence of redirects whereby an incoming message is received and analyzed to determine if the message contains any URLs. Any detected URLs are then extracted and activated to analyze the contents of the web page linked to by the URL. The HTTP response headers and content sent from a web page server in response to the browser HTTP requests to activate the URL link are analyzed to determine if the response includes a redirect to a new, or destination, URL, and associated web page, i.e., to determine if the detected URLs result in redirects. If the HTTP response indicates a redirect, a URL redirect analysis process is initiated that includes a set of redirect processing procedures that are selectively applied depending on the type of redirect encountered, and each redirect is automatically followed. For chains of redirects, the process is recursive, i.e., is repeated automatically for each redirect, from the beginning, and as if the new (destination) URL is itself an original URL.

Claims20

Dependent claims

A non-transitory computer readable storage medium having stored thereon computer executable instructions that, when executed by a computer system, cause the computer system to perform a method for automatically obtaining web page content in the presence of redirects, comprising:

receiving a message addressed to a user computing system, the message including a Uniform Resource Locator (URL);
extracting the URL from the message and label the extracted URL as an original URL extracted from the message;
activating the original URL, thereby causing a browser to send a Hypertext Transfer Protocol (HTTP) request for a web page associated with the original URL and a server to send an HTTP request response associated with the original URL, the HTTP request response associated with the original URL including a header and content;
analyzing the HTTP request response header and content associated with the original URL to determine if the original URL results in a redirect or includes one or more access issues;
determining whether the original URL results in a redirect, wherein the redirect is classified by type and is either standard or interstitial;
accessing the web page associated with the original URL upon determining that the original URL does not result in a redirect;
associating the original URL with spam without further processing upon determining that activation of the original URL results in a redirect that further comprises an interstitial page redirect from the server;
applying a URL redirect analysis process, the URL redirect analysis process applying a redirect processing procedure selected based on the type of redirect or other access issue associated with a URL upon determining that the original URL results in a standard redirect or includes one or more access issues;
associating a new URL generated by the URL redirect analysis process with the original URL and accessing the web page upon determining that, as a result of the URL redirect analysis process, access to a web page is obtained;
and
re-applying the URL redirect analysis process to all additional new URLs, or access issues, until either, as a result of an application of the URL redirect analysis process, access to a web page is obtained, or until a timeout period, or timeout iteration count, has been exceeded upon determining that, as a result of the URL redirect analysis process, the new URL also results in a redirect to an additional new URL, or includes one or more access issues.

8 more elements

A method for automatically obtaining web page content in the presence of redirects comprising:

receiving a message addressed to a user computing system, the message including a Uniform Resource Locator (URL);
extracting the URL from the message and labeling the extracted URL as an original URL extracted from the message;
activating the original URL, thereby causing a browser to send a Hypertext Transfer Protocol (HTTP) request for a web page associated with the original URL and a server to send an HTTP request response associated with the original URL, the HTTP request response associated with the original URL including a header and content;
automatically analyzing the HTTP request response header and content associated with the original URL to determine if the original URL results in a redirector includes one or more access issues;
determining whether the original URL results in a redirect, wherein the redirect is classified by type and is either standard or interstitial;
accessing the web page associated with the original URL upon determining that the original URL does not result in a redirect;
associating the original URL with spam without further processing upon determining that activation of the original URL results in a redirect that further comprises an interstitial page redirect from the server;
applying a URL redirect analysis process including a redirect processing procedure selected based on the type of redirect, or other access issue, associated with the a URL upon determining that the original URL results in a standard redirect or includes one or more access issues;
associating a new URL generated by the URL redirect analysis process with the original URL and accessing the web page upon determining that as a result of the URL redirect analysis process, access to a web page is obtained;
and
re-applying the URL redirect analysis process to all additional new URLs, or access issues, until either, as a result of an application of the URL redirect analysis process, access to a web page is obtained, or until a timeout period, or timeout iteration count, has been exceeded upon determining that, as a result of the URL redirect analysis process, the new URL also results in a redirect to an additional new URL, or includes one or more access issues.

8 more elements

A system for automatically obtaining web page content in the presence of redirects comprising:

a user computing system;
a link following computing system;
a URL analysis process implemented on the link following computing system;
a URL redirect analysis process implemented on the link following computing system;
a message directed to the user computing system;
and
one or more processors associated with one or more computing systems, the one or more processors implementing a process for automatically obtaining web page content in the presence of redirects, the process for automatically obtaining web page content in the presence of redirects comprising:
receiving the message directed to the user computing system at the link following computing system, the message including a Uniform Resource Locator (URL);
using the one or more processors associated with one or more computing systems to extract the URL from the message and label the extracted URL as an original URL extracted from the message;
using the one or more processors associated with one or more computing systems to transfer the extracted original URL to the URL analysis process;
executing the URL analysis process;
and
executing the URL redirect analysis process;
wherein the URL analysis process comprises:
activating the original URL, thereby causing a browser to send a Hypertext Transfer Protocol (HTTP) request for a web page associated with the original URL and a server to send an HTTP request response associated with the original URL, the HTTP request response associated with the original URL including a header and content;
analyzing the HTTP request response header and content associated with the original URL to determine if the original URL results in a redirect, or includes one or more access issues, the process being configured to at least determine whether activation of the original URL results in an interstitial page a redirect, wherein the redirect is classified by type and is either standard or interstitial;
accessing the web page associated with the original URL upon determining that the original URL does not result in a redirect;
associating the original URL with spam without further processing upon determining that activation of the original URL results redirect that further comprises an interstitial page redirect from the server;
and
transferring the original URL to the URL redirect analysis process upon determining that the original URL results in a standard redirect, or includes one or more access issues;
wherein the URL redirect analysis process comprises:
automatically determining the type of redirect, or access issue, associated with the original URL;
selectively applying one or more of the redirect processing procedures based on the type of redirect, or access issue, associated with the original URL;
associating a new URL with the original URL and access the web page upon determining that, as a result of the URL redirect analysis process, access to a web page is obtained;
and
re-applying the URL redirect analysis process to all additional new URLs, or access issues, until either, as a result of an application of the URL redirect analysis process, access to a web page is obtained, or until a timeout period, or timeout iteration count, has been exceeded upon determining that as a result of the URL redirect analysis process, it is determined the new URL also results in a redirect to an additional new URL, or includes one or more access issues.

23 more elements

Dependencies are inferred from the claim text.