Automatic crawling of encoded dynamic URLs

A method, comprising:

• Receiving, using a computing device, an encoded URL corresponding to a web-portal page hosted by a web-portal server, wherein a text of the encoded URL does not provide information about content of the web-portal page, the content comprising a plurality of re-arrangeable portlets, and wherein the web-portal server provides, via a representational state transfer service, to a web-crawler, a navigational state information of the web-portal page using a predetermined format;
• initializing a list of parameters, wherein the list identifies that the web-portal page corresponding to the encoded URL has been visited, and determining the parameters to add to the list by:
• sending the encoded URL to the representational state transfer service from the web-portal server hosting the web-portal page for decoding the encoded URL;
• receiving the navigational state information of the web-portal page from the representational state transfer service, wherein the navigational state information is decoded from the encoded URL, and comprises a selection-node-ID and a resource-ID, the selection-node-ID being an identifier assigned to the web-portal page, and the resource-ID being an identifier associated with a resource used by the web-portal page, wherein,
• determining a type of the encoded URL based on the received representational state information, and
• when the type is determined to be a resource URL, storing the resource-ID in the list of parameters;
• and
• when the type is determined to be an engine URL, storing the selection-node-ID in the list of parameters, and
• when the type is determined to be a portlet, storing information of the portlet in the list of parameters;
• determining if the encoded URL has been previously visited based on the list of parameters;
• and
• retrieving the content associated with the encoded URL if the encoded URL has not been previously visited.