Peer-to-peer architecture for processing big data

A system for managing large datasets comprising:

• a physical network comprising a plurality of computing devices and a plurality of processors;
• a peer-to-peer (P2P) network, the P2P network comprising a plurality of nodes and a logical network derived from the physical network;
• a distributed file system for distributing data and jobs randomly across the plurality of nodes in the P2P network, by:
• receiving a file by an originating data node of the plurality of nodes, with a first processor of the plurality of processors assigned to the originating data node and being configured to:
• divide the file into a plurality of pages,
• assign a value to a first page of the plurality of pages, and
• transfer the first page of the plurality of pages to an initial responsible data node, the initial responsible node including a name that closely matches the value of the first page of the plurality of pages;
• replicating the first page of the plurality of page from the initial responsible data node to a first responsible data node and a second responsible data node;
• receiving a job at an originating job node and dividing the job into a plurality of tasks;
• routing a first task of the plurality of tasks to an initial responsible job node;
• and
• assigning the first task of the plurality of tasks to a first processing node using the initial responsible job node;
• and
• a task scheduler for delegating the first task of the plurality of tasks as necessary to optimize load distribution, by:
• assigning the first task of the plurality of tasks to a first queue of at least two queues in the first processing node, and
• forwarding the first task of the plurality of tasks to a second processing node remote from the first processing node.