Map/reduce controller scheduler

A big data platform for executing spark jobs in a batch pipeline used for building data products, comprising:

• a scheduler service, comprising an app written in Scala, capable of starting and/or stopping jobs, where the scheduler service abstracts away the complexity of running a Spark job;
• a controller, capable of communication with the scheduler service, the controller orchestrating and managing running Spark clusters, provisioning clusters, and managing cluster scaling;
• where the scheduler service takes as input a job request including a json object;
• where the schedule service maintains a queue of active jobs, and matches job configuration with capacity for the job to run on a cluster.