Job Execution Systems: What is the difference between Jenkins, Rundeck, Airflow, Gitlab CI and others

Illustration of a young girl lying on her stomach playing with a row of stylized circles that resemble cryptocurrency logos.

Let's say you have a job to run.

This could be patching a server, making a database backup, validating configuration files, processing some data or whatever else you have to do.

Illustration of a person holding a coffee mug and laptop with a to-do list containing tech tasks like patching a server, beside a logo for "mkdev."

The first decision to make is where to run this job. Let's call such place a "job server".

Schematic illustration showing a labeled box representing a 'Job Server' with a nested, smaller box inside labeled 'The Job', with 'mkdev' logo in bottom right corner.

Your job relies on certain programming languages, system packages and source code. "A job server" has all of them available.

Illustration of a job server with components like dependencies and the job encapsulated within, and a list including programming languages and other dependencies.

The next decision to make is who exactly is running this job. Let's call it a "job executor". At this point, you are the "job executor" - you go to your "job server" and just run the job by hand.

Illustration of a 'Job Executor' person with a laptop and coffee connecting to a 'Job Server' showing the execution of a job with dependencies.

But then you have to run another job, which needs another "job server". And then another one. As times goes by, you end up with lots of various jobs and many different "job servers" to run them.

Illustration of a woman labeled as "Job Executor" holding a mug and laptop, with arrows pointing to "Job Server A" containing "The Job A" and "Job Server B" with "The Job B1" and "The Job B2". The mkdev logo is at the bottom right.

You don't want to be the "job executor" anymore. So you need another part of the system, which we will call a "job scheduler".

Flowchart showing decision-making for job execution with a "Job" block, a "Job Scheduler" with the question "Where to run?", and arrows pointing to "Job Server A" and "Job Server B".

The very basic feature a "job scheduler" needs to have is to be able to understand on which particular "job server" to run your job.

Illustration of a woman with glasses holding a coffee cup and a laptop with continuous integration pipeline results displayed on the screen.

In addition, it would be very convenient if this "job scheduler" also had some kind of an interface, where you can see the history of your job executions and from which you can trigger your jobs.

"A workflow diagram displaying 'Test' and 'Deploy' stages with checkmarks next to 'audit' and 'test' under 'Test,' and a checkmark next to 'deploy' under 'Deploy', indicating completed tasks."

A really good "job scheduler" would be also capable of chaining your jobs together, allowing you to build complex workflows and pipelines.

Screenshot of a software interface for scheduling automated tasks with fields for description, interval pattern, cron timezone, target branch, variables, and an 'Activated' checkbox.

Ideally, by the way, you don't even start jobs yourself anymore. You let your automation do it for you, because your "job scheduler" provides a number of ways to trigger jobs, like an API or a scheduled execution.

A diagram illustrating a Job Execution System with two components labeled 'Job Scheduler' and 'Job Server' connected by a line, with 'Job Executor' as a subcomponent within the 'Job Server'.

All together - the job executors, the job servers and the job scheduler comprise something we could call "a job execution system".

This simple concepts are hidden under different names inside numerous different systems. Those systems were built specifically to solve particular class of job execution problems. Let's take a look at some of them.

One of the most popular applications of a job execution system is to provide CI/CD for your applications.

Illustration of Jenkins continuous integration setup, showing a Jenkins Server connected to an Any Server with a Job Executor and Jenkins Agent indicated within dotted lines.

If we will look at Jenkins, the Jenkins Server is a "job scheduler", "job server" is any server that has "jenkins agent" installed and "jenkins agents" are your "job executors".

Your typical jobs inside a Jenkins "job execution system" is to build, test and deploy your code. Jenkins has a huge amount of different features and third party plugins to make running these kind of jobs especially easy and convenient. Another "job execution system" for CI/CD is Gitlab CI.

A schematic representation of GitLab CI showing a GitLab Server connected to Any Server with components labeled "Job Executor" and "GitLab Runner" inside a dashed box.

Gitlab Server is a "job scheduler", "job server" is any server that has a "gitlab runner" installed and the "gitlab runner" is your "job executor". Same as Jenkins, Gitlab was purpose built to build, test and deploy your code, so it's a great job execution system for this particular purpose.

Schematic overview of a Rundeck server connection to a generic server with a Job Executor via SSHD, including the Rundeck logo and mkdev branding.

Rundeck, a popular automation tool, was not built with the CI/CD in mind. It's a more generic system, in which Rundeck server is a "job scheduler" and any other server that Rundeck can connect to is a "job server". Rundeck is perfect for doing various server automation task, like applying latest patches. It's pretty flexible, which can be both an advantage and a disadvantage.

Logos of Apache Spark and Apache Airflow with respective stylized icons on left and text on right, against a white background with "mkdev" in the lower right corner.

Some other examples of a "job execution engine" could be Apache Spark, that is more on a big data and analytics and machine learning side, or Airflow, that is quite popular for really complex workflows and data processing.

A diagram of a Job Execution System with a 'Job Scheduler' linked to a 'Job Server,' which contains a dashed outline of a 'Job Executor,' and a 'mkdev' logo in the corner.

It's a bit confusing sometimes to understand which tool to use to run jobs. When you start thinking about those "job execution engines", you begin to realize that beneath the different interace and slighlty different set of features they all come down to the same simple concepts.

What doesn't help is the fact that under one "job execution engine" hindes another one. For example, you can connect Kubernetes cluster to Jenkins server, so that Jenkins creates new temporal "jenkins agents" as Kubernetes pods.

Illustration of a Jenkins pipeline workflow with Kubernetes integration showing scheduling between Kubernetes and a Jenkins Agent within a Kubernetes Pod on a Node.

In such case, under the Jenkins "job execution" engine hides another one, called "Kubernetes Scheduler", that is responsible to schedule "pods", which are basically "job executors" on the Kubernetes nodes - which are basically "job servers".

Illustration of Kubernetes architecture showing a Kubernetes Node connected to a nested Kubernetes Job which contains a Kubernetes Pod, with labels and logos.

And to make this particular case too confusing too handle, Kubernetes itself has a native concept of Jobs and CronJobs. And I did not even mention Tekton, that is a cloud native framework for building CI/CD systems.

An illustrated diagram showing the integration of Kubernetes with Tekton CI/CD system, featuring Kubernetes logo, Tekton mascot, and a person gesturing towards the workflow with annotations.

DevOps consulting: DevOps is a cultural and technological journey. We'll be thrilled to be your guides on any part of this journey. About consulting

To sum it up, there are many many ways to run jobs these days. If I would give few tips for picking the right tool, they would sound like this:

An educational diagram showing 'Tip #1: Learn for what the tool was built' with a Jenkins Server connected to an 'Any Server' with a Jenkins Agent, the connection marked with a red 'X', and a script named 'patch_server.sh'.

First, try to understand for which purpose each tool was created. For example, you could totally run your data processing jobs inside Jenkins. You could also patch your servers with Airflow. But only because you can do it, doesn't mean you should. Some tools are perfect for CI/CD, others for other things.

Illustration presenting Tip #2: Learn what integrates with what best, with GitLab CI and Bitbucket logos, an 'X' symbol denoting incompatibility and the mkdev logo.

Second, check which of the tools integrates with your existing system best. You would not use Gitlab CI if your primary source code repository is Bitbucket or Github. Similarly, if your operations team is effecient with Ansible, then check if Ansible Tower makes more sense rather than Rundeck or Jenkins.

Illustration of tip #3 "Keep global concepts in mind" showing a workflow with a "Job Scheduler" connected to a "Job Server" that encloses a "Job Executor." Brand logo "mkdev" at the bottom right.

And third, always keep the global concepts in mind. Underneath all of those tools you will find something like a job server, a job executor and a job scheduler. The web interface might be different, integrations might vary and different terminology might be used. But in the end it all comes down to the same set of basic components. Don't let the plethora of tools and tools specific features to discourage you.

Here's the same article in video form, so you can listen to it on the go: