Dynamic Provisioning of Agents in Jenkins Using Kubernetes or Docker

As your Jenkins workloads scale, managing agents efficiently becomes critical. One way to handle this challenge is through dynamic agent provisioning, which allows Jenkins to create and destroy agents on-demand. Instead of relying on a fixed set of agents, dynamic provisioning scales Jenkins agents dynamically according to the job requirements.

Two common methods for dynamic agent provisioning in Jenkins are:

Using Kubernetes: With the Kubernetes plugin, Jenkins can automatically create new agents (pods) on a Kubernetes cluster when needed, scaling as per the load.
Using Docker: With the Docker plugin, Jenkins can spin up agents in isolated Docker containers, utilizing Docker to provide on-demand environments.

This blog post will cover how to dynamically provision Jenkins agents using Kubernetes and Docker, walking you through the setup, configuration, and best practices for both methods.

1. Introduction to Dynamic Agent Provisioning

Traditionally, Jenkins uses a pool of static agents to handle job execution. However, as CI/CD pipelines evolve, the number of builds, tests, and deployments grows, putting more demand on the existing agents. A static set of agents may lead to under-utilization during idle periods and resource exhaustion during peak loads.

Dynamic agent provisioning allows Jenkins to create agents on-the-fly, depending on job demand. This means that agents are spun up when needed and destroyed when the job completes. Two powerful ways to achieve this dynamic scaling in Jenkins are through Kubernetes and Docker.

2. Benefits of Dynamic Agent Provisioning

Dynamic agent provisioning provides several key benefits:

Scalability: Agents are created and destroyed on-demand, allowing Jenkins to scale seamlessly according to load.
Cost Efficiency: Resources are used only when jobs are running. Idle agents do not consume compute resources.
Isolation: Each job can run in its own isolated agent (pod or container), minimizing the risk of conflicts between jobs.
Resource Utilization: Only required resources are allocated, ensuring that no extra agents consume compute power unnecessarily.
Custom Environments: Each agent can be customized with specific dependencies, allowing tailored environments for different jobs.

3. Prerequisites

Before setting up dynamic agent provisioning in Jenkins, make sure you have the following:

Jenkins Master: A running Jenkins instance with administrative access.
Kubernetes Cluster (if using Kubernetes): A Kubernetes cluster where Jenkins can schedule pods.
Docker Engine (if using Docker): A Docker host or Docker Swarm for Jenkins to manage Docker containers.
Jenkins Plugins: Ensure the necessary plugins for Kubernetes or Docker are installed (we will cover the installation steps below).
Basic Knowledge of Jenkins Pipelines: Familiarity with writing Jenkinsfiles and configuring Jenkins jobs.

4. Setting Up Dynamic Agents with Kubernetes

In a Kubernetes-based dynamic agent setup, Jenkins uses the Kubernetes plugin to dynamically provision agents as pods in a Kubernetes cluster. When a job is triggered, a new pod is created, and once the job completes, the pod is destroyed.

4.1 Installing the Kubernetes Plugin

To get started, install the Kubernetes plugin on your Jenkins instance.

Log in to your Jenkins dashboard as an administrator.
Navigate to Manage Jenkins > Manage Plugins.
Under the Available tab, search for Kubernetes.
Select the Kubernetes plugin and click Install without restart.

4.2 Configuring Kubernetes Cloud in Jenkins

Once the plugin is installed, you'll need to configure Jenkins to connect to your Kubernetes cluster.

Go to Manage Jenkins > Manage Nodes and Clouds.
Under the Clouds section, click Add a new cloud and select Kubernetes.
Configure the Kubernetes Cloud settings:
- Kubernetes URL: Provide the URL of your Kubernetes API server (e.g., https://your-k8s-api-server).
- Jenkins URL: Provide the URL of your Jenkins instance (e.g., http://your-jenkins-url).
- Credentials: Add credentials to authenticate Jenkins with your Kubernetes cluster (e.g., service account credentials or a kubeconfig file).
Click Save to apply the changes.

4.3 Defining Pod Templates

Next, you’ll define pod templates that specify the structure of the Jenkins agents (pods) that will be created.

In the Kubernetes Cloud configuration, scroll down to the Pod Templates section.
Click Add Pod Template.
Configure the pod template:
- Name: Give the pod template a descriptive name.
- Labels: Assign labels that can be used in Jenkins pipelines to select this pod template.
- Containers: Define the containers within the pod. For example, you might add a container with an image like maven:3.6.3-jdk-8 for Java builds.
Save the configuration.

4.4 Example Jenkinsfile for Kubernetes Agents

Once your Kubernetes cloud and pod templates are configured, you can use the following example Jenkinsfile to dynamically provision agents on Kubernetes:

pipeline {
    agent {
        kubernetes {
            label 'k8s-agent'
            defaultContainer 'jnlp'
            yaml """
            apiVersion: v1
            kind: Pod
            spec:
              containers:
              - name: maven
                image: maven:3.6.3-jdk-8
                command:
                - cat
                tty: true
            """
        }
    }
    stages {
        stage('Build') {
            steps {
                container('maven') {
                    sh 'mvn clean install'
                }
            }
        }
        stage('Test') {
            steps {
                container('maven') {
                    sh 'mvn test'
                }
            }
        }
    }
}

In this example, Jenkins will dynamically create a Kubernetes pod with a Maven container for the build and test stages.

5. Setting Up Dynamic Agents with Docker

In a Docker-based dynamic agent setup, Jenkins uses the Docker plugin to spin up containers on demand. These containers act as Jenkins agents and are created and destroyed based on job requirements.

5.1 Installing the Docker Plugin

To begin, you’ll need to install the Docker plugin.

Log in to Jenkins as an administrator.
Go to Manage Jenkins > Manage Plugins.
Under the Available tab, search for Docker.
Select the Docker plugin and click Install without restart.

5.2 Configuring Docker Cloud in Jenkins

Once the plugin is installed, configure Jenkins to connect to your Docker host.

Go to Manage Jenkins > Manage Nodes and Clouds.
Under the Clouds section, click Add a new cloud and select Docker.
Configure the Docker Cloud settings:
- Docker Host URI: Enter the URI of your Docker host (e.g., tcp://your-docker-host:2376).
- Credentials: Add credentials to authenticate Jenkins with the Docker host.
- Connect Timeout and Read Timeout: Set appropriate timeouts for Docker operations.
Save the configuration.

5.3 Defining Docker Agent Templates

Define Docker agent templates that specify how the Jenkins agents (containers) should be created.

In the Docker Cloud configuration, scroll down to the Docker Agent Templates section.
Click Add Docker Template.
Configure the template:
- Labels: Assign labels to this template, which can be used to refer to the agent in your Jenkins pipelines.
- Docker Image: Specify the Docker image to use for the agent (e.g., jenkins/agent or a custom image).
- Volumes: Define any volumes that should be mounted in the container.
Save the configuration.

5.4 Example Jenkinsfile for Docker Agents

Here’s an example Jenkinsfile that dynamically provisions a Docker container

as an agent for a pipeline job:

pipeline {
    agent {
        docker {
            image 'maven:3.6.3-jdk-8'
            args '-v /root/.m2:/root/.m2'
        }
    }
    stages {
        stage('Build') {
            steps {
                sh 'mvn clean install'
            }
        }
        stage('Test') {
            steps {
                sh 'mvn test'
            }
        }
    }
}

In this example, Jenkins will dynamically spin up a Docker container using the maven:3.6.3-jdk-8 image to execute the build and test stages.

6. Best Practices for Dynamic Agent Provisioning

When dynamically provisioning agents in Jenkins, it’s essential to follow these best practices to ensure efficient and stable operation:

Resource Allocation: Ensure that Kubernetes nodes or Docker hosts have sufficient resources (CPU, memory) to handle the dynamic provisioning of agents.
Label Management: Use descriptive labels for agent templates to easily manage and assign them to different pipelines.
Monitor Performance: Continuously monitor agent performance and pipeline execution times to optimize the allocation of resources.
Use Custom Images: Customize agent images (Docker or Kubernetes) to preinstall dependencies required by your jobs, reducing the time spent installing tools during runtime.
Clean Up Resources: Ensure that agents (pods or containers) are automatically destroyed after job completion to free up resources.

Conclusion

Dynamic agent provisioning with Kubernetes and Docker offers a powerful solution for scaling Jenkins workloads efficiently. By provisioning agents on-demand, you can ensure that your CI/CD pipelines are both cost-effective and scalable, with the ability to handle any load dynamically.

In this post, we covered the steps to set up dynamic agents with both Kubernetes and Docker, along with example configurations and Jenkinsfiles. Whether you’re running a Kubernetes cluster or Docker infrastructure, dynamic provisioning will help you build a more flexible and efficient Jenkins setup.