How to Setup Jenkins Build Agents on Kubernetes Pods

Introduction

In previous tutorial we learnt how to set up jenkins master(jenkins controller)on local Kubernetes cluster with helm chart & argocd. For more info visit : https://cldop.com/deploying-jenkins-on-local-kubernetes-cluster-with-helm-chart-and-argocd-part-1/

In this part we will learn how to configure jenkins build agent as Kubernetes pods.

Here are the advantages of using Kubernetes pods as Jenkins build agents:

1.Enhanced Build Isolation: Running Jenkins agents on Kubernetes pods provides excellent isolation for different application versions. Each build can operate in its own dedicated environment, minimizing the risk of conflicts between builds and ensuring consistency in the build process.

2.Cost Efficiency: Kubernetes pod-based Jenkins agents operate ephemerally, spinning up only when a build request starts. This dynamic provisioning reduces CI environment costs by eliminating continuously running, idle build agents. You pay for the resources only while they are in active use.

How Do Jenkins Kubernetes Pod Agents Work?

Before getting into the implementation, let’s understand how this setup works.

The following image shows the high-level workflow.

1. Triggering a Jenkins Job

• When a developer or automation system triggers a Jenkins job (1), typically from a source control repository like GitHub, the Jenkins Controller begins the process of executing the defined pipeline.

2. API Call to Deploy Agent Pods

• The Jenkins Kubernetes plugin, which is installed in the Jenkins Controller, makes an API request to the Kubernetes API Server (2). This request is to deploy a new Jenkins agent pod specifically for this job. This step ensures that each Jenkins job has a fresh, isolated environment for execution, avoiding conflicts with other jobs.

3. Deploying the Agent Pod

• Upon receiving the API request, Kubernetes deploys the requested Jenkins agent pod (3). This pod is a lightweight, disposable instance that will execute the build steps defined in the Jenkinsfile. The pod is configured with environment variables that include details such as the Jenkins server URL, agent name, and authentication secrets.

4. Connecting the Agent Pod to Jenkins

• Once the agent pod is up and running, it uses the provided environment variables to establish a connection back to the Jenkins Controller (4). This connection is made using the JNLP (Java Network Launch Protocol) method over a predefined port (typically port 50000 as shown in the diagram). JNLP allows the agent to securely communicate with the Jenkins Controller and receive instructions on the build steps to execute.

5. Executing Build Steps

• The Jenkins Controller sends the build steps defined in the Jenkinsfile to the agent pod (5). The pod then executes these steps, such as compiling code, running tests, packaging the application, etc. Since each agent pod is isolated, it ensures that the build process is clean and does not interfere with other builds or jobs.

6. Post-Build Completion

• After the build steps are completed, the Kubernetes plugin handles the cleanup process. The Jenkins Controller sends a request to the Kubernetes API Server to terminate the agent pod (6), unless it’s configured to be retained for debugging purposes.

7. Build Notifications

• Once the build is complete, Jenkins sends out notifications (7) to the designated communication channels like Slack or email. This step keeps the development team informed about the build status, whether it’s a success or failure.

8. Pod Deletion

• Finally, the Kubernetes API Server deletes the agent pod (8), freeing up resources and ensuring that the CI/CD environment remains cost-effective by only using resources when necessary.

Setting Up Jenkins Build Pods On Kubernetes

To work on this setup, we need the following.

1. A working Kubernetes cluster (Here we have a local k8s cluster running on ubuntu via kind)
2. A running Jenkins controller(jenkins master)-Already we have a running jenkins deployed via helm & argocd
3. Install Jenkins Kubernetes Plugin on Jenkins Controller

4. Create a Kubernetes Cloud Configuration

Since we have completed 2 steps already will discuss about 3 & 4 step only

Jenkins Kubernetes Plugin Configuration

Jenkins Kubernetes plugin is required to set up Kubernetes-based build agents. Let’s configure the plugin.

Step 3-Install Jenkins Kubernetes Plugin

Go to Manage Jenkins –> Manage Plugins, search for the Kubernetes Plugin in the available tab, and install it.

Step 4-Create a Kubernetes Cloud Configuration

Once installed, go to Manage Jenkins –> Clouds

Since we have deployed jenkins kubernetes plugin we can see that under clouds section

Here in our case Jenkins server running inside the same Kubernetes cluster.So we don’t have to mention the Kubernetes URL or certificate key.

However, to validate the connection use the Test Connection button as shown below. It should show a connected message if the Jenkins pod can connect to the Kubernetes API Server.

Since the Jenkins Controller running inside the cluster, you can use the Kubernetes cluster’s service endpoint as the Jenkins URL because agent pods can connect to the cluster via internal service DNS.

The URL is derived using the following syntax.

http://<service-name>.<namespace>.svc.cluster.local:8080

In our case, the service DNS will be,

http://jenkins.jenkins.svc.cluster.local:8080

namespace is jenkins

The pod label with the key jenkins/jenkins-jenkins-agent and value true in the Jenkins Kubernetes cloud configuration is used to identify and manage the pods that Jenkins dynamically provisions as agents.

Here’s what it means:

1. Labeling: This label is automatically applied to the agent pods that Jenkins creates. The key jenkins/jenkins-jenkins-agent is a namespace-specific label, meaning it’s used within the Jenkins namespace (or any specific namespace you may have configured for your Jenkins installation).
2. Selection Criteria: The label helps Jenkins to recognize and manage the pods as Jenkins agents. Jenkins uses this label to ensure that it only interacts with and manages the pods that it is responsible for, which are those created as build agents.
3. Helm Chart Configuration: Since we’ve installed Jenkins using the official Helm chart, this label is pre-configured to be applied to all dynamic Jenkins agent pods. This ensures that Jenkins can track, manage, and clean up these pods appropriately.
4. Pod Management: When Jenkins needs to provision an agent pod to execute a job, it uses this label to keep track of the pods it has deployed. After the job is done, Jenkins can identify these pods and terminate them if necessary.

POD and Container Template

To view the POD configuration got to Clouds-Kubernetes -Pod Templates section

We can see that by default we can see one Pod templates named “default”

For easy understanding we can modify that name as “k8s-agent”

Also change the label name from jenkins-jenkins-agent (Might have created due to helm chart deployment) to jenkins-agent for simplicity

The label jenkins-agent will be used as an identifier to pick this pod as the build agent with this name, when describing the job.

The other configuration is the container template. If you don’t add a container template, the Jenkins Kubernetes plugin will use the default JNLP image from the Docker hub to spin up the agents. ie, jenkins/inbound-agent

We can add multiple container templates to the POD template and use them in the pipeline.Will implement that also later.

Now save all the configurations, and let’s test if we can build a job with a pod agent.

Go to Jenkins home –> New Item and create a freestyle project.

In the job description, add the label jenkins-agent as shown below. It is the label we assigned to the pod template. This way, Jenkins knows which pod template to use for the agent container.

Add a shell build step with an echo command to validate the job as shown below.

Now, save the job configuration and click “Build Now”

You should see a pending agent in the job build history. In a couple of minutes because it may take some time to start because it will try to download the image from docker hub then you will see a successful build.

If you check the logs, the executed shell will be shown.

Jenkinsfile With Pod Template

Whatever we have seen till now is to understand and validate the Kubernetes Jenkins plugin setup.

When it comes to actual project pipelines, it is better to have the POD templates in the Jenkinsfile

Here is what you should know about the POD template.

1. By default, the plugin uses a JNLP container image to connect to the Jenkins server. You can override with a custom JNLP image provided you give the name jnlp in the container template.
2. You can have multiple container templates in a single pod template. Then, each container can be used in different pipeline stages.
3. POD_LABEL will assign a random build label to the pod when the build is triggered. You cannot give any other names other than POD_LABEL

Here is an example Jenkinsfile with a POD template.

podTemplate {
    node(POD_LABEL) {
        stage('Run shell') {
            sh 'echo Testing jenkins agent'
        }
    }
}

Building the above Jenkinsfile in a pipeline job will use the default JNLP image and execute the commands in the “Run Shell” stage. When I say default, the plugin will use the JNLP image from the docker hub if you don’t specify any.

Here I am going to add maven image under Container Template section as below so that can build maven applications using that container.

Mention Name : maven & Docker image name: maven:3.8.1-jdk-8

you can go to Jenkins main page and clicking on the new item, add a new pipeline job then I write simple script pipeline for the running pipeline example. Now we are ready, I can get a build on jenkins.

podTemplate(containers: [
    containerTemplate(
        name: 'maven', 
        image: 'maven:3.8.1-jdk-8',
    )
]) {
    node("jenkins-agent") {
        stage('Get a Maven project') {
            container('maven') {
                stage('Check Maven Version') {
                    sh 'mvn --version'
                }
            }
        }
    }
}

Conclusion

If you are using Jenkins & Kubernetes, you should definitely try out the container-based agents.

Scaling your Jenkins agents on Kubernetes helps you avoid the administrative overhead associated with static build VMs.You don’t have to worry about running out of resources for Jenkins builds.