Pipeline as code with Jenkins This chapter covers How pipeline as code works with…
- User
Pipeline as code with Jenkins This chapter covers How pipeline as code works with Jenkins An overview of Jenkinsfile structure and syntax Introduction to Blue Ocean, the new Jenkins user experience Declarative versus scripted Jenkins pipelines Integration of a GitFlow model within Jenkins projects Tips for productivity and efficiency while writing Jenkinsfiles for complex CI/CD pipeline
There’s no doubt that cloud computing has had a major impact on the way companies build, scale, and maintain technology products. The ability to click a few buttons to provision machines, databases, and other infrastructure has led to an increase in developer productivity we’ve never seen before. → cloud = increase agility in software teams While it was easy to spin up simple cloud architectures, mistakes can easily be made while provisioning complex ones. Human error will always be present, especially when you can launch cloud infrastructure by clicking buttons on the cloud provider’s web console.
The only way to avoid these kinds of errors is through automation, and infrastructure as code (IaC) is helping engineers automatically launch cloud environments quickly and without mistakes. The growth of DevOps and the adoption of its practices have led to more tooling that can implement the IaC paradigm to a larger degree.
In the past, setting up CI/CD workflow has been a manual process. It was commonly done via defining a series of individual jobs for the various pipeline tasks. Each job was configured via web forms—filling in text boxes, selecting entries from dropdown lists, and so forth. And then the series of jobs were strung together, each triggering the next, into a pipeline. Jenkins somewhat lagged in this area until the release of Jenkins 2. Although widely used and a primary workflow tool for creating CI/CD pipelines, this way of creating and connecting Jenkins jobs to form a pipeline was challenging. It did not meet the definition of IaC. Job configurations were stored only as Extensible Markup Language (XML) files within the Jenkins configuration area. This meant that the files were not easily readable or directly modifiable. And the Jenkins application itself provided the user’s primary view and access to them. NOTE Jenkins 2 is the name we are generally applying to newer versions that support the pipeline-as-code functionality, as well as other features. → from jenkins 2 version the concept of pipeline as code is introduced
Introducing the Jenkinsfile Pipeline as code (PaC) describes a set of features that allow Jenkins users to define pipelined job processes with code, stored and versioned in a source repository. These features allow Jenkins to discover, manage, and run jobs for multiple source repositories and branches—eliminating the need for manual job creation and management.
PaC helps you automate the CI/CD workflows in a repeatable, consistent manner, which has many benefits: Speed—You can quickly and easily write a CI/CD workflow for sandbox, staging, and production environments, which can help you deliver your product on time. Consistency—PaC completely standardizes the setup of CI/CD, so there’s a reduced possibility of any human errors or deviations. Risk management—Because the pipeline can be version-controlled, PaC allows every change to your CI/CD workflow to be documented, logged, tracked, and tested just like application code. Hence, you can revert to a working version in case of failure. Efficiency—It minimizes the introduction of human errors and helps your application’s deployment run more smoothly.
The bottom line is simple: adopting the PaC paradigm will create a culture that generates better software, and will save you a lot of money, time, and headaches trying to implement complex CI/CD workflows through UIs and web forms. So how does PaC work with Jenkins? → adopting Pac paradigm will create a culture that generates better software → save mony → save times —> and headaches The core ideas presented about using Pipeline as Code (PaC) with Jenkins are: Jenkinsfile for PaC: Projects using Jenkins for CI/CD should have a Jenkinsfile at the root of their repository. This file defines the pipeline. Stages Define the Pipeline: The Jenkinsfile contains stages, which are the individual steps or tasks in the CI/CD process (e.g., build, test, deploy). Automated Pipeline Execution: Whenever code is pushed to the repository, Jenkins automatically executes the pipeline defined in the Jenkinsfile. This is the core of continuous integration. Jenkinsfile Under Version Control: The Jenkinsfile lives alongside the source code and is treated like any other code file. This enables: Versioning: Track changes to the pipeline process itself. Collaboration: Multiple developers can work on and improve the pipeline. Rollback: Revert to earlier versions of the pipeline if necessary.
Code Reviews for Pipelines: Just like application code, the Jenkinsfile can undergo code reviews. This ensures quality, consistency, and best practices in the pipeline definition. In essence, the passage emphasizes that using a Jenkinsfile brings the CI/CD pipeline under version control, making it more manageable, collaborative, and reliable. It treats the pipeline as code, subject to the same best practices as application code. Jenkinsfile uses a domain-specific language (DSL) based on the Groovy programming language to define the entire CI/CD workflow. Figure 2.1 is an example of a classic CI/CD workflow. —> Jenkinsfile use domain-specific language (DSL) based on the Groovy programming language Those phases can be described in a Jenkinsfile by using the stage keyword. A stage is a block that contains a series of steps. It can be used to visualize the pipeline process. The following listing is an example of a simple Jenkinsfile for figure 2.1. → stage is a block that contains a series of steps , → it can be visualize as the pipeline process The following listing is an example of a simple Jenkinsfile for figure 2.1. node('workers'){ try { stage('Checkout'){ checkout scm } stage('Quality Test'){ echo "Running quality tests" } stage('Unit Test'){ echo "Running unit tests" } stage('Security Test'){ echo "Running security checks" } stage('Build'){ echo "Building artifact" } stage('Push'){ echo "Storing artifact" } Listing 2.1 Jenkinsfile stages 24 CHAPTER 2 Pipeline as code with Jenkins stage('Deploy'){ echo "Deploying artifact" } stage('Acceptance Tests'){ echo "Running post-integrations tests" } } catch(err){ echo "Handling errors" } finally{ echo "Cleaning up" } } We’ll dive deep into the syntax in the next chapter, but for now, let’s focus on what the stages are doing: Checkout—Pulls the latest changes from the source code repository, which can be GitHub, Bitbucket, Mercurial, or any SCM. Quality tests—Contains instructions on how to execute static code analysis to measure code quality, and identify bugs, vulnerabilities, and code smell. It can be automated by integrating external tools like SonarQube to fix code-quality violations and reduce technical debt. Unit tests—In this stage, unit tests are executed. If tests are successful, a code coverage report will be generated that can be consumed by Jenkins plugins to show a visual overview of the project’s health and keep track of the code coverage metrics as your project grows. Code coverage can be an indication of how much your application code is executed during your tests, and can give some indication as to how well your team is applying good testing practices such testdriven development (TDD) or behavior-driven development (BDD). Security tests—Responsible for identifying project dependencies and checks if any known, publicly disclosed vulnerabilities exist. A security report will be published with the total number of findings grouped by severity (critical, high, medium, or low). A well-known open source Jenkins plugin is OWASP Dependency-Check (http://mng.bz/MvR7). Build—In this phase, the needed dependencies will be installed, the source code will be compiled, and an artifact will be built (Docker image, zip file, Maven JAR, and so forth). Push—The artifact built in the previous stage will be versioned and stored in a remote repository. Deploy—In this stage, the artifact will be deployed to a sandbox/testing environment for quality assurance or to production after the user has approved the deployment. Acceptance tests—After the changes are deployed, a series of smoke and validation tests will be executed against the deployed application to verify that the application is running as expected. The tests can be simple health checks with cURL commands or sophisticated API calls. Introducing the Jenkinsfile 25 If any of these stages throws an exception or error, the pipeline build’s status will be set to fail. This default behavior can be overridden by using try-catch blocks. The finally block can be used to clean up the Jenkins workspace (temporary files or build packages) or to execute post-script commands such as sending Slack notifications to alert the development team about the build status.
Q- just for the proof of concept using docker , spring boot , and jenkins give me step by step tutor from spring insilser inter running the pipeline how to create this demo ?
- Assistant
Creating a proof of concept (PoC) that integrates Spring Boot, Docker, and Jenkins for a CI/CD pipeline involves several steps. This tutorial will guide you through the process, from setting up a Spring Boot application using Spring Initializr to automating the build, test, and deployment processes with Jenkins Pipeline as Code.
Prerequisites
Before you begin, ensure you have the following tools installed:
- Java Development Kit (JDK): Preferably version 11 or higher.
- Maven: For building the Spring Boot application.
- Git: For version control.
- Docker: For containerizing the application.
- Jenkins: For setting up the CI/CD pipeline.
- GitHub Account: To host your repository.
Step 1: Create a Spring Boot Application Using Spring Initializr
-
Access Spring Initializr:
- Navigate to Spring Initializr in your web browser.
-
Configure Your Project:
- Project: Maven Project
- Language: Java
- Spring Boot: Choose the latest stable version (e.g., 3.0.0)
- Project Metadata:
- Group:
com.example - Artifact:
demo - Name:
demo - Package Name:
com.example.demo
- Group:
- Packaging: Jar
- Java: 11 or higher
-
Add Dependencies:
- Spring Web: For building RESTful APIs.
- Spring Boot DevTools: For development convenience.
-
Generate the Project:
- Click on the "Generate" button to download the project as a ZIP file.
-
Extract and Import:
- Extract the ZIP file.
- Import the project into your favorite IDE (e.g., IntelliJ IDEA, Eclipse).
-
Create a Simple REST Controller:
// src/main/java/com/example/demo/DemoApplication.java package com.example.demo; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; @SpringBootApplication public class DemoApplication { public static void main(String[] args) { SpringApplication.run(DemoApplication.class, args); } } // src/main/java/com/example/demo/HelloController.java package com.example.demo; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RestController; @RestController public class HelloController { @GetMapping("/hello") public String hello() { return "Hello, World!"; } } -
Test the Application:
- Run the application.
- Navigate to
http://localhost:8080/helloto verify it returns "Hello, World!"
Step 2: Containerize the Application with Docker
-
Create a
Dockerfile:In the root directory of your project, create a file named
Dockerfilewith the following content:# Use an official OpenJDK runtime as the base image FROM openjdk:17-jdk-alpine # Set the working directory WORKDIR /app # Add the application's jar to the container COPY target/demo-0.0.1-SNAPSHOT.jar app.jar # Expose port 8080 EXPOSE 8080 # Define the entry point ENTRYPOINT ["java","-jar","app.jar"] -
Update
pom.xmlfor Packaging:Ensure your
pom.xmlis set to create an executable jar:<build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> -
Build the Jar:
Open your terminal, navigate to the project directory, and run:
mvn clean packageThis will generate the jar file in the
targetdirectory. -
Build the Docker Image:
docker build -t demo-app:latest . -
Run the Docker Container:
docker run -p 8080:8080 demo-app:latest -
Verify:
Navigate to
http://localhost:8080/helloto ensure the application is running inside Docker.
Step 3: Initialize a Git Repository and Push to GitHub
-
Initialize Git:
git init git add . git commit -m "Initial commit" -
Create a Repository on GitHub:
- Log in to your GitHub account.
- Click on "New" to create a new repository (e.g.,
demo-poc).
-
Add Remote and Push:
git remote add origin https://github.com/your-username/demo-poc.git git branch -M main git push -u origin main
Step 4: Set Up Jenkins for CI/CD
Option 1: Install Jenkins Locally
-
Download Jenkins:
- Visit the Jenkins Download Page and choose your OS.
-
Install Jenkins:
- Follow the installation instructions for your operating system.
-
Access Jenkins:
- Once installed, navigate to
http://localhost:8080/in your browser. - Complete the initial setup by following on-screen instructions.
- Once installed, navigate to
Option 2: Use Jenkins Docker Image
-
Pull Jenkins Image:
docker pull jenkins/jenkins:lts -
Run Jenkins Container:
docker run -p 8080:8080 -p 50000:50000 -v jenkins_home:/var/jenkins_home jenkins/jenkins:lts -
Access Jenkins:
- Open
http://localhost:8080/in your browser. - Complete the initial setup as prompted.
- Open
Step 5: Configure Jenkins
-
Install Necessary Plugins:
- Pipeline: Enables Jenkins Pipeline support.
- Docker Pipeline: Integrates Docker with Jenkins.
- Git: For Git integration.
- Blue Ocean: Provides a modern UI for Jenkins.
To install plugins:
- Navigate to
Manage Jenkins>Manage Plugins>Available. - Search for each plugin and install them.
-
Create Jenkins Credentials:
-
GitHub Credentials:
- Navigate to
Manage Jenkins>Manage Credentials>(global)>Add Credentials. - Choose Username with password or Personal Access Token (recommended for security), and enter your GitHub details.
- Navigate to
-
Docker Hub Credentials (Optional):
- If you plan to push Docker images to Docker Hub, add Docker Hub credentials similarly.
-
Step 6: Create a Jenkins Pipeline with
Jenkinsfile-
Define the Pipeline Stages:
Create a
Jenkinsfilein the root directory of your project repository with the following content:pipeline { agent any environment { // Docker Hub credentials ID (if pushing to Docker Hub) DOCKER_CREDENTIALS = 'docker-hub-credentials-id' DOCKER_IMAGE = 'your-dockerhub-username/demo-app' } stages { stage('Checkout') { steps { git branch: 'main', url: 'https://github.com/your-username/demo-poc.git', credentialsId: 'github-credentials-id' } } stage('Build') { steps { sh 'mvn clean package' } } stage('Build Docker Image') { steps { script { dockerImage = docker.build "${env.DOCKER_IMAGE}:latest" } } } stage('Push Docker Image') { steps { script { docker.withRegistry('https://registry.hub.docker.com', "${env.DOCKER_CREDENTIALS}") { dockerImage.push() } } } } stage('Deploy') { steps { // Example deployment step (e.g., running the Docker container) sh """ docker stop demo-app || true docker rm demo-app || true docker run -d -p 8080:8080 --name demo-app ${env.DOCKER_IMAGE}:latest """ } } } post { success { echo 'Pipeline completed successfully!' } failure { echo 'Pipeline failed. Check the logs.' } } }Explanation of Stages:
- Checkout: Clones the repository from GitHub.
- Build: Compiles the Spring Boot application using Maven.
- Build Docker Image: Builds a Docker image from the
Dockerfile. - Push Docker Image: Pushes the Docker image to Docker Hub.
- Deploy: Deploys the Docker container (this example stops any existing container and runs a new one).
-
Commit and Push
Jenkinsfile:git add Jenkinsfile git commit -m "Add Jenkinsfile for CI/CD pipeline" git push
Step 7: Configure Jenkins to Use the Jenkinsfile
-
Create a New Pipeline Job:
- In Jenkins, click on New Item.
- Enter an item name (e.g.,
demo-poc-pipeline). - Select Pipeline and click OK.
-
Configure the Pipeline:
- Description: (Optional) Add a description for your pipeline.
- Pipeline Section:
- Definition: Pipeline script from SCM
- SCM: Git
- Repository URL:
https://github.com/your-username/demo-poc.git - Credentials: Select the GitHub credentials you added earlier.
- Branches to build:
main - Script Path:
Jenkinsfile
-
Save the Configuration:
Click Save to store the pipeline configuration.
Step 8: Run the Jenkins Pipeline
-
Trigger the Pipeline:
- Navigate to your pipeline job (
demo-poc-pipeline). - Click Build Now to start the pipeline run.
- Navigate to your pipeline job (
-
Monitor the Pipeline:
- Click on the build number (e.g.,
#1) to view the build details. - Use the Blue Ocean interface for a more user-friendly visualization by clicking on Open Blue Ocean.
- Click on the build number (e.g.,
-
Verify the Deployment:
- After the pipeline completes successfully, navigate to
http://localhost:8080/helloto ensure the application is running. - If you pushed the Docker image to Docker Hub and deployed to a different environment, verify accordingly.
- After the pipeline completes successfully, navigate to
Step 9: Automate Pipeline Triggers (Optional)
To automate the pipeline to run on every code push:
-
Set Up Webhooks on GitHub:
- Go to your GitHub repository settings.
- Navigate to Webhooks > Add webhook.
- Payload URL:
http://your-jenkins-url/github-webhook/ - Content type:
application/json - Events: Select Just the push event.
- Click Add webhook.
-
Configure Jenkins GitHub Integration:
- Ensure Jenkins is accessible from GitHub (might require configuring firewall or using services like ngrok for local setups).
- Install the GitHub Integration Plugin if not already installed.
-
Enable GitHub Hooks in Jenkins Job:
- In your pipeline job configuration, under Build Triggers, check GitHub hook trigger for GITScm polling.
- Save the configuration.
Now, every time you push changes to the
mainbranch of your GitHub repository, the Jenkins pipeline will automatically trigger, building, testing, and deploying your application.Additional Tips for Enhancing Your CI/CD Pipeline
-
Implement Testing Stages:
- Add stages for unit tests, integration tests, and quality checks using tools like JUnit, Mockito, and SonarQube.
-
Use Environment Variables:
- Manage sensitive information like API keys and passwords using Jenkins credentials and environment variables.
-
Parallel Stages:
- Optimize build times by running independent stages in parallel.
-
Notifications:
- Integrate Slack or email notifications to alert your team about build statuses.
-
Rollback Mechanisms:
- Implement strategies to revert to previous stable versions in case of deployment failures.
Conclusion
By following this tutorial, you've set up a simple yet effective CI/CD pipeline using Spring Boot, Docker, and Jenkins. This pipeline automates the processes of building, testing, containerizing, and deploying your application, ensuring consistent and reliable deployments. As your project grows, you can enhance this pipeline with more advanced features, such as multi-environment deployments, automated testing, and infrastructure provisioning using Infrastructure as Code (IaC) tools like Terraform or Ansible.
Feel free to explore and customize each component to better fit your project's specific needs!