Jenkins Pipeline: Complete Configuration Guide

What is a Jenkins Pipeline?

A Jenkins Pipeline is a suite of plugins and a domain-specific language (DSL) that allows you to model your entire software delivery process as code. Rather than configuring individual build steps through the Jenkins UI with freestyle jobs, a Pipeline defines the complete build, test, and deployment workflow in a text file called a Jenkinsfile, which can be checked into source control alongside your application code.

Pipelines are built on two fundamental concepts:

Stages – Logical divisions in your delivery process (e.g., Build, Test, Deploy). Each stage contains one or more steps.
Steps – Individual tasks executed within a stage, such as running a shell command, invoking a build tool, or calling another Pipeline.

A Pipeline also provides durable execution: it can survive planned and unplanned restarts of the Jenkins controller, pause for human approval, and resume right where it left off. This makes Pipelines far more resilient than traditional freestyle jobs.

Declarative versus Scripted Pipeline

Jenkins supports two flavors of Pipeline syntax:

Declarative Pipeline – A structured, opinionated syntax that wraps the entire pipeline inside a pipeline { } block. It enforces a clear, predictable structure and is the recommended approach for most teams because of its readability, built-in error checking, and ease of integration with Blue Ocean and other visual tools.
Scripted Pipeline – A more flexible, Groovy-based syntax that gives you full programmatic control. It is wrapped in a node { } block and allows imperative logic, loops, and advanced flow control. It is best suited for complex workflows that cannot be expressed easily in declarative form.

Why Jenkins Pipeline Matters

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Adopting Jenkins Pipeline brings several concrete benefits to your development workflow:

Pipeline as Code – The entire CI/CD process lives in a version-controlled Jenkinsfile, enabling code review, audit trails, and the same branching/merging strategies you use for application code.
Reusability – You can define shared library modules containing common logic (e.g., notification routines, deployment patterns) and reuse them across multiple projects, reducing duplication.
Resilience – If Jenkins restarts, the Pipeline resumes automatically from the last completed step, thanks to its durable state stored on the controller.
Parallel Execution – Run independent stages or steps concurrently (e.g., testing on multiple platforms simultaneously), dramatically reducing total build time.
Human Approval Gates – Insert manual approval steps (using input) to enforce sign-offs before deploying to production or sensitive environments.
Visualization – The Blue Ocean plugin and Pipeline Stage View give you a graphical representation of every run, making it easy to diagnose failures at a glance.

How to Use Jenkins Pipeline

1. Creating Your First Jenkinsfile

A Jenkinsfile is the heart of your Pipeline. Place it in the root of your repository. Below is a minimal declarative Pipeline that checks out code, builds, tests, and archives artifacts:

pipeline {
    agent any
    
    tools {
        maven 'maven-3.9'  // Name must match a configured Maven installation in Jenkins
    }
    
    stages {
        stage('Checkout') {
            steps {
                checkout scm  // Checks out from the SCM configured in the job
            }
        }
        stage('Build') {
            steps {
                sh 'mvn clean compile'
            }
        }
        stage('Test') {
            steps {
                sh 'mvn test'
            }
            post {
                always {
                    junit 'target/surefire-reports/**/*.xml'
                }
            }
        }
        stage('Package') {
            steps {
                sh 'mvn package -DskipTests'
                archiveArtifacts artifacts: 'target/*.jar', fingerprint: true
            }
        }
    }
    
    post {
        success {
            echo 'Pipeline succeeded! Sending notification...'
        }
        failure {
            echo 'Pipeline failed! Notifying team...'
        }
    }
}

This example demonstrates the essential building blocks: an agent declaration, tools for build environment setup, multiple stages with steps, and a post section for conditional actions after completion.

2. Choosing and Configuring Agents

The agent directive tells Jenkins where to execute the Pipeline. You have several options:

// Run on any available agent with the specified label
agent { label 'linux-docker' }

// Run on a specific node by name
agent { node { label 'builder-01' } }

// Run inside a Docker container (ephemeral, clean environment)
agent {
    docker {
        image 'openjdk:17-jdk-slim'
        args '-v /tmp:/tmp'
    }
}

// Run on the controller itself (use sparingly for lightweight tasks only)
agent { node { label 'master' } }

// Run entirely on the controller (discouraged for production pipelines)
agent none  // Top-level; then assign per-stage agents with same syntax

Docker agents are particularly powerful because they guarantee a reproducible, isolated environment on every run, eliminating "works on my machine" issues.

3. Defining Environment Variables

Environment variables can be set at the pipeline level, stage level, or dynamically from scripts. They are essential for parameterizing builds and injecting secrets:

pipeline {
    agent any
    
    environment {
        APP_NAME = 'payment-service'
        ARTIFACT_REPO = 'https://nexus.internal.example.com'
        // Credential reference: pulls username/password from Jenkins credential store
        DB_PASSWORD = credentials('prod-db-password-id')
    }
    
    stages {
        stage('Deploy') {
            environment {
                TARGET_ENV = 'staging'
            }
            steps {
                sh '''
                    echo "Deploying ${APP_NAME} to ${TARGET_ENV}"
                    # DB_PASSWORD is automatically masked in logs
                '''
            }
        }
    }
}

Use the credentials() helper to inject secrets from the Jenkins credential store. Their values are automatically redacted in console output, preventing accidental exposure.

4. Working with Parameters

Parameters allow you to prompt users for input when triggering a Pipeline manually. Declare them in a parameters block:

pipeline {
    agent any
    
    parameters {
        string(name: 'BRANCH', defaultValue: 'main', description: 'Branch to build')
        choice(name: 'DEPLOY_ENV', choices: ['dev', 'staging', 'production'], description: 'Target environment')
        booleanParam(name: 'RUN_INTEGRATION_TESTS', defaultValue: true)
        password(name: 'DEPLOY_KEY', description: 'Deployment encryption key')
    }
    
    stages {
        stage('Initialize') {
            steps {
                script {
                    echo "Building branch: ${params.BRANCH}"
                    echo "Target environment: ${params.DEPLOY_ENV}"
                }
            }
        }
    }
}

Parameters are accessed via the params object. They are only presented on the first run or when you click "Build with Parameters" in the UI; subsequent automated triggers (SCM polling, webhooks) use default values.

5. Handling Failures and Post-Build Actions

The post section lets you run steps conditionally based on the completion status of the Pipeline or a specific stage. This is critical for notifications, cleanup, and artifact retention:

pipeline {
    agent any
    
    stages {
        stage('Run Tests') {
            steps {
                sh './run-tests.sh'
            }
            post {
                always {
                    // Always run, regardless of outcome
                    sh 'docker-compose down'
                    cleanWs()  // Delete workspace to save disk space
                }
                success {
                    archiveArtifacts artifacts: 'results/**/*.xml'
                }
                failure {
                    sh 'gather-logs.sh'
                    mail to: 'team@example.com',
                         subject: "Tests failed on ${env.JOB_NAME} #${env.BUILD_NUMBER}",
                         body: "See ${env.BUILD_URL} for details."
                }
                unstable {
                    echo 'Tests passed but coverage threshold not met'
                }
            }
        }
    }
    
    post {
        always {
            echo 'Pipeline complete. Cleanup initiated.'
            deleteDir()  // Alternative to cleanWs for complete directory removal
        }
        success {
            slackSend channel: '#deployments',
                      color: 'good',
                      message: "Release ${env.BUILD_NUMBER} deployed successfully."
        }
    }
}

The available post conditions are: always, success, failure, unstable, aborted, fixed, regression, and changed. Stage-level post blocks inherit the stage's status; top-level post blocks apply to the entire Pipeline.

6. Parallel Execution

To speed up your Pipeline, run independent stages or steps in parallel. The parallel directive achieves this easily:

pipeline {
    agent none
    
    stages {
        stage('Static Analysis and Unit Tests') {
            parallel {
                stage('Lint') {
                    agent { label 'linux' }
                    steps {
                        sh 'npm run lint'
                    }
                }
                stage('Unit Tests - Linux') {
                    agent { label 'linux' }
                    steps {
                        sh 'npm test'
                    }
                }
                stage('Unit Tests - Windows') {
                    agent { label 'windows' }
                    steps {
                        bat 'npm test'  // bat for Windows batch commands
                    }
                }
                stage('Security Scan') {
                    agent { docker { image 'aquasec/trivy' } }
                    steps {
                        sh 'trivy filesystem --severity HIGH,CRITICAL --exit-code 1 .'
                    }
                }
            }
        }
        stage('Build and Package') {
            // This runs only after all parallel stages succeed
            agent { label 'linux' }
            steps {
                sh 'npm run build'
                sh 'docker build -t myapp:${BUILD_NUMBER} .'
            }
        }
    }
}

Each parallel branch can run on a different agent, making this pattern ideal for cross-platform testing. The pipeline progresses to the next stage only after all parallel branches complete successfully.

7. Manual Approval Gates with Input

The input step pauses the Pipeline and waits for a human to approve or reject it. This is essential for production deployments and compliance workflows:

pipeline {
    agent any
    
    stages {
        stage('Deploy to Staging') {
            steps {
                sh './deploy-staging.sh'
            }
        }
        stage('Approval Gate') {
            steps {
                input(
                    message: 'Proceed with production deployment?',
                    ok: 'Deploy to Production',
                    submitter: 'ops-team, lead-developer',
                    parameters: [
                        choice(name: 'ROLLBACK_WINDOW_HOURS',
                               choices: ['1', '2', '4', '24'],
                               description: 'Rollback window in hours')
                    ]
                )
            }
        }
        stage('Deploy to Production') {
            steps {
                sh './deploy-production.sh'
            }
        }
    }
}

The submitter parameter restricts who can approve the input by specifying user IDs or group names. You can also attach parameters to the input, allowing the approver to supply additional configuration at approval time.

8. Using Triggers for Automated Execution

Triggers define when a Pipeline should run automatically. The most common trigger is polling SCM, but webhook-driven triggers (via the Multibranch Pipeline or GitHub/Bitbucket branch source plugins) are preferred for responsiveness:

pipeline {
    agent any
    
    triggers {
        // Poll SCM every 5 minutes (legacy approach; webhooks are better)
        pollSCM 'H/5 * * * *'
        
        // Cron-style scheduling: run daily at 2 AM
        cron '0 2 * * *'
        
        // Trigger when upstream job completes successfully
        upstream upstreamProjects: 'build-libs', threshold: 'SUCCESS'
    }
    
    stages {
        stage('Build') {
            steps {
                echo 'Triggered automatically!'
            }
        }
    }
}

For modern setups, use a Multibranch Pipeline with a GitHub or Bitbucket organization folder. This automatically discovers repositories, creates branches for pull requests, and triggers builds on webhook events without any manual triggers block configuration.

9. Shared Libraries for Code Reuse

When you have common functionality repeated across multiple Pipelines, extract it into a Shared Library. Configure it once in Jenkins Global Settings under "Global Pipeline Libraries," then reference it in your Jenkinsfile:

// Jenkinsfile
@Library('my-shared-lib') _

pipeline {
    agent any
    stages {
        stage('Deploy') {
            steps {
                script {
                    // Call a custom function defined in vars/deployApp.groovy
                    deployApp(
                        environment: 'staging',
                        artifactPath: 'target/app.war'
                    )
                }
            }
        }
    }
}

Inside the shared library repository, you would have a file vars/deployApp.groovy:

// vars/deployApp.groovy
def call(Map config) {
    def env = config.environment
    def artifact = config.artifactPath
    
    echo "Deploying ${artifact} to ${env}"
    
    // Common deployment logic
    sh """
        scp ${artifact} deploy-server-${env}:/opt/apps/
        ssh deploy-server-${env} 'sudo systemctl restart app'
    """
    
    // Return deployment metadata for downstream use
    return [host: "deploy-server-${env}", timestamp: new Date().time]
}

Shared libraries can also contain Groovy classes under src/ for more complex logic and step implementations, giving you full object-oriented capabilities within your Pipeline code.

10. Scripted Pipeline Essentials

While Declarative Pipeline covers 90% of use cases, there are situations where you need the full power of Groovy scripting. Scripted Pipeline gives you fine-grained control over flow, loops, and dynamic stage creation:

node('linux') {
    // Clean workspace and checkout
    deleteDir()
    checkout scm
    
    // Define stages dynamically based on discovered modules
    def modules = sh(script: 'find . -name "pom.xml" -maxdepth 2 | sed "s|./||;s|/pom.xml||"', returnStdout: true).trim().split('\n')
    
    def buildResults = [:]
    
    for (int i = 0; i < modules.size(); i++) {
        def module = modules[i]
        def moduleName = module.replace('/', '-')
        
        buildResults[moduleName] = {
            stage("Build ${moduleName}") {
                dir(module) {
                    sh 'mvn clean install'
                    stash name: moduleName, includes: 'target/**/*.jar'
                }
            }
        }
    }
    
    // Execute all module builds in parallel
    parallel buildResults
    
    stage('Aggregate') {
        for (moduleName in buildResults.keySet()) {
            unstash moduleName
        }
        sh 'mvn verify -DskipTests'
    }
    
    // Conditional deployment logic
    if (env.BRANCH_NAME == 'main') {
        stage('Deploy') {
            input 'Proceed with deployment?'
            sh './deploy.sh production'
        }
    } else {
        stage('Deploy to Preview') {
            sh './deploy.sh preview --branch=${env.BRANCH_NAME}'
        }
    }
}

Scripted Pipeline allows try/catch/finally blocks for granular error handling, dynamic parallel stages via map construction, and complex conditional branching that would be difficult in declarative syntax. Use it when you genuinely need programmatic flow control, but default to declarative for simpler, more maintainable Pipelines.

Best Practices

Following these practices will keep your Pipelines maintainable, secure, and performant over time:

Prefer Declarative Syntax – Unless you need dynamic stage creation or complex Groovy logic, stick with declarative. It is easier to read, provides better IDE support, and integrates seamlessly with Blue Ocean.
Keep Jenkinsfiles Short and Delegate Logic – Move heavy logic into shell scripts, Makefiles, or shared libraries. A Jenkinsfile should be a high-level orchestration document, not a 500-line script.
Use Docker Agents for Reproducibility – Docker-based agents ensure every run starts from a clean, identical environment. This eliminates drift caused by accumulated state on long-lived VM agents.
Never Hardcode Secrets – Use the credentials() helper or inject secrets via environment variables backed by the credential store. Never commit tokens, passwords, or keys directly into a Jenkinsfile.
Leverage Parallel Stages – Identify stages that do not depend on each other and run them concurrently to shrink total Pipeline duration.
Clean Workspaces After Runs – Use cleanWs() or deleteDir() in an always post block to prevent disk exhaustion on agents, especially for projects that generate large intermediate artifacts.
Archive Only Necessary Artifacts – Be selective with archiveArtifacts. Archiving everything bloats Jenkins storage and slows down downstream jobs that fetch artifacts.
Use Timeouts to Prevent Hanging Pipelines – Wrap long-running steps in timeout blocks to avoid Pipelines that hang indefinitely due to network issues or stuck processes:

stage('Integration Tests') {
    steps {
        timeout(time: 30, unit: 'MINUTES') {
            sh './run-long-tests.sh'
        }
    }
}

Version Your Shared Libraries – Reference shared libraries by version (e.g., @Library('my-lib@1.2.3')) in production Pipelines to avoid unexpected breakage when library code changes.
Test Your Jenkinsfile Changes on a Branch – Before merging a Jenkinsfile modification, test it on a dedicated branch or in a Pipeline replay session to ensure it behaves as expected without breaking the main build.
Keep Global Configuration Minimal – Rely on tools definitions and agent labels that match your actual infrastructure, but avoid baking environment-specific assumptions into Pipelines. Use parameters or environment variables to adapt behavior per environment.

Conclusion

Jenkins Pipeline transforms your CI/CD process from a fragile, UI-configured sequence of jobs into a resilient, version-controlled, code-driven delivery pipeline. By mastering the Declarative Pipeline syntax, understanding agent configuration, leveraging parallel execution, incorporating manual approval gates, and extracting reusable logic into shared libraries, you can build workflows that scale across teams and projects with minimal duplication. The investment in learning Pipeline as Code pays dividends through faster feedback loops, reproducible builds, and a delivery process that evolves alongside your application code. Start by converting one freestyle job into a simple Jenkinsfile, then progressively adopt the advanced patterns covered here to unlock the full potential of continuous delivery with Jenkins.