Jenkins as Code - Enterprise CI/CD Automation

Complete Jenkins automation framework with Infrastructure as Code, Configuration as Code, and dynamic job generation using JobGenie

Updated Jan 15, 2024

15 min read

Advanced

tools

Jenkins as Code - Enterprise CI/CD Automation

Jenkins as Code is a comprehensive automation framework that transforms Jenkins from a manually configured tool into a fully automated, code-driven CI/CD platform. This solution provides Infrastructure as Code (IaC), Configuration as Code (CaC), and dynamic job generation capabilities, enabling DevOps as a Service for development teams.

Quick Start: New to Jenkins as Code? Start with the [Architecture](architecture/) overview, then follow the [Setup Guide](setup/) for installation. For job creation, see the [JobGenie Guide](jobgenie/).

📋 Table of Contents

Getting Started

• Overview - Framework overview and key metrics
• Architecture - System architecture and design
• Core Components - Infrastructure, Configuration, and JobGenie

Deep Dives

• JobGenie Deep Dive - Complete JobGenie guide
• DevOps as a Service - Self-service CI/CD platform
• Implementation Guide - Step-by-step setup

Advanced Topics

• Advanced Features - Advanced capabilities
• Best Practices - Recommended practices
• Troubleshooting - Common issues and solutions
• Real-World Examples - Case studies

Additional Resources

• Performance Optimization - Performance tuning
• Monitoring & Observability - Monitoring setup
• Security Best Practices - Security guidelines
• Disaster Recovery - Backup and recovery
• Comparison with Alternatives - Framework comparison
• Future Enhancements - Roadmap

Documentation Pages

• Architecture Documentation - Detailed architecture guide
• JobGenie Complete Guide - JobGenie usage and examples
• DevOps as a Service - Self-service platform guide
• Best Practices - Best practices and guidelines
• Setup Guide - Setup and configuration
• Quick Reference - Quick reference guide

Overview

Traditional Jenkins management involves manual configuration, inconsistent environments, and operational overhead. This framework eliminates these challenges by providing:

• Infrastructure as Code: Complete Jenkins setup via Ansible playbooks
• Configuration as Code: All Jenkins settings managed through YAML files
• Jobs as Code: Dynamic job generation using JobDSL and JobGenie engine
• Self-Service Onboarding: Teams can provision their own CI/CD pipelines via Git PRs

Key Metrics

• 90% reduction in time to onboard new projects (from days to hours)
• 80% reduction in DevOps support tickets
• 100% consistency across environments
• Zero manual configuration required
• 95% faster disaster recovery

Problem Statement

Organizations face several challenges with traditional Jenkins management:

1. Manual Configuration Bottleneck: Every new project requires DevOps intervention
2. Inconsistent Environments: Configuration drift between dev/staging/prod
3. Scalability Issues: Cannot scale with growing number of projects
4. Security & Compliance Risks: Difficult to audit and maintain security policies
5. Operational Overhead: High maintenance burden and difficult troubleshooting

Solution Overview

This framework provides a complete solution through:

1. Automated Infrastructure: Ansible playbooks for complete Jenkins setup
2. Version-Controlled Configuration: All settings in Git with full audit trail
3. Dynamic Job Generation: JobGenie engine for automated job creation
4. Self-Service Platform: Teams onboard themselves via Git PRs

Architecture

The framework is built on four distinct layers, each serving a specific purpose:

┌─────────────────────────────────────────────────────────────┐
│              Infrastructure Layer (Ansible)                 │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐     │
│  │   Ansible    │  │  Monitoring  │  │   Jenkins    │     │
│  │   Playbook   │  │   Stack      │  │   Stack      │     │
│  └──────────────┘  └──────────────┘  └──────────────┘     │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│              Configuration Layer (CaC)                      │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐     │
│  │ jenkins.yaml │  │  seed-jobs   │  │  init.groovy │     │
│  │   (CaC)      │  │  (JobDSL)    │  │  (Plugins)   │     │
│  └──────────────┘  └──────────────┘  └──────────────┘     │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│              Job Generation Layer (JobGenie)                │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐     │
│  │  JobGenie    │  │  Shared      │  │   Pipeline   │     │
│  │  Scripts     │  │  Libraries   │  │   Templates  │     │
│  └──────────────┘  └──────────────┘  └──────────────┘     │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│              Execution Layer (Jenkins Jobs)                  │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐     │
│  │   Build      │  │   Deploy     │  │   Manage     │     │
│  │   Jobs       │  │   Jobs       │  │   Jobs       │     │
│  └──────────────┘  └──────────────┘  └──────────────┘     │
└─────────────────────────────────────────────────────────────┘

Data Flow

Developer → Git Repository → Seed Job → JobGenie → JobDSL → Jenkins Jobs
    │            │              │          │          │          │
    │            │              │          │          │          └─> Execute Pipeline
    │            │              │          │          └─> Generate Job Definition
    │            │              │          └─> Parse Configuration
    │            │              └─> Read JobGenie.groovy
    │            └─> Commit JobGenie Config
    └─> Edit Configuration

Core Components

1. Infrastructure Automation (Ansible)

Complete Jenkins infrastructure provisioning and configuration management using Ansible playbooks.

Features

• Automated Installation: Jenkins installation with specific version control
• Plugin Management: Automated plugin installation with version pinning
• System Configuration: Users, permissions, security settings
• Monitoring Integration: Telegraf, Filebeat, ELK stack setup
• Idempotent Deployments: Safe to run multiple times without side effects

Ansible Playbook Structure

# packer.yml - Main playbook
---
- name: Deploy Jenkins Infrastructure
  hosts: jenkins_servers
  become: yes
  roles:
    - role: monitoring
      tags: [monitoring, deploy]
    - role: users
      tags: [users, deploy]
  tasks:
    - name: Install Jenkins
      include_role:
        name: monitoring
        tasks_from: install/jenkins.yml
      tags: deploy

Configuration Variables

# group_vars/jenkins.yml
jenkins_version: 2.528.2
jenkins_home: "/var/lib/jenkins"
jenkins_project_name: "amazon"

# Plugin Configuration
jenkins_plugins:
  - docker-slaves
  - docker-workflow
  - ansicolor
  - google-login
  - aws-java-sdk-secretsmanager
  - favorite
  - job-dsl
  - configuration-as-code

# System Configuration
jenkins_num_executors: 2
jenkins_quiet_period: 5
jenkins_scm_checkout_retry_count: 0

Monitoring Stack Integration

# Monitoring configuration
active_stacks:
  - java
  - jenkins

active_agents: []
agent_enabled: 'true'

# Monitoring tools
monitoring_tools:
  telegraf:
    enabled: true
    config_path: /etc/telegraf/telegraf.conf
  filebeat:
    enabled: true
    log_path: /var/log/jenkins

2. Configuration as Code (CaC)

All Jenkins settings managed through version-controlled YAML files using Jenkins Configuration as Code (JCasC) plugin.

Configuration Structure

# jenkins.yaml - Main configuration file
jenkins:
  numExecutors: 2
  mode: NORMAL
  projectNamingStrategy:
    roleBased:
      forceExistingJobs: false
  quietPeriod: 5
  scmCheckoutRetryCount: 0

  # Authorization Strategy
  authorizationStrategy:
    roleBased:
      permissionTemplates:
        - name: "build"
          permissions:
            - "Job/Cancel"
            - "Job/Build"
            - "Job/Read"
            - "View/Read"
        - name: "write"
          permissions:
            - "Job/Cancel"
            - "Job/Build"
            - "Job/Read"
            - "Job/Configure"
            - "Job/Create"
      roles:
        global:
          - name: "admin"
            pattern: ".*"
            permissions: ["Overall/Administer"]
            entries:
              - user: "admin"
              - user: "hari_25585"

Security Configuration

# Security realm configuration
jenkins:
  securityRealm:
    local:
      allowsSignup: false
      enableCaptcha: false
      users:
        - id: "admin"
          name: "admin"
          password: "${ADMIN_PASSWORD}"
          properties:
            - "apiToken"
            - "myView"
            - "timezone"
            - "mailer"
            - "slack"

Global Libraries Configuration

# Global shared libraries
jenkins:
  unclassified:
    globalLibraries:
      libraries:
        - name: "sharedPipelineUtils"
          defaultVersion: "master"
          retriever:
            modernSCM:
              libraryPath: "sharedlibs"
              scm:
                gitSource:
                  credentialsId: "jenkins_repo_key"
                  remote: "git@github.com:org/repo.git"
                  traits:
                    - "gitBranchDiscovery"

3. JobGenie - Dynamic Job Generation

JobGenie is the heart of the framework - a powerful job generation engine that transforms simple Groovy map definitions into fully functional Jenkins pipelines.

Core Concept

JobGenie uses a simple map-based configuration approach:

// Basic JobGenie structure
def jobGenie = [
    "team-name": [
        [ /* Job 1 Configuration */ ],
        [ /* Job 2 Configuration */ ],
        [ /* Job 3 Configuration */ ]
    ]
]

Configuration Hierarchy

Variables are merged in this order (later overrides earlier):

1. defaultConfigs: Global defaults for all jobs
2. projectConfigs: Project-specific overrides
3. jobGenie entries: Individual job configurations

Simple Application Job

def defaultConfigs = [
    EKS_CLUSTER_NAME: "${PROJECT_NAME}-eks-${PROJECT_ENV}",
    CICD_TEMPLATE_BASE: 'templates/cicd',
    CICD_BASE: "${HOME_DIR}/jenkinsfiles",
    IMAGE_ECR: "${awsAccountId}.dkr.ecr.${region}.amazonaws.com",
    GROUP: "v2",
    ENV: "stage"
]

def jobGenie = [
    "mcloud": [
        [
            APP_REPO: 'user-service',
            APP_BRANCH: 'develop',
            appNames: ['user-api'],
            DOCKER_BUILD_ARGS: 'ENV,TECHTEAM'
        ]
    ]
]

Generated Output:

• Complete Jenkins pipeline job
• Build and deployment stages
• Parameterized build options (APP_BRANCH, OPERATION, EXTENT, STEPS)
• Environment variable configuration
• Folder structure: amazon/mcloud/nonprod/deploy/v2/stage/user-api/

Multi-Service Deployment

def jobGenie = [
    "mcloud": [
        [
            APP_REPO: 'microservices-platform',
            APP_BRANCH: 'main',
            appNames: [
                'api-gateway',
                'auth-service',
                'payment-service',
                'order-service'
            ],
            DOCKER_BUILD_ARGS: 'ENV,TECHTEAM,SERVICE',
            DOCKERFILE_PATH: 'services/Dockerfile'
        ]
    ]
]

This single configuration generates 4 separate pipeline jobs, one for each service.

Advanced Configuration with Build + Deploy Pattern

def jobGenie = [
    "mcloud": [
        [
            APP_REPO: 'complex-application',
            APP_BRANCH: 'release/v2.0',
            appNames: ['api', 'worker', 'scheduler'],
            BUILD: [
                NAME: 'complex-app'
            ],
            DOCKER_BUILD_ARGS: 'ENV,VERSION',
            MAVEN_VERSION: 'maven3.9.7',
            JDK_VERSION: 'jdk21'
        ]
    ]
]

Generated Jobs:

• stage-complex-app-build: Build-only job
• stage-api-deploy: Deploy job with ECR image selection
• stage-worker-deploy: Deploy job with ECR image selection
• stage-scheduler-deploy: Deploy job with ECR image selection

Custom Job Templates

def jobGenie = [
    "mcloud": [
        [
            SERVICE: 'data-pipeline',
            CICD_TEMPLATE_NAME: 'data-processing',
            DEFAULT_JENKINSFILE: 'templates/data-pipeline/Jenkinsfile',
            JOB_PARAMS: [
                [
                    name: 'DATA_SOURCE',
                    string: 's3://bucket/data',
                    description: 'Data source S3 path'
                ],
                [
                    name: 'PROCESSING_MODE',
                    choices: ['batch', 'stream'],
                    description: 'Processing mode: batch or stream'
                ],
                [
                    name: 'DRY_RUN',
                    bool: false,
                    description: 'Enable dry run mode'
                ]
            ]
        ]
    ]
]

Dynamic Parameter Generation

JobGenie supports dynamic parameters using Groovy scripts:

def jobGenie = [
    "mcloud": [
        [
            APP_REPO: 'my-app',
            APP_BRANCH: 'main',
            appNames: ['my-service'],
            BUILD: [
                NAME: 'my-app'
            ],
            // Dynamic ECR image tag selection
            JOB_PARAMS: [
                [
                    name: 'ImageTag',
                    choiceType: 'SINGLE_SELECT',
                    description: 'Select Docker image tag from ECR',
                    arChoiceGroovy: """
                        def command = "aws ecr describe-images --repository-name ${ECR_REPO} --query 'sort_by(imageDetails[?imageTags != null], &imagePushedAt)[-5:].imageTags[]' --output text"
                        def aws_command_output = ["/bin/sh", "-c", "\${command}"].execute()
                        def files = aws_command_output.text.tokenize().reverse()
                        return files
                    """
                ]
            ]
        ]
    ]
]

Environment-Specific Configuration

def defaultConfigs = [
    // Base configuration
    EKS_CLUSTER_NAME: "${PROJECT_NAME}-eks-${PROJECT_ENV}",
    CICD_BASE: "${HOME_DIR}/jenkinsfiles",
    IMAGE_ECR: "${awsAccountId}.dkr.ecr.${region}.amazonaws.com"
]

def projectConfigs = [
    "mcloud": [
        GROUP: "v3",
        ENV: "stage",
        ARGOCD_ENDPOINT: 'argocd.stage.example.com'
    ],
    "ncloud": [
        GROUP: "v2",
        ENV: "prod",
        ARGOCD_ENDPOINT: 'argocd.prod.example.com'
    ]
]

4. Shared Libraries

Reusable pipeline utilities and functions for common operations.

Common Utilities

// vars/pipelineUtils.groovy

/**
 * Generate Docker build arguments from environment variables
 */
def generateDockerArgs(Map opts = [:]) {
    def buildArgs = (opts.buildArgs ?: [])
        .collect { key -> 
            env[key] ? "--build-arg ${key}=${env[key]}" : null 
        }
        .findAll { it != null }

    def sshArgs = (opts.sshArgs ?: [])
        .findAll { it && it.contains(':') }
        .collect { entry ->
            def (key, val) = entry.split(':', 2)
            "--ssh ${key}=${val}"
        }

    return (buildArgs + sshArgs).join(' ')
}

/**
 * Update canary deployment steps in YAML configuration
 */
def updateCanarySteps(int steps, String file) {
    if (steps < 1) {
        throw new IllegalArgumentException("Number of steps must be greater than 0")
    }

    def cfg = readYaml(file: file)
    
    if (!cfg?.rollout?.strategy?.canary) {
        throw new Exception("YAML file must contain rollout.strategy.canary structure")
    }

    cfg.rollout.strategy.canary.steps = steps == 1 ? 
        [[setWeight: 100]] : 
        (1..<steps).collectMany { i ->
            int weight = (100 / steps) * i
            weight < 100 ? [[setWeight: weight], [pause: [:]]] : []
        }

    writeYaml(file: file, data: cfg, overwrite: true)
    echo "Successfully updated ${steps} canary steps in ${file}"
}

Notification Utilities

// vars/sendNotifications.groovy

def call(String buildStatus = 'STARTED') {
    buildStatus = buildStatus ?: 'SUCCESS'

    def color = [
        'STARTED': '#3498db',
        'SUCCESS': '#3eb991',
        'FAILURE': '#e74c3c',
        'ABORTED': '#95a5a6',
        'ROLLBACK': '#e9a820',
        'INPROGRESS': '#56d19f'
    ][buildStatus.toUpperCase()] ?: '#95a5a6'

    def notificationMessage = [
        attachments: [
            [
                color: color,
                title: "Deployment ${buildStatus.toLowerCase().capitalize()}",
                title_link: env.BUILD_URL,
                fields: [
                    [
                        title: 'Job Details',
                        value: """
                            • JobName: ${env.JOB_BASE_NAME}
                            • Build: `#${env.BUILD_NUMBER}`
                            • Service: ${env.SERVICE ?: 'N/A'}
                            • Environment: ${env.ENV ?: 'N/A'}
                        """.stripIndent(),
                        short: false
                    ]
                ],
                footer: "Jenkins Build #${env.BUILD_NUMBER}",
                ts: System.currentTimeMillis() / 1000
            ]
        ]
    ]

    slackSend(
        color: color,
        message: "Deployment ${buildStatus.toLowerCase().capitalize()} - ${env.JOB_NAME} #${env.BUILD_NUMBER}",
        attachments: notificationMessage.attachments
    )
}

JobGenie Deep Dive

Understanding JobGenie Architecture

JobGenie operates on a simple principle: Configuration as Data. Instead of writing complex Groovy scripts, you define jobs using structured data (maps and lists).

JobGenie Processing Flow

1. Seed Job Execution
   ↓
2. Read JobGenie.groovy File
   ↓
3. Parse Configuration Maps
   ↓
4. Merge Configurations (defaultConfigs + projectConfigs + jobConfig)
   ↓
5. Generate Job Definitions via JobDSL
   ↓
6. Create Jenkins Jobs
   ↓
7. Jobs Available for Use

Configuration Merging Logic

// Simplified merging logic
def pConfig = defaultConfigs + (projectConfigs[techteam] ?: [:]) + jobConfig

// Final configuration includes:
pConfig += [
    TECHTEAM: techteam,
    DEPLOY_HOME: "${ORGANIZATION}/${techteam}/${PROJECT_ENV}/deploy",
    JENKINS_CICDDIR: "${CICD_BASE}",
    JOB_SUFFIX_DIR: "${GROUP}/${ENV}"
]

JobGenie Variable Reference

Required Variables

Variable	Type	Description	Example
APP_REPO	String	Git repository name	'user-service'
APP_BRANCH	String	Git branch name	'develop'
appNames	List	List of service names	['user-api']

Optional Variables

Variable	Type	Description	Example
DOCKER_BUILD_ARGS	String	Docker build arguments	'ENV,TECHTEAM'
DOCKERFILE_PATH	String	Path to Dockerfile	'services/Dockerfile'
DOCKER_IMAGE_ARCH	String	Image architecture	'linux/arm64'
SSH_KEYS	String	SSH key path	'default:/path/to/key'
MAVEN_VERSION	String	Maven version	'maven3.9.7'
JDK_VERSION	String	JDK version	'jdk21'
NODE_VERSION	String	Node.js version	'node18'
ARGOCD_ENDPOINT	String	ArgoCD endpoint	'argocd.example.com'
HELM_CHART	String	Helm chart name	'application-chart'
HELM_CHART_VERSION	String	Chart version	'1.0.0'

JobGenie Examples

Example 1: Standard Microservice

def jobGenie = [
    "mcloud": [
        [
            APP_REPO: 'payment-service',
            APP_BRANCH: 'main',
            appNames: ['payment-api'],
            DOCKER_BUILD_ARGS: 'ENV',
            SSH_KEYS: 'default:/var/lib/jenkins/.ssh/id_rsa'
        ]
    ]
]

Generated Job Path:

amazon/mcloud/nonprod/deploy/v2/stage/payment-api/stage-payment-api-pipeline

Job Parameters:

• APP_BRANCH: Application git branch

• OPERATION: build

  promote

  abort

• EXTENT: none

  full

• STEPS: Number of canary steps (1-10)

Example 2: Multi-Module Application

def jobGenie = [
    "mcloud": [
        [
            APP_REPO: 'monorepo-platform',
            APP_BRANCH: 'develop',
            appNames: [
                'api-gateway',
                'user-service',
                'order-service',
                'payment-service',
                'notification-service'
            ],
            DOCKER_BUILD_ARGS: 'ENV,TECHTEAM,SERVICE',
            DOCKERFILE_PATH: 'services/${SERVICE}/Dockerfile',
            MAVEN_VERSION: 'maven3.9.7',
            JDK_VERSION: 'jdk21'
        ]
    ]
]

Generated: 5 separate pipeline jobs, one per service.

Example 3: Frontend Application

def jobGenie = [
    "mcloud": [
        [
            SERVICE: 'customer-portal',
            CICD_TEMPLATE_NAME: 'frontend-template',
            DEFAULT_JENKINSFILE: 'templates/frontend/Jenkinsfile',
            JOB_PARAMS: [
                [
                    name: 'NODE_VERSION',
                    choices: ['node16', 'node18', 'node20'],
                    description: 'Node.js version'
                ],
                [
                    name: 'BUILD_ENV',
                    choices: ['development', 'staging', 'production'],
                    description: 'Build environment'
                ],
                [
                    name: 'DEPLOY_TARGET',
                    string: 's3://bucket/customer-portal',
                    description: 'S3 deployment target'
                ]
            ]
        ]
    ]
]

Example 4: Data Processing Pipeline

def jobGenie = [
    "mcloud": [
        [
            SERVICE: 'etl-pipeline',
            CICD_TEMPLATE_NAME: 'data-pipeline',
            DEFAULT_JENKINSFILE: 'templates/data-pipeline/Jenkinsfile',
            JOB_PARAMS: [
                [
                    name: 'DATA_SOURCE',
                    string: 's3://data-lake/raw/',
                    description: 'Source data path'
                ],
                [
                    name: 'DATA_TARGET',
                    string: 's3://data-lake/processed/',
                    description: 'Target data path'
                ],
                [
                    name: 'PROCESSING_MODE',
                    choices: ['batch', 'stream', 'hybrid'],
                    description: 'Data processing mode'
                ],
                [
                    name: 'SPARK_CONFIG',
                    string: 'spark-defaults.conf',
                    description: 'Spark configuration file'
                ]
            ]
        ]
    ]
]

DevOps as a Service

Self-Service Onboarding Workflow

The framework enables true DevOps as a Service, where development teams can provision their own CI/CD pipelines without DevOps intervention.

Step-by-Step Onboarding Process

Step 1: Developer Creates Configuration

// Edit: amazon/mcloud/nonprod/pipelines/jobs/JobGenie.groovy
def jobGenie = [
    "mcloud": [
        [
            APP_REPO: 'new-service',
            APP_BRANCH: 'develop',
            appNames: ['new-service-api'],
            DOCKER_BUILD_ARGS: 'ENV'
        ]
    ]
]

Step 2: Git Workflow

# Create feature branch
git checkout -b feature/onboard-new-service

# Edit JobGenie.groovy
vim amazon/mcloud/nonprod/pipelines/jobs/JobGenie.groovy

# Commit changes
git add amazon/mcloud/nonprod/pipelines/jobs/JobGenie.groovy
git commit -m "Onboard new-service-api to mcloud nonprod"

# Push and create PR
git push origin feature/onboard-new-service

Step 3: Automated Job Generation

1. PR is merged to main branch
2. Seed job (0-mCloud-DevOps) runs automatically
3. JobGenie reads configuration
4. Jobs generated via JobDSL
5. Jobs appear in Jenkins UI

Step 4: Team Uses Jobs

1. Navigate to: amazon/mcloud/nonprod/deploy/v2/stage/new-service-api/
2. Run stage-new-service-api-pipeline
3. Set parameters and execute
4. Monitor build and deployment

Time to Value Comparison

Metric	Traditional	Jenkins as Code
Initial Request	Submit ticket	Create Git PR
DevOps Review	1-2 days	Optional (self-service)
Job Creation	2-3 days	Automated (< 1 hour)
Testing	1 day	Immediate
Total Time	4-6 days	< 1 hour

Benefits of Self-Service Model

1. Reduced Wait Times: No dependency on DevOps team availability
2. Faster Iteration: Teams can experiment and iterate quickly
3. Consistency: All jobs follow same structure and patterns
4. Version Control: All changes tracked in Git
5. Audit Trail: Complete history of all job configurations

Implementation Guide

Prerequisites

System Requirements

• OS: Amazon Linux 2023 / Amazon Linux 2 (ARM64 or x86_64)
• CPU: Minimum 2 cores, recommended 4+ cores
• RAM: Minimum 4GB, recommended 8GB+
• Disk: Minimum 50GB, recommended 100GB+

Software Requirements

• Ansible: 2.9 or higher
• Python: 3.8 or higher
• Git: Latest version
• Docker: For containerized builds
• AWS CLI: For ECR/ECS integration

Installation Steps

Step 1: Clone Repositories

# Clone infrastructure repository
git clone <infra-repo-url>
cd mCloud-infra/ansible

# Clone Jenkins configuration repository
git clone <jenkins-repo-url>
cd mCloud-Jenkins

Step 2: Configure Ansible Variables

# Edit group variables
vim group_vars/packer_al2023_aarch64_devops_jenkins.yml

Essential Configuration:

# Jenkins Version
jenkins_version: 2.528.2

# Jenkins Home Directory
jenkins_home: "/var/lib/jenkins"

# Project Name
jenkins_project_name: amazon

# Jenkins Plugins
jenkins_plugins:
  - docker-slaves
  - docker-workflow
  - ansicolor
  - google-login
  - aws-java-sdk-secretsmanager
  - favorite
  - job-dsl
  - configuration-as-code

# Jenkins URL
jenkins_location:
  url: "https://jenkins.example.com/"
  adminAddress: "jenkins-admin@example.com"

# Security Configuration
jenkins_securityRealm:
  local:
    allowsSignup: false
    users:
      - id: "admin"
        name: "admin"
        password: "${ADMIN_PASSWORD}"

Step 3: Configure User Access

# Read Users
overall_read_users:
  amazon:
    mcloud:
      - user: "dev"
    qa:
      - user: "qa-user"

# Admin Users
overall_admin_users:
  devops_managers:
    - user: "hari_25585"
    - user: "admin"

Step 4: Run Ansible Playbook

# Test connectivity
ansible-playbook packer.yml \
  -e "target_host=packer_al2023_aarch64_devops_jenkins" \
  --check

# Run full deployment
ansible-playbook packer.yml \
  -e "target_host=packer_al2023_aarch64_devops_jenkins" \
  -t deploy,monitoring

Step 5: Verify Jenkins Installation

1. Access Jenkins: https://jenkins.example.com/
2. Login with admin credentials
3. Verify Configuration as Code is loaded
4. Check plugins are installed
5. Verify seed jobs exist

Step 6: Configure Credentials

1. Navigate to: Manage Jenkins → Credentials
2. Add SSH key credential:
   • Kind: SSH Username with private key
   • ID: jenkins_repo_key
   • Username: git
   • Private Key: Upload SSH key

Step 7: Create First JobGenie Configuration

# Edit JobGenie file
vim amazon/mcloud/nonprod/pipelines/jobs/JobGenie.groovy

def jobGenie = [
    "mcloud": [
        [
            APP_REPO: 'test-service',
            APP_BRANCH: 'master',
            appNames: ['test-api']
        ]
    ]
]

Step 8: Run Seed Job

1. Navigate to: amazon/mcloud/nonprod/0-mCloud-DevOps
2. Click Build with Parameters
3. Set GitBranch to your branch
4. Click Build
5. Monitor job execution
6. Verify jobs are created

Advanced Features

Custom Pipeline Templates

Create reusable pipeline templates for specific use cases:

// Custom template configuration
def jobGenie = [
    "mcloud": [
        [
            SERVICE: 'special-workflow',
            CICD_TEMPLATE_NAME: 'custom-workflow',
            DEFAULT_JENKINSFILE: 'templates/custom-workflow/Jenkinsfile',
            JOB_PARAMS: [
                [ name: 'WORKFLOW_TYPE', choices: ['type1', 'type2'], description: 'Workflow type' ],
                [ name: 'CUSTOM_PARAM', string: 'default', description: 'Custom parameter' ]
            ]
        ]
    ]
]

Multi-Environment Support

Configure different settings per environment:

# Ansible variables
jenkins_onboarding:
  amazon:
    mcloud:
      jobs:
        - NAME: "0-mCloud-DevOps"
          VARS:
            CONFIG_REPO: "git@github.com:org/repo.git"
            JENKINS_GIT_KEY: 'jenkins_repo_key'
      env: ["nonprod", "prod"]

Integration with External Systems

ArgoCD Integration

def jobGenie = [
    "mcloud": [
        [
            APP_REPO: 'k8s-app',
            APP_BRANCH: 'main',
            appNames: ['k8s-service'],
            ARGOCD_ENDPOINT: 'argocd.example.com',
            ARGOCD_PROJECT: 'apps',
            HELM_CHART: 'application-chart',
            HELM_CHART_VERSION: '1.0.0'
        ]
    ]
]

ECR Integration

def defaultConfigs = [
    IMAGE_ECR: "${awsAccountId}.dkr.ecr.${region}.amazonaws.com",
    ECR_REGION: 'ap-south-1'
]

Feature Flags

Enable beta/testing features:

def jobGenie = [
    "mcloud": [
        [
            APP_REPO: 'new-service',
            APP_BRANCH: 'feature/new-deployment',
            appNames: ['new-service-api'],
            BETA: true  // Uses template-based Jenkinsfile
        ]
    ]
]

Best Practices

Configuration Management

1. Use Environment-Specific Variables

✅ DO:

jenkins_location:
  url: "https://jenkins-{{ environment }}.example.com/"

❌ DON’T:

jenkins_location:
  url: "https://jenkins-prod.example.com/"  # Hardcoded

2. Organize Variables Logically

✅ DO:

jenkins_onboarding:
  amazon:
    mcloud:
      jobs: [ /* ... */ ]
      env: ["nonprod", "prod"]

❌ DON’T:

jenkins_mcloud_jobs: [ /* ... */ ]
jenkins_mcloud_env: ["nonprod", "prod"]
jenkins_amazon_mcloud: [ /* ... */ ]  # Scattered

3. Document Custom Variables

✅ DO:

// Custom variable for feature flag
// Purpose: Enable beta features for testing
// Usage: Set to true to use template-based Jenkinsfiles
BETA: false

Job Creation

1. Follow Naming Conventions

✅ DO:

appNames: ['user-service', 'payment-service', 'order-service']

❌ DON’T:

appNames: ['UserService', 'payment_service', 'orderService']  // Inconsistent

2. Use Appropriate Job Types

✅ DO:

// Standard application job
[
    APP_REPO: 'my-app',
    APP_BRANCH: 'master',
    appNames: ['my-service']
]

❌ DON’T:

// Using custom job for standard application
[
    SERVICE: 'my-service',
    CICD_TEMPLATE_NAME: 'default',  // Should use standard job
    JOB_PARAMS: []
]

3. Group Related Jobs

✅ DO:

def jobGenie = [
    "mcloud": [
        // All mcloud jobs together
        [ APP_REPO: 'app1', appNames: ['service1'] ],
        [ APP_REPO: 'app2', appNames: ['service2'] ]
    ],
    "ncloud": [
        // All ncloud jobs together
        [ APP_REPO: 'app3', appNames: ['service3'] ]
    ]
]

Security

1. Role-Based Access Control

✅ DO:

# Granular permissions
overall_read_users:
  amazon:
    mcloud:
      - user: "dev"

❌ DON’T:

# Overly broad permissions
overall_read_users:
  amazon:
    mcloud:
      - user: "*"  # Everyone has access

2. Store Secrets Securely

✅ DO:

• Use Jenkins Credentials Plugin
• Use AWS Secrets Manager
• Use Ansible Vault for sensitive variables

❌ DON’T:

• Hardcode passwords in files
• Commit secrets to Git
• Share credentials via email

3. Regular Security Audits

✅ DO:

• Review access permissions quarterly
• Audit credential usage
• Check for unused credentials
• Review audit logs

Troubleshooting

Common Issues and Solutions

Issue 1: Jobs Not Generated

Symptoms:

• Seed job runs successfully
• No jobs appear in Jenkins UI

Solutions:

1. Check createjob Flag

   def globalConfigs = [ createjob: true ]  // Must be true
   

2. Verify JobGenie Map Structure

   // Ensure proper structure
   def jobGenie = [
       "mcloud": [
           [ APP_REPO: '...', appNames: [...] ]
       ]
   ]
   

3. Review Seed Job Logs
   • Navigate to seed job console output
   • Look for parsing errors
   • Check for variable resolution issues
4. Verify File Path
   • Ensure JobGenie.groovy is in correct location
   • Check file permissions
   • Verify Git branch is correct

Issue 2: Configuration Not Applied

Symptoms:

• Ansible playbook runs successfully
• Jenkins configuration doesn’t reflect changes

Solutions:

1. Reload Configuration as Code
   • Navigate to: amazon/manage-infra/Reload-ConfigAsCode
   • Run the job
   • Verify configuration reloaded
2. Check Ansible Variable Syntax

   # Verify YAML syntax
   jenkins_version: 2.528.2  # Correct
   jenkins_version: "2.528.2"  # Also correct
   jenkins_version: 2.528.2.  # Wrong - trailing dot
   

3. Verify Template Rendering

   # Test template rendering
   ansible-playbook packer.yml \
     -e "target_host=..." \
     --check \
     -v
   

4. Check Jenkins Logs

   # View Jenkins logs
   tail -f /var/log/jenkins/jenkins.log
      
   # Check for CaC errors
   grep -i "configuration as code" /var/log/jenkins/jenkins.log
   

Issue 3: Permission Denied

Symptoms:

• Users cannot access jobs
• Build fails with permission errors

Solutions:

1. Verify User in Configuration

   overall_read_users:
     amazon:
       mcloud:
         - user: "username"  # Verify username is correct
   

2. Check Role Pattern Matching

   # Pattern must match job path
   roles:
     items:
       - name: "amazon-mcloud-dev"
         pattern: "amazon/mcloud/.*/nonprod/deploy/.*"  # Verify pattern
         templateName: "build"
   

3. Test Permissions
   • Login as test user
   • Try to access job
   • Check Jenkins audit log
4. Review RBAC Settings
   • Navigate to: Manage Jenkins → Configure Global Security
   • Verify authorization strategy is “Role-Based”
   • Check role assignments

Issue 4: Jobs Generated in Wrong Location

Symptoms:

• Jobs created but in unexpected folder
• Job path doesn’t match expected structure

Solutions:

1. Check DEPLOY_HOME Variable

   // Verify DEPLOY_HOME calculation
   DEPLOY_HOME: "${ORGANIZATION}/${techteam}/${PROJECT_ENV}/deploy"
   

2. Verify JOB_SUFFIX_MKDIR

   // Check suffix calculation
   JOB_SUFFIX_MKDIR: "${GROUP}/${ENV}/${appName}"
   

3. Review Folder Creation Logic
   • Check CommonUtils.createFolders() calls
   • Verify base path configuration
   • Review JobGenie processing logs

Issue 5: Template Not Found

Symptoms:

• Jenkinsfile not found error
• Pipeline fails to load

Solutions:

1. Verify DEFAULT_JENKINSFILE Path

   DEFAULT_JENKINSFILE: 'templates/custom/Jenkinsfile'  // Check path
   

2. Check CONFIG_REPO and CONFIG_BRANCH

   CONFIG_REPO: 'git@github.com:org/repo.git'  // Verify repo
   CONFIG_BRANCH: 'master'  // Verify branch exists
   

3. Ensure Jenkinsfile Exists

   # Verify file exists in repository
   git ls-tree -r HEAD --name-only | grep Jenkinsfile
   

4. Check Credentials
   • Verify JENKINS_GIT_KEY credential exists
   • Test SSH key access
   • Check credential permissions

Issue 6: Variable Resolution Issues

Symptoms:

• Variables not resolving correctly
• Undefined variable errors

Solutions:

1. Check Variable Precedence

   // Order: defaultConfigs → projectConfigs → jobConfig
   def pConfig = defaultConfigs + (projectConfigs[techteam] ?: [:]) + jobConfig
   

2. Verify Variable Names (Case-Sensitive)

   // Correct
   APP_REPO: 'my-app'
      
   // Wrong
   app_repo: 'my-app'  // Different variable
   

3. Review Configuration Merging
   • Check for variable name conflicts
   • Verify map merging logic
   • Review JobGenie processing logs

Debug Mode

Enable detailed logging for troubleshooting:

// In JobGenie.groovy
def globalConfigs = [
    createjob: true,
    debug: true  // Enable debug mode
]

// Add debug output
if (globalConfigs.debug) {
    println("Default Configs: ${defaultConfigs}")
    println("Project Configs: ${projectConfigs}")
    println("Job Config: ${it}")
    println("Merged Config: ${pConfig}")
}

Real-World Examples

Example 1: E-Commerce Platform

Scenario: Large e-commerce platform with 50+ microservices

Challenge:

• Manual job creation taking weeks
• Inconsistent job configurations
• Difficult to maintain

Solution:

def jobGenie = [
    "ecommerce": [
        // Product Services
        [
            APP_REPO: 'product-catalog',
            APP_BRANCH: 'main',
            appNames: ['product-api', 'product-search', 'product-recommendations']
        ],
        // Order Services
        [
            APP_REPO: 'order-management',
            APP_BRANCH: 'main',
            appNames: ['order-api', 'order-processor', 'order-notifications']
        ],
        // Payment Services
        [
            APP_REPO: 'payment-gateway',
            APP_BRANCH: 'main',
            appNames: ['payment-api', 'payment-processor'],
            BUILD: [ NAME: 'payment-gateway' ]
        ]
    ]
]

Result:

• 50+ jobs generated in < 1 hour
• Consistent configuration across all services
• Easy to add new services

Example 2: Financial Services

Scenario: Financial services company with strict compliance requirements

Challenge:

• Need complete audit trail
• Role-based access control
• Environment isolation

Solution:

# Ansible Configuration
jenkins_onboarding:
  finance:
    trading:
      jobs:
        - NAME: "0-Trading-DevOps"
          VARS:
            CONFIG_REPO: "git@github.com:finance/trading-config.git"
      env: ["nonprod", "prod"]
      dev_leads:
        - user: "trading-lead-1"
        - user: "trading-lead-2"

# Role-Based Access
overall_read_users:
  finance:
    trading:
      - user: "trader-1"
      - user: "trader-2"
    compliance:
      - user: "compliance-officer"

Result:

• Complete audit trail in Git
• Fine-grained access control
• Compliance-ready configuration

Example 3: Startup Scaling

Scenario: Startup scaling from 5 to 100 services

Challenge:

• Limited DevOps resources
• Need rapid onboarding
• Cost optimization

Solution:

// Template-based approach
def jobGenie = [
    "startup": [
        // Standard microservice template
        [
            APP_REPO: 'user-service',
            APP_BRANCH: 'main',
            appNames: ['user-api']
        ],
        // Frontend template
        [
            SERVICE: 'web-app',
            CICD_TEMPLATE_NAME: 'frontend',
            JOB_PARAMS: [
                [ name: 'NODE_VERSION', choices: ['node18', 'node20'] ]
            ]
        ]
    ]
]

Result:

• Self-service onboarding
• Reduced DevOps burden by 80%
• Consistent deployments

Performance Optimization

Build Performance

Docker Layer Caching

// Optimize Docker builds
def jobGenie = [
    "mcloud": [
        [
            APP_REPO: 'my-app',
            appNames: ['my-service'],
            DOCKER_BUILD_ARGS: 'ENV',
            // Enable build cache
            DOCKER_BUILD_CACHE: true
        ]
    ]
]

Dependency Caching

// Maven caching
MAVEN_VERSION: 'maven3.9.7'
MAVEN_CACHE_DIR: '/var/cache/maven'

// Node.js caching
NODE_VERSION: 'node18'
NPM_CACHE_DIR: '/var/cache/npm'

Resource Management

Build Timeouts

# Ansible configuration
jenkins_build_timeout: 1800  # 30 minutes

# Per-job timeout
jenkins_job_timeout: 900  # 15 minutes

Concurrent Build Limits

# Limit concurrent builds
jenkins_num_executors: 4
jenkins_concurrent_builds: 2

Monitoring & Observability

Metrics Collection

Build Metrics

• Build success rate
• Average build duration
• Deployment frequency
• Mean time to recovery (MTTR)

System Metrics

• CPU usage
• Memory consumption
• Disk I/O
• Network throughput

Logging Strategy

Centralized Logging

# Filebeat configuration
filebeat:
  enabled: true
  log_path: /var/log/jenkins
  output:
    elasticsearch:
      hosts: ["elasticsearch:9200"]

Audit Trail

# Jenkins audit configuration
jenkins_audit_trail:
  logBuildCause: true
  logCredentialsUsage: true
  logFile: "/var/log/jenkins/audit-trail.log"

Security Best Practices

Credential Management

AWS Secrets Manager Integration

// Use AWS Secrets Manager
def jobGenie = [
    "mcloud": [
        [
            APP_REPO: 'secure-service',
            appNames: ['secure-api'],
            AWS_SECRET_NAME: 'prod/database/credentials',
            CREATE_AWS_SECRET: true
        ]
    ]
]

SSH Key Rotation

# Automated SSH key rotation
jenkins_ssh_key_rotation:
  enabled: true
  rotation_interval: 90  # days
  key_storage: "aws-secrets-manager"

Network Security

• HTTPS only for Jenkins UI
• Firewall rules for Jenkins agents
• Network policies for Kubernetes
• VPN access for sensitive operations

Disaster Recovery

Backup Strategy

Configuration Backup

# Git as source of truth
jenkins_config_backup:
  git_repo: "git@github.com:org/jenkins-config.git"
  backup_interval: "daily"
  
# S3 backup for data
jenkins_s3_backup_dir: "s3://backup-bucket/jenkins/data"

Recovery Procedures

Full Recovery:

1. Provision new server
2. Run Ansible playbook
3. Restore from Git
4. Verify configuration

Partial Recovery:

1. Restore specific jobs
2. Re-run seed jobs
3. Verify functionality

RTO/RPO Targets

• RTO (Recovery Time Objective): < 30 minutes
• RPO (Recovery Point Objective): < 1 hour

Comparison with Alternatives

Jenkins as Code vs. Traditional Jenkins

Feature	Traditional Jenkins	Jenkins as Code
Job Creation	Manual UI	Automated via JobGenie
Configuration	Manual changes	Version-controlled YAML
Onboarding	3-5 days	< 1 hour
Consistency	Varies	100% consistent
Scalability	Limited	Scales to thousands
Audit Trail	Limited	Complete Git history
Disaster Recovery	Manual, hours	Automated, minutes
Self-Service	No	Yes

Jenkins as Code vs. GitLab CI

Feature	GitLab CI	Jenkins as Code
Infrastructure	Managed	Self-hosted with IaC
Job Definition	YAML	Groovy maps (JobGenie)
Scalability	Good	Excellent
Customization	Limited	Highly customizable
Cost	Per user	Self-hosted
Enterprise Features	Limited	Extensive

Jenkins as Code vs. GitHub Actions

Feature	GitHub Actions	Jenkins as Code
Hosting	Cloud	Self-hosted
Job Definition	YAML	Groovy maps
Integration	GitHub only	Multi-platform
Cost	Usage-based	Infrastructure only
Enterprise Control	Limited	Full control

Future Enhancements

Planned Features

1. Web UI for JobGenie
   • Visual job configuration
   • Real-time preview
   • Validation before commit
2. Multi-Cloud Support
   • Azure DevOps integration
   • GCP Cloud Build integration
   • Hybrid cloud deployments
3. Advanced Analytics
   • Build trend analysis
   • Cost optimization recommendations
   • Performance insights
4. AI-Powered Optimization
   • Build failure prediction
   • Resource optimization
   • Deployment risk assessment

Conclusion

Jenkins as Code framework represents a paradigm shift in CI/CD management, transforming Jenkins from a manually configured tool into a fully automated, scalable platform. By combining:

• Infrastructure as Code (Ansible)
• Configuration as Code (YAML/CaC)
• Jobs as Code (JobGenie/JobDSL)

Organizations can achieve:

✅ 90% reduction in onboarding time
✅ 80% reduction in DevOps support tickets
✅ 100% consistency across environments
✅ Zero manual configuration errors
✅ Complete auditability for compliance
✅ Self-service capabilities for development teams

The framework has been successfully deployed in production environments, managing hundreds of microservices with zero manual configuration overhead. It enables organizations to scale their CI/CD infrastructure while maintaining consistency, security, and compliance.

Key Takeaways

1. Automation is Key: Automate everything - infrastructure, configuration, and jobs
2. Version Control Everything: All configurations in Git with full audit trail
3. Self-Service Model: Enable teams to provision their own pipelines
4. Consistency Matters: Same configuration across all environments
5. Security First: Role-based access, credential management, audit logging

Getting Started

Ready to transform your Jenkins infrastructure? Start with:

1. Review the architecture and components
2. Set up Ansible automation
3. Configure Jenkins as Code
4. Create your first JobGenie configuration
5. Run seed job and verify results

For detailed implementation guides, refer to the setup documentation and best practices sections.

---

Built with ❤️ for DevOps Excellence

“Soch Wahi, Approach Nai” - Same Vision, New Approach

---

📚 Additional Resources

📖 Documentation Pages

#### Core Documentation - [🏗️ Architecture Documentation](architecture/) - System architecture and design patterns - [⚙️ JobGenie Complete Guide](jobgenie/) - JobGenie usage, examples, and reference - [🚀 DevOps as a Service](devops-as-service/) - Self-service CI/CD platform guide - [✅ Best Practices](best-practices/) - Recommended practices and guidelines #### Setup & Reference - [🔧 Setup Guide](setup/) - Step-by-step setup and configuration - [📝 Quick Reference](quick-reference/) - Quick reference for common tasks

🧭 Navigation

Previous/Next: Use the navigation menu or browse related topics:

• Start with Architecture to understand the system design
• Learn JobGenie for job generation
• Review Best Practices before implementation
• Check Setup Guide for installation steps

💬 Support

For questions, issues, or contributions:

• Email: HarryTheDevOpsGuy@gmail.com
• Documentation: Browse the documentation pages above
• Issues: GitHub Issues (if applicable)

---

---

Last Updated: January 15, 2024

Version: 1.0.0

Maintained by: DevOps Team

← Back to Tools

Tags: #Jenkins #CI/CD #Infrastructure as Code #Configuration as Code #JobDSL #Ansible #Automation #DevOps #JobGenie

Related Documentation

More from Tools

BG Deployer

Automated blue-green deployment for zero-downtime AWS releases

mCert

SSL certificate monitoring with Slack/email alerts & Telegram

Related by Tags

No related documentation found by tags

Related Blog Posts

OpenResty Production Setup: Supercharge with Lua-Based Metrics and Monitoring

Complete guide to deploying production-ready OpenResty with advanced Lua-based metrics collection...

Oct 16, 2024

KEDA on EKS: Complete Guide to Event-Driven Autoscaling with Real-World Examples

Master KEDA implementation on Amazon EKS with comprehensive examples for multiple scaling scenari...

Oct 15, 2024

AIOps: AI-Powered DevOps Automation and Intelligent Operations

Comprehensive guide to implementing AIOps - using AI and machine learning to transform DevOps pra...

Oct 14, 2024

Related Tools & Projects

BG Deployer

Automated blue-green deployment for zero-downtime AWS releases

AWS Blue-Green

mCert

SSL certificate monitoring with Slack/email alerts & Telegram

SSL Telegram

mTracker

Real-time Linux user activity monitoring with Slack notifications

Linux Monitoring

mWatcher

Server health monitoring for CPU, memory, disk with alerting

Health Alerts

gCrypt

Git-crypt wrapper for secure file encryption & access management

Security Git-crypt

Interactive Tools

AWS VPC Designer, EKS Cost Calculator, and more utilities

Tools Calculator

External Resources

Official Documentation

AWS Documentation

Complete AWS service documentation

Community Resources

GitHub Profile

Open source projects and repositories

LinkedIn

Professional network and updates

Dev.to

Technical articles and tutorials

Contact

Get in touch for questions or collaboration

Quick Actions

All Docs

Browse all documentation

Blog

Latest articles

Tools

DevOps utilities

Portfolio

My work & projects

Found this helpful?

Help us improve this documentation by sharing your feedback or suggesting improvements.

Report Issue Start Discussion