Architecture Documentation - Jenkins as Code
Comprehensive architecture guide for Jenkins as Code automation framework
Architecture Documentation
Comprehensive architecture guide for Jenkins as Code automation framework
📚 Related Guides:
- Setup Guide - Complete Jenkins setup and JobGenie integration
- JobGenie Job Creation Guide - Create jobs with YAML
- JobGenie Reference - Complete technical reference
- Best Practices - Recommended patterns
📋 Table of Contents
- System Overview
- Architecture Layers
- Component Details
- Data Flow
- Security Architecture
- Scalability & Performance
- Disaster Recovery
🎯 System Overview
High-Level Architecture
┌─────────────────────────────────────────────────────────────────┐
│ Infrastructure Layer (Ansible) │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Packer │ │ Monitoring │ │ Jenkins │ │
│ │ Playbook │ │ Stack │ │ Stack │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Configuration Layer (CaC) │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ jenkins.yaml │ │ seed-jobs │ │ init.groovy │ │
│ │ (CaC) │ │ (JobDSL) │ │ (Plugins) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Job Generation Layer (JobGenie) │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ JobGenie │ │ Shared │ │ Pipeline │ │
│ │ Scripts │ │ Libraries │ │ Templates │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Execution Layer (Jenkins Jobs) │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Build │ │ Deploy │ │ Manage │ │
│ │ Jobs │ │ Jobs │ │ Jobs │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────┘
🏗️ Architecture Layers
Layer 1: Infrastructure Layer
Purpose: Provision and configure Jenkins infrastructure
Components:
- Ansible Playbooks: Infrastructure automation
- Packer Templates: AMI/image building
- Monitoring Stack: Telegraf, Filebeat, ELK
Key Files:
mCloud-infra/ansible/
├── packer.yml # Main playbook
├── roles/
│ ├── monitoring/ # Monitoring stack
│ └── users/ # User management
└── group_vars/
└── packer_al2023_aarch64_devops_jenkins.yml
Responsibilities:
- Server provisioning
- Jenkins installation
- Plugin management
- System configuration
- Monitoring setup
Layer 2: Configuration Layer
Purpose: Manage Jenkins configuration as code
Components:
- Jenkins Configuration as Code (CaC): YAML-based config
- JobDSL Seed Jobs: Initial job generation
- Init Scripts: Plugin installation
Key Files:
mCloud-infra/ansible/roles/monitoring/templates/stacks/jenkins/
├── jenkins.yaml.j2 # Main CaC template
├── seed-jobs.groovy.j2 # Seed job DSL
└── init.groovy.j2 # Plugin initialization
Configuration Sources:
- Ansible Variables → Jinja2 Templates → YAML Files → Jenkins CaC
Responsibilities:
- Jenkins system configuration
- Security settings
- Global libraries
- Credential management
- View configuration
Layer 3: Job Generation Layer
Purpose: Dynamically generate Jenkins jobs
Components:
- JobGenie Scripts: Job definition files
- Shared Libraries: Reusable pipeline utilities
- Pipeline Templates: Standardized CI/CD patterns
Key Files:
mCloud-Jenkins/
├── amazon/
│ └── mcloud/
│ └── nonprod/
│ └── pipelines/
│ └── jobs/
│ └── JobGenie.groovy
└── sharedlibs/
├── src/org/devops/
│ └── CommonUtils.groovy
└── vars/
├── pipelineUtils.groovy
└── sendNotifications.groovy
Job Generation Flow:
JobGenie.groovy → Seed Job → JobDSL → Jenkins Jobs
Responsibilities:
- Job definition parsing
- Job generation logic
- Folder structure creation
- Parameter configuration
- Pipeline linking
Layer 4: Execution Layer
Purpose: Execute CI/CD pipelines
Components:
- Generated Jenkins Jobs: Build, deploy, manage jobs
- Pipeline Templates: Reusable Jenkinsfiles
- External Integrations: ArgoCD, ECR, Slack
Job Types:
- Build Jobs: Docker image building
- Deploy Jobs: Kubernetes deployments
- Manage Jobs: Infrastructure management
🔧 Component Details
1. Ansible Automation
Playbook Structure
# packer.yml
- hosts: packer_al2023_aarch64_devops_jenkins
roles:
- monitoring
tasks:
- name: Install Jenkins
- name: Configure Jenkins
- name: Install plugins
Role Organization
roles/
├── monitoring/
│ ├── tasks/
│ │ ├── install/
│ │ └── main.yml
│ ├── templates/
│ │ └── stacks/jenkins/
│ └── defaults/
│ └── main.yml
2. Jenkins Configuration as Code
Configuration Structure
jenkins:
numExecutors: 2
authorizationStrategy:
roleBased:
roles:
global:
- name: "admin"
permissions: ["Overall/Administer"]
views:
- all:
name: "all"
Template Generation
# jenkins.yaml.j2
jenkins:
numExecutors: {{ jenkins_num_executors | default(2) }}
authorizationStrategy:
roleBased:
roles:
global:
- name: "admin"
entries: {{ overall_admin_users | to_json }}
3. JobGenie Engine
Job Definition Format
# File: amazon/mcloud/prod/jobs/mcloud-prod-jobs.yml
jobgenie:
default:
HOME_DIR: prod
GROUP: "v4"
ENV: "prod"
jobs:
- NAME: "my-service"
PARAMETERS:
- { name: 'GitBranch', string: 'master', description: 'Application branch.' }
CONFIGS:
APP_REPO: "my-app"
APP_BRANCH: "master"
DOCKER_BUILD_ARGS: "ENV"
Generation Process
1. Seed job scans repository for YAML files ending with "-jobs.yml"
2. Parses YAML structure (jobgenie.default and jobgenie.jobs)
3. Merges default configurations with job-specific CONFIGS
4. Creates folder structure: {org}/{project}/{env}/deploy/{GROUP}/{ENV}/{JOB_NAME}
5. Generates Jenkins jobs via JobDSL
6. Jobs appear in Jenkins UI
4. Shared Libraries
CommonUtils Class
class CommonUtils {
static void createFolders(DslFactory dsl, List<Map> configs)
static void pipelineGenerator(DslFactory dsl, Map deploy, ...)
static void mCloudJobsGenerator(DslFactory dsl, Map jobGenie, ...)
}
Pipeline Utilities
// vars/pipelineUtils.groovy
def generateDockerArgs(Map opts = [:])
def updateCanarySteps(int steps, String file)
def sendJobNotification(String status, String message = '')
🔄 Data Flow
Configuration Flow
┌─────────────┐
│ Developer │
└──────┬──────┘
│ 1. Edit YAML job definition file
│ (e.g., mcloud-prod-jobs.yml)
▼
┌─────────────┐
│ Git Repo │
└──────┬──────┘
│ 2. Commit & Push
▼
┌─────────────┐
│ Seed Job │
└──────┬──────┘
│ 3. Auto-discover YAML files
│ (files ending with -jobs.yml)
▼
┌─────────────┐
│ JobDSL │
└──────┬──────┘
│ 4. Generate Jobs
▼
┌─────────────┐
│ Jenkins │
└─────────────┘
Deployment Flow
┌─────────────┐
│ Build Job │
└──────┬──────┘
│ 1. Build Docker Image
▼
┌─────────────┐
│ ECR │
└──────┬──────┘
│ 2. Push Image
▼
┌─────────────┐
│ Deploy Job │
└──────┬──────┘
│ 3. Update Config
▼
┌─────────────┐
│ ArgoCD │
└──────┬──────┘
│ 4. Deploy to K8s
▼
┌─────────────┐
│ EKS │
└─────────────┘
🔐 Security Architecture
Authentication & Authorization
Security Realm
jenkins:
securityRealm:
local:
allowsSignup: false
users:
- id: "admin"
password: "Admin@123"
Role-Based Access Control
authorizationStrategy:
roleBased:
roles:
items:
- name: "amazon-mcloud-dev"
pattern: "amazon/mcloud/.*/nonprod/deploy/.*"
templateName: "build"
entries:
- user: "dev"
Credential Management
Credential Storage
- Jenkins Credentials Plugin
- AWS Secrets Manager integration
- Encrypted at rest
SSH Key Management
SSH_KEYS: 'default:/var/lib/jenkins/.ssh/id_rsa'
Network Security
- HTTPS only
- Firewall rules
- Network policies
- VPN access for sensitive operations
📈 Scalability & Performance
Horizontal Scaling
Jenkins Agents
jenkins_slave_nodes:
linux-slave:
host: "10.20.0.2"
numExecutors: 4
Load Distribution
- Multiple Jenkins agents
- Job distribution
- Resource optimization
Performance Optimization
Caching
- Docker layer caching
- Maven/Gradle caching
- Git repository caching
Resource Management
- Build timeouts
- Concurrent build limits
- Resource quotas
🛡️ Disaster Recovery
Backup Strategy
Configuration Backup
- Git repository (source of truth)
- Jenkins configuration export
- Regular backups
Data Backup
jenkins_s3_backup_dir: "s3://mys3-bucket/amazon/mcloud/backup/jenkins"
Recovery Procedures
Full Recovery
- Provision new server
- Run Ansible playbook
- Restore from Git
- Verify configuration
Partial Recovery
- Restore specific jobs
- Re-run seed jobs
- Verify functionality
RTO/RPO Targets
- RTO: < 30 minutes
- RPO: < 1 hour
📊 Monitoring & Observability
Metrics Collection
System Metrics
- CPU, Memory, Disk
- Network I/O
- Jenkins performance
Application Metrics
- Build success rate
- Deployment frequency
- Mean time to recovery
Logging
Log Aggregation
- Filebeat → ELK Stack
- Centralized logging
- Log retention policies
Audit Trail
audit-trail:
logBuildCause: true
logCredentialsUsage: true
logFile: "/var/log/jenkins/audit-trail.log"
🔄 Continuous Improvement
Version Control
- All configurations in Git
- Semantic versioning
- Change tracking
Testing
- Ansible playbook testing
- JobDSL validation
- Integration testing
Documentation
- Architecture documentation
- Runbooks
- Troubleshooting guides
📚 Additional Resources
📖 Related Documentation
- 🏠 Main Documentation - Complete Jenkins as Code framework overview
- ⚙️ JobGenie Guide - JobGenie usage and examples
- 🚀 DevOps as a Service - Self-service CI/CD platform guide
- ✅ Best Practices - Recommended practices and guidelines
- 🔧 Setup Guide - Setup and configuration guide
- 📝 Quick Reference - Quick reference guide
🧭 Navigation
Related Topics:
- Learn JobGenie to understand job generation
- Review Best Practices for architecture decisions
- See DevOps as a Service for implementation patterns
Last Updated: January 15, 2024
Version: 1.0.0
Related Documentation
More from Tools
Best Practices Guide - Jenkins as Code
Recommended practices for using Jenkins as Code automation framework
DevOps as a Service - Automated CI/CD Management
Transforming Jenkins from a bottleneck to a self-service platform for develop...
JobGenie - Complete Job Creation Guide
Step-by-step guide to creating and configuring Jenkins jobs using JobGenie YA...
JobGenie - Complete Guide
Your friendly Jenkins job generator - Comprehensive guide to JobGenie job gen...
Quick Reference Guide - Jenkins as Code
Quick reference for common tasks and configurations in Jenkins as Code
Jenkins as Code - Complete Setup Guide
Step-by-step guide to set up Jenkins as Code with plugins, access control, an...
Jenkins as Code - Enterprise CI/CD Automation
Complete Jenkins automation framework with Infrastructure as Code, Configurat...
BG Deployer
Automated blue-green deployment for zero-downtime AWS releases
DevOps Tools & Utilities | Hari Prasad
Custom-built DevOps tools for automation, monitoring, deployment, and security
JobGenie Getting Started
Your DevOps Superpower Unleashed - Transform CI/CD with YAML-driven Jenkins j...
JobGenie
Your DevOps Superpower Unleashed - Transform CI/CD with YAML-driven Jenkins j...
mCert
SSL certificate monitoring with Slack/email alerts & Telegram
mTracker
Real-time Linux user activity monitoring with Slack notifications
mWatcher
Server health monitoring for CPU, memory, disk with alerting
Sample DevOps Tool Documentation
A comprehensive guide to using our sample DevOps tool for automation and moni...
Typography Demo
Demonstration of enhanced typography features in the documentation template
Related Blog Posts
JobGenie: Transform Jenkins Job Creation with Jobs as Code
Learn how to integrate JobGenie into your existing Jenkins instance to create jobs as code using ...
OpenResty Production Setup: Supercharge with Lua-Based Metrics and Monitoring
Complete guide to deploying production-ready OpenResty with advanced Lua-based metrics collection...
KEDA on EKS: Complete Guide to Event-Driven Autoscaling with Real-World Examples
Master KEDA implementation on Amazon EKS with comprehensive examples for multiple scaling scenari...
Related Tools & Projects
BG Deployer
Automated blue-green deployment for zero-downtime AWS releases
mCert
SSL certificate monitoring with Slack/email alerts & Telegram
mTracker
Real-time Linux user activity monitoring with Slack notifications
mWatcher
Server health monitoring for CPU, memory, disk with alerting
gCrypt
Git-crypt wrapper for secure file encryption & access management
Interactive Tools
AWS VPC Designer, EKS Cost Calculator, and more utilities
External Resources
Quick Actions
Found this helpful?
Help us improve this documentation by sharing your feedback or suggesting improvements.