JobGenie

Your DevOps Superpower Unleashed - Transform CI/CD with YAML-driven Jenkins job automation

Updated Jan 15, 2025

12 min read

Intermediate

tools

JobGenie - Your DevOps Superpower Unleashed

“Soch Wahi, Approach Nai” - Redefining CI/CD Automation

JobGenie is a smart, lightweight, and powerful Jenkins job generation engine that transforms complex pipeline creation into simple YAML definitions. It eliminates the need for manual Jenkins UI configuration, enabling teams to define, version, and manage CI/CD pipelines as code.

Overview

JobGenie revolutionizes how DevOps teams manage Jenkins jobs by providing a GitOps-native approach to job creation and management. Instead of manually clicking through Jenkins UI, you define jobs in YAML files, commit them to Git, and JobGenie automatically generates the Jenkins jobs.

What Makes JobGenie Special?

• 🚀 Zero Manual Configuration: Define jobs in YAML, JobGenie creates them automatically
• ⚡ Auto-Discovery: Automatically detects and processes YAML job definitions
• 🛡️ GitOps Native: Full version control and audit trail for all job configurations
• 🎨 Declarative Syntax: Simple, readable YAML syntax for complex pipelines
• 🔄 Multi-Environment: Support for multiple environments (dev, staging, prod) from single definitions
• 📦 Template Support: Reusable pipeline templates for consistent deployments

Why JobGenie?

Problems It Solves

1. Manual Job Creation Overhead
   • ❌ Before: Create jobs manually in Jenkins UI, prone to errors and inconsistencies
   • ✅ After: Define jobs in YAML, commit to Git, auto-generated in Jenkins
2. Configuration Drift
   • ❌ Before: Manual changes get lost, environments diverge
   • ✅ After: All configurations in Git, always consistent across environments
3. Scalability Challenges
   • ❌ Before: Onboarding new services requires manual job creation
   • ✅ After: Add YAML entry, commit, jobs created automatically
4. Lack of Audit Trail
   • ❌ Before: No history of who changed what and when
   • ✅ After: Full Git history with pull requests and reviews
5. Environment Inconsistency
   • ❌ Before: Different configurations across dev/staging/prod
   • ✅ After: Single source of truth with environment-specific overrides

Benefits

• ⚡ Faster Onboarding: New services onboarded in minutes, not days
• 🛡️ Reduced Errors: YAML validation catches errors before deployment
• 📊 Better Visibility: All job configurations visible in Git
• 🔄 Easy Rollback: Revert job changes via Git revert
• 👥 Team Collaboration: Multiple developers can work on job definitions simultaneously
• 📈 Scalability: Handle hundreds of jobs with minimal effort

Key Features

Auto-Discovery Engine

JobGenie automatically scans your repository for YAML files matching the pattern *-jobs.yml or *-jobs.yaml and generates Jenkins jobs from them.

File Structure:

{organization}/{project}/{environment}/jobs/{project}-{environment}-jobs.yml

Example:

amazon/mcloud/prod/jobs/mcloud-prod-jobs.yml
amazon/mcloud/nonprod/jobs/mcloud-nonprod-jobs.yml

Declarative YAML Syntax

Define jobs using simple, readable YAML syntax:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "my-service"
      PARAMETERS:
        - { name: 'GitBranch', string: 'main', description: 'Git branch to build.' }
      CONFIGS:
        APP_REPO: "my-service-repo"
        APP_BRANCH: "main"

Multiple Job Types

Support for both Pipeline and Freestyle jobs:

• Pipeline Jobs: Standard Jenkinsfile-based pipelines
• Freestyle Jobs: Custom script-based jobs with full flexibility

Parameterized Builds

Rich parameter support:

• String Parameters: Text input fields
• Choice Parameters: Dropdown selections
• Boolean Parameters: Checkbox options
• Active Choice Parameters: Dynamic choices based on other parameters

Multi-Environment Support

Define jobs once, deploy to multiple environments:

# Non-production
jobgenie:
  default:
    ENV: "nonprod"
  jobs:
    - NAME: "service"
      CONFIGS:
        APP_BRANCH: "develop"

# Production
jobgenie:
  default:
    ENV: "prod"
  jobs:
    - NAME: "service"
      CONFIGS:
        APP_BRANCH: "production"

GitOps Integration

• Full Git-based workflow
• Pull request reviews for job changes
• Automatic synchronization on commit
• Rollback via Git revert

Security & Compliance

• All configurations version-controlled
• Audit trail via Git history
• Role-based access control integration
• Secure credential management

Architecture

graph TB
    A[JobGenie Seed Job] --> B[Checkout Config Repo]
    B --> C[Scan for YAML Files]
    C --> D[Parse YAML Definitions]
    D --> E[Generate Jobs via JobDSL]
    E --> F[Jenkins Jobs Created]
    
    G[YAML Job Definitions] --> C
    H[Git Repository] --> G
    
    F --> I[Pipeline Jobs]
    F --> J[Freestyle Jobs]

Component Flow

1. Seed Job Execution: JobGenie seed job runs (manually or via webhook)
2. YAML Discovery: Scans repository for *-jobs.yml files
3. YAML Parsing: Parses YAML using SnakeYAML library
4. Job Generation: Uses JobDSL to create/update Jenkins jobs
5. Job Execution: Generated jobs are ready for use

Getting Started

Prerequisites

• Jenkins instance with JobDSL plugin installed
• Git repository access
• Basic understanding of YAML syntax
• Jenkins credentials configured

Step 1: Create Your First YAML Job Definition

Create a YAML file following the naming convention:

File: amazon/myproject/prod/jobs/myproject-prod-jobs.yml

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v2"
    ENV: "prod"
  jobs:
    - NAME: "hello-world-service"
      PARAMETERS:
        - { name: 'GitBranch', string: 'main', description: 'Application git branch.' }
      CONFIGS:
        APP_REPO: "hello-world"
        APP_BRANCH: "main"
        DOCKER_BUILD_ARGS: "ENV,SERVICE"

Step 2: Commit and Push

git add amazon/myproject/prod/jobs/myproject-prod-jobs.yml
git commit -m "Add hello-world-service job definition"
git push origin main

Step 3: Run JobGenie Seed Job

1. Navigate to the JobGenie seed job in Jenkins
2. Click “Build with Parameters”
3. Set GitBranch to your branch (e.g., main)
4. Click “Build”
5. JobGenie will scan, parse, and create your jobs automatically

Step 4: Verify Job Creation

Check Jenkins for the newly created job:

• Path: amazon/myproject/prod/deploy/v2/prod/hello-world-service
• Job should be ready to build immediately

Creating Pipelines with YAML

Basic Pipeline Job

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v2"
    ENV: "prod"
  jobs:
    - NAME: "user-service"
      PARAMETERS:
        - { name: 'GitBranch', string: 'production', description: 'Production branch.' }
        - { name: 'DeployVersion', string: 'latest', description: 'Version to deploy.' }
      CONFIGS:
        APP_REPO: "user-service"
        APP_BRANCH: "production"
        DOCKER_BUILD_ARGS: "ENV,SERVICE"
        SSH_KEYS: "default:/opt/jenkins/keys/prod_key_rsa"

Freestyle Job

jobs:
  - NAME: "database-migration"
    PARAMETERS:
      - { name: 'GitBranch', string: 'master', description: 'Migration scripts branch.' }
      - { name: 'MigrationVersion', string: '', description: 'Specific migration version.' }
      - { name: 'DryRun', bool: false, description: 'Perform dry run.' }
    CONFIGS:
      JOB_TYPE: "freestyle"
      SERVICE: "database-migration"
      APP_REPO: "database-migrations"
      APP_BRANCH: "master"
      SCRIPT: |
        echo "Running database migrations..."
        # Your migration logic here
        echo "Migrations completed successfully"

Job with Choice Parameters

jobs:
  - NAME: "deploy-to-environment"
    PARAMETERS:
      - { name: 'Environment', choice: ['prod', 'prod-dr', 'staging'], description: 'Target environment.' }
      - { name: 'DeployVersion', string: 'latest', description: 'Version to deploy.' }
    CONFIGS:
      APP_REPO: "my-service"
      APP_BRANCH: "production"
      DOCKER_BUILD_ARGS: "ENV,Environment"

Job with Custom Path

jobs:
  - NAME: "shared/notification-service"
    PARAMETERS:
      - { name: 'GitBranch', string: 'main', description: 'Application branch.' }
    CONFIGS:
      APP_REPO: "notification-service"
      APP_BRANCH: "main"
      PIPELINE_PATH: "shared"

Job with Custom Dockerfile

jobs:
  - NAME: "order-service"
    PARAMETERS:
      - { name: 'GitBranch', string: 'main', description: 'Application branch.' }
    CONFIGS:
      APP_REPO: "ecommerce-order-api"
      APP_BRANCH: "main"
      DOCKERFILE_PATH: "services/order/Dockerfile"
      DOCKER_BUILD_ARGS: "ENV"

Examples

Example 1: Simple Microservice

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v2"
    ENV: "prod"
  jobs:
    - NAME: "api-gateway"
      PARAMETERS:
        - { name: 'GitBranch', string: 'production', description: 'Production branch.' }
      CONFIGS:
        APP_REPO: "api-gateway"
        APP_BRANCH: "production"
        DOCKER_BUILD_ARGS: "ENV"

Example 2: Multi-Parameter Job

jobs:
  - NAME: "microservices-orchestrator"
    PARAMETERS:
      - { name: 'TAG', string: 'v3.2.1', description: 'Application version tag.' }
      - { name: 'Services', string: 'all', description: 'Services to deploy (comma-separated or "all").' }
      - { name: 'RollbackOnFailure', bool: true, description: 'Auto-rollback on failure.' }
    CONFIGS:
      SERVICE: "microservices-orchestrator"
      CICD_TEMPLATE_NAME: "microservices-deployment-template"
      APP_REPO: "microservices-platform"
      APP_BRANCH: "production"

Example 3: Infrastructure Job

jobs:
  - NAME: "terraform-infrastructure-deploy"
    PARAMETERS:
      - { name: 'GitBranch', string: 'main', description: 'Terraform code branch.' }
      - { name: 'TerraformAction', choice: ['plan', 'apply', 'destroy'], description: 'Terraform action.' }
      - { name: 'Region', string: 'us-east-1', description: 'AWS region.' }
    CONFIGS:
      JOB_TYPE: "freestyle"
      SERVICE: "terraform-infrastructure-deploy"
      APP_REPO: "terraform-infrastructure"
      APP_BRANCH: "main"
      SCRIPT: |
        echo "Executing Terraform ${TerraformAction} in ${Region}"
        terraform init
        terraform ${TerraformAction}

Example 4: Security Scanning Job

jobs:
  - NAME: "security-vulnerability-scan"
    PARAMETERS:
      - { name: 'ScanType', choice: ['container', 'code', 'dependency'], description: 'Scan type.' }
      - { name: 'Severity', choice: ['critical', 'high', 'medium', 'all'], description: 'Severity level.' }
    CONFIGS:
      JOB_TYPE: "freestyle"
      SERVICE: "security-vulnerability-scan"
      APP_REPO: "security-scanner"
      APP_BRANCH: "main"
      SCRIPT: |
        echo "Running ${ScanType} security scan with ${Severity} severity"
        # Security scanning logic here

Example 5: Database Operations

jobs:
  - NAME: "database-backup-job"
    PARAMETERS:
      - { name: 'DatabaseType', choice: ['postgresql', 'mysql', 'mongodb'], description: 'Database type.' }
      - { name: 'BackupType', choice: ['full', 'incremental'], description: 'Backup type.' }
    CONFIGS:
      JOB_TYPE: "freestyle"
      SERVICE: "database-backup-job"
      APP_REPO: "backup-scripts"
      APP_BRANCH: "master"
      SCRIPT: |
        echo "Performing ${BackupType} backup for ${DatabaseType}"
        # Backup logic here

Advanced Features

Custom Pipeline Templates

Use custom pipeline templates for specialized deployment patterns:

jobs:
  - NAME: "blue-green-deployment"
    PARAMETERS:
      - { name: 'ApplicationName', string: '', description: 'Application name.' }
      - { name: 'TrafficPercentage', string: '10', description: 'Traffic percentage.' }
    CONFIGS:
      SERVICE: "blue-green-deployment"
      CICD_TEMPLATE_NAME: "blue-green-deploy-template"
      APP_REPO: "deployment-automation"
      APP_BRANCH: "main"

DSL Script Jobs

Execute Groovy DSL scripts directly:

jobs:
  - NAME: "jenkins-config-sync"
    PARAMETERS:
      - { name: 'ConfigBranch', string: 'master', description: 'Config branch.' }
    CONFIGS:
      JOB_TYPE: "freestyle"
      SKIP_GIT: true
      SKIP_ENJECTED_VARS: true
      DSL_SCRIPT: |
        import io.jenkins.plugins.casc.ConfigurationAsCode;
        ConfigurationAsCode.get().configure()
        println("Configuration sync completed")

Environment Variable Injection

All CONFIGS are automatically injected as environment variables in your jobs:

jobs:
  - NAME: "my-service"
    CONFIGS:
      APP_REPO: "my-service"
      AWS_REGION: "ap-south-1"
      EKS_CLUSTER: "my-cluster"

These become available as $APP_REPO, $AWS_REGION, $EKS_CLUSTER in your pipelines.

Skip Git Checkout

For jobs that don’t need source code:

jobs:
  - NAME: "system-maintenance"
    CONFIGS:
      JOB_TYPE: "freestyle"
      SKIP_GIT: true
      SCRIPT: |
        echo "Running system maintenance"

Custom Jenkinsfile Path

Specify custom Jenkinsfile location:

jobs:
  - NAME: "custom-pipeline"
    CONFIGS:
      DEFAULT_JENKINSFILE: "custom/path/to/Jenkinsfile"
      APP_REPO: "my-repo"

Configuration Reference

YAML Structure

jobgenie:
  default:                    # Default configurations applied to all jobs
    HOME_DIR: prod            # Home directory
    GROUP: "v2"               # Version group
    ENV: "prod"               # Environment name
  jobs:                       # List of job definitions
    - NAME: "job-name"        # Job name (required)
      PARAMETERS: []          # Build parameters (optional)
      CONFIGS: {}             # Job configurations (required)

Common CONFIGS

Parameter	Description	Example
APP_REPO	Application repository name	"user-service"
APP_BRANCH	Default application branch	"production"
JOB_TYPE	Job type (pipeline or freestyle)	"freestyle"
DOCKER_BUILD_ARGS	Docker build arguments	"ENV,SERVICE"
DOCKERFILE_PATH	Custom Dockerfile path	"services/order/Dockerfile"
SSH_KEYS	SSH key path	"default:/opt/jenkins/keys/key_rsa"
SCRIPT	Shell script (freestyle only)	"echo 'Hello'"
DSL_SCRIPT	Groovy DSL script	"import jenkins.model.Jenkins"
SKIP_GIT	Skip Git SCM checkout	true or false
SKIP_ENJECTED_VARS	Skip environment variable injection	true or false
CICD_TEMPLATE_NAME	Custom pipeline template	"blue-green-deploy-template"
PIPELINE_PATH	Custom job path	"shared"
DISABLED	Disable job	true or false
KEEP_BUILDS	Number of builds to keep	50

Parameter Types

PARAMETERS:
  # String parameter
  - { name: 'GitBranch', string: 'main', description: 'Git branch.' }
  
  # Choice parameter
  - { name: 'Environment', choice: ['prod', 'staging'], description: 'Environment.' }
  
  # Boolean parameter
  - { name: 'DryRun', bool: false, description: 'Dry run mode.' }

Best Practices

1. Naming Conventions

• Use descriptive job names: user-authentication-service not user-svc
• Follow kebab-case for job names
• Use consistent naming across environments

2. YAML Organization

• One YAML file per environment
• Group related jobs together
• Use comments to document complex configurations

3. Parameter Design

• Provide clear descriptions for all parameters
• Use choice parameters for limited options
• Set sensible defaults for string parameters

4. Configuration Management

• Store sensitive data in Jenkins credentials, not YAML
• Use environment variables for environment-specific values
• Leverage default section for common configurations

5. Version Control

• Commit YAML changes via pull requests
• Review job definitions before merging
• Tag releases for production job definitions

6. Testing

• Test job definitions in non-production first
• Validate YAML syntax before committing
• Verify job creation after seed job runs

Troubleshooting

Jobs Not Created

Problem: Seed job runs but no jobs are created.

Solutions:

• Verify YAML file naming: must end with -jobs.yml or -jobs.yaml
• Check YAML syntax is valid
• Review seed job console output for errors
• Ensure file path matches expected structure: {org}/{project}/{env}/jobs/{file}.yml

YAML Parsing Errors

Problem: Seed job fails with YAML parsing errors.

Solutions:

• Validate YAML syntax using online validators
• Check indentation (must be spaces, not tabs)
• Verify all required fields are present
• Review console output for specific error messages

Jobs Created in Wrong Location

Problem: Jobs appear in unexpected folders.

Solutions:

• Check HOME_DIR, GROUP, and ENV in default section
• Verify PIPELINE_PATH if using custom paths
• Review job name structure (slashes create subfolders)

Parameters Not Appearing

Problem: Build parameters don’t show up in job.

Solutions:

• Verify parameter syntax is correct
• Check parameter names don’t conflict with reserved words
• Ensure PARAMETERS is a list, not a map

Environment Variables Not Available

Problem: CONFIGS values not available as environment variables.

Solutions:

• Ensure SKIP_ENJECTED_VARS is not set to true
• Check that CONFIGS values don’t contain newlines (use SCRIPT for multi-line)
• Verify job type supports environment variables

Contributing

JobGenie is part of the mCloud-Jenkins project. To contribute:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Test thoroughly
5. Submit a pull request

Support

• Maintainer: Hari Prasad (HarryTheDevOpsGuy)
• Email: HarryTheDevOpsGuy@gmail.com
• Documentation: Full Documentation
• Website: https://harrythedevopsguy.github.io/

License

Proprietary - Internal Use Only

Acknowledgments

• Jenkins Community
• JobDSL Plugin Maintainers
• All contributors and users

---

**Need Help?** Check out the [JobGenie README](https://github.com/HarryTheDevOpsGuy/mCloud-Jenkins/blob/main/JobGenie/README.md) for more detailed information or reach out to the maintainer for support.

Built with ❤️ by the DevOps Team

“Soch Wahi, Approach Nai” - Same Vision, New Approach

From Zero to Hero in Minutes, Not Months 🚀

Tags: #Jenkins #JobDSL #YAML #GitOps #CI/CD #Automation #DevOps

Related Documentation

More from Tools

BG Deployer

Automated blue-green deployment for zero-downtime AWS releases

mCert

SSL certificate monitoring with Slack/email alerts & Telegram

Related by Tags

No related documentation found by tags

Related Blog Posts

OpenResty Production Setup: Supercharge with Lua-Based Metrics and Monitoring

Complete guide to deploying production-ready OpenResty with advanced Lua-based metrics collection...

Oct 16, 2024

KEDA on EKS: Complete Guide to Event-Driven Autoscaling with Real-World Examples

Master KEDA implementation on Amazon EKS with comprehensive examples for multiple scaling scenari...

Oct 15, 2024

AIOps: AI-Powered DevOps Automation and Intelligent Operations

Comprehensive guide to implementing AIOps - using AI and machine learning to transform DevOps pra...

Oct 14, 2024

Related Tools & Projects

BG Deployer

Automated blue-green deployment for zero-downtime AWS releases

AWS Blue-Green

mCert

SSL certificate monitoring with Slack/email alerts & Telegram

SSL Telegram

mTracker

Real-time Linux user activity monitoring with Slack notifications

Linux Monitoring

mWatcher

Server health monitoring for CPU, memory, disk with alerting

Health Alerts

gCrypt

Git-crypt wrapper for secure file encryption & access management

Security Git-crypt

Interactive Tools

AWS VPC Designer, EKS Cost Calculator, and more utilities

Tools Calculator

External Resources

Official Documentation

AWS Documentation

Complete AWS service documentation

Community Resources

GitHub Profile

Open source projects and repositories

LinkedIn

Professional network and updates

Dev.to

Technical articles and tutorials

Contact

Get in touch for questions or collaboration

Quick Actions

All Docs

Browse all documentation

Blog

Latest articles

Tools

DevOps utilities

Portfolio

My work & projects

Found this helpful?

Help us improve this documentation by sharing your feedback or suggesting improvements.

Report Issue Start Discussion