JobGenie

Your DevOps Superpower Unleashed - Transform CI/CD with YAML-driven Jenkins job automation

Updated Jan 15, 2025

12 min read

Intermediate

tools

JobGenie - Your DevOps Superpower Unleashed

“Soch Wahi, Approach Nai” - Redefining CI/CD Automation

JobGenie is a smart, lightweight, and powerful Jenkins job generation engine that transforms complex pipeline creation into simple YAML definitions. It eliminates the need for manual Jenkins UI configuration, enabling teams to define, version, and manage CI/CD pipelines as code.

Visit JobGenie Homepage • Examples • GitHub

🚀 New to JobGenie? Start with our comprehensive Getting Started Guide to integrate JobGenie into your Jenkins instance and create your first YAML-driven jobs.

📋 Quick Navigation

• Getting Started Guide - Complete integration instructions
• Key Features - Core capabilities and benefits
• Architecture - How JobGenie works
• Creating Pipelines - YAML job definition examples
• Configuration Reference - Complete parameter reference
• Examples - Real-world use cases
• Best Practices - Recommended patterns
• Troubleshooting - Common issues and solutions

Overview

JobGenie revolutionizes how DevOps teams manage Jenkins jobs by providing a GitOps-native approach to job creation and management. Instead of manually clicking through Jenkins UI, you define jobs in YAML files, commit them to Git, and JobGenie automatically generates the Jenkins jobs.

What Makes JobGenie Special?

• 🚀 Zero Manual Configuration: Define jobs in YAML, JobGenie creates them automatically - no complex logic needed!
• ⚡ Auto-Discovery: Automatically detects and processes *-jobs.yml files
• 🛡️ GitOps Native: Full version control and audit trail for all job configurations
• 🎨 Declarative Syntax: Simple, readable YAML syntax for complex pipelines
• 🔄 Multi-Environment: Support for multiple environments (dev, staging, prod) from single definitions
• 📦 Template Support: Reusable pipeline templates for consistent deployments

💡 Key Benefit: You don’t need to write complex Groovy code - just simple YAML definitions!

Why JobGenie?

Problems It Solves

1. Manual Job Creation Overhead
   • ❌ Before: Create jobs manually in Jenkins UI, prone to errors and inconsistencies
   • ✅ After: Define jobs in YAML, commit to Git, auto-generated in Jenkins
2. Configuration Drift
   • ❌ Before: Manual changes get lost, environments diverge
   • ✅ After: All configurations in Git, always consistent across environments
3. Scalability Challenges
   • ❌ Before: Onboarding new services requires manual job creation
   • ✅ After: Add YAML entry, commit, jobs created automatically
4. Lack of Audit Trail
   • ❌ Before: No history of who changed what and when
   • ✅ After: Full Git history with pull requests and reviews
5. Environment Inconsistency
   • ❌ Before: Different configurations across dev/staging/prod
   • ✅ After: Single source of truth with environment-specific overrides

Benefits

• ⚡ Faster Onboarding: New services onboarded in minutes, not days
• 🛡️ Reduced Errors: YAML validation catches errors before deployment
• 📊 Better Visibility: All job configurations visible in Git
• 🔄 Easy Rollback: Revert job changes via Git revert
• 👥 Team Collaboration: Multiple developers can work on job definitions simultaneously
• 📈 Scalability: Handle hundreds of jobs with minimal effort

Key Features

Auto-Discovery Engine

JobGenie automatically scans your repository for YAML files matching the pattern *-jobs.yml or *-jobs.yaml and generates Jenkins jobs from them.

File Structure:

{organization}/{project}/{environment}/jobs/{project}-{environment}-jobs.yml

Example:

amazon/mcloud/prod/jobs/mcloud-prod-jobs.yml
amazon/mcloud/nonprod/jobs/mcloud-nonprod-jobs.yml

Declarative YAML Syntax

Define jobs using simple, readable YAML syntax:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "my-service"
      PARAMETERS:
        - { name: 'GitBranch', string: 'main', description: 'Git branch to build.' }
      CONFIGS:
        APP_REPO: "my-service-repo"
        APP_BRANCH: "main"

Multiple Job Types

Support for both Pipeline and Freestyle jobs:

• Pipeline Jobs: Standard Jenkinsfile-based pipelines
• Freestyle Jobs: Custom script-based jobs with full flexibility

Parameterized Builds

Rich parameter support:

• String Parameters: Text input fields
• Choice Parameters: Dropdown selections
• Boolean Parameters: Checkbox options
• Active Choice Parameters: Dynamic choices based on other parameters

Multi-Environment Support

Define jobs once, deploy to multiple environments:

# Non-production
jobgenie:
  default:
    ENV: "nonprod"
  jobs:
    - NAME: "service"
      CONFIGS:
        APP_BRANCH: "develop"

# Production
jobgenie:
  default:
    ENV: "prod"
  jobs:
    - NAME: "service"
      CONFIGS:
        APP_BRANCH: "production"

GitOps Integration

• Full Git-based workflow
• Pull request reviews for job changes
• Automatic synchronization on commit
• Rollback via Git revert

Security & Compliance

• All configurations version-controlled
• Audit trail via Git history
• Role-based access control integration
• Secure credential management

Architecture

JobGenie follows a simple yet powerful architecture that enables GitOps-native job management:

graph TB
    A[JobGenie Seed Job] --> B[Checkout Config Repo]
    B --> C[Scan for YAML Files]
    C --> D[Parse YAML Definitions]
    D --> E[Generate Jobs via JobDSL]
    E --> F[Jenkins Jobs Created]
    
    G[YAML Job Definitions] --> C
    H[Git Repository] --> G
    
    F --> I[Pipeline Jobs]
    F --> J[Freestyle Jobs]

Component Flow

1. Seed Job Execution: JobGenie seed job runs (manually or via webhook)
2. YAML Discovery: Automatically scans repository for *-jobs.yml files
3. YAML Parsing: Parses YAML definitions using SnakeYAML library
4. Job Generation: Uses JobDSL to create/update Jenkins jobs programmatically
5. Job Execution: Generated jobs are ready for immediate use

Core Technologies

• JobDSL Plugin: Programmatic job creation and management
• Configuration as Code (JCasC): System configuration as YAML
• SnakeYAML: YAML parsing and validation
• Git: Version control and GitOps workflow
• Ansible: Automated Jenkins setup and configuration (optional)

💡 Tip: For detailed integration steps, see the Getting Started Guide.

Getting Started

Ready to transform your Jenkins workflow? Follow our comprehensive guide to get started with JobGenie.

Quick Start

If you already have Jenkins configured, you can create your first job in minutes:

1. Create a YAML job definition following the naming convention
2. Commit and push to your Git repository
3. Run the JobGenie seed job to generate Jenkins jobs automatically

Detailed Integration Guide

For complete setup instructions, including:

• ✅ Prerequisites and system requirements
• ✅ Automated setup with Ansible (recommended for production)
• ✅ Manual integration (for existing Jenkins instances)
• ✅ Directory structure setup
• ✅ Creating your first job
• ✅ Verification and testing
• ✅ Troubleshooting common issues

👉 Read the Complete Getting Started Guide →

Quick Example

Here’s a simple example to get you started:

File: amazon/myproject/prod/jobs/myproject-prod-jobs.yml

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v2"
    ENV: "prod"
  jobs:
    - NAME: "hello-world-service"
      PARAMETERS:
        - { name: 'GitBranch', string: 'main', description: 'Application git branch.' }
      CONFIGS:
        APP_REPO: "hello-world"
        APP_BRANCH: "main"
        DOCKER_BUILD_ARGS: "ENV,SERVICE"

After committing and pushing, run the JobGenie seed job to automatically create the Jenkins job.

Creating Pipelines with YAML

Learn how to define Jenkins jobs using JobGenie’s declarative YAML syntax. For integration setup, see the Getting Started Guide.

Basic Pipeline Job

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v2"
    ENV: "prod"
  jobs:
    - NAME: "user-service"
      PARAMETERS:
        - { name: 'GitBranch', string: 'production', description: 'Production branch.' }
        - { name: 'DeployVersion', string: 'latest', description: 'Version to deploy.' }
      CONFIGS:
        APP_REPO: "user-service"
        APP_BRANCH: "production"
        DOCKER_BUILD_ARGS: "ENV,SERVICE"
        SSH_KEYS: "default:/opt/jenkins/keys/prod_key_rsa"

Freestyle Job

jobs:
  - NAME: "database-migration"
    PARAMETERS:
      - { name: 'GitBranch', string: 'master', description: 'Migration scripts branch.' }
      - { name: 'MigrationVersion', string: '', description: 'Specific migration version.' }
      - { name: 'DryRun', bool: false, description: 'Perform dry run.' }
    CONFIGS:
      JOB_TYPE: "freestyle"
      SERVICE: "database-migration"
      APP_REPO: "database-migrations"
      APP_BRANCH: "master"
      SCRIPT: |
        echo "Running database migrations..."
        # Your migration logic here
        echo "Migrations completed successfully"

Job with Choice Parameters

jobs:
  - NAME: "deploy-to-environment"
    PARAMETERS:
      - { name: 'Environment', choices: ['prod', 'prod-dr', 'staging'], description: 'Target environment.' }
      - { name: 'DeployVersion', string: 'latest', description: 'Version to deploy.' }
    CONFIGS:
      APP_REPO: "my-service"
      APP_BRANCH: "production"
      DOCKER_BUILD_ARGS: "ENV,Environment"

Job with Custom Path

jobs:
  - NAME: "shared/notification-service"
    PARAMETERS:
      - { name: 'GitBranch', string: 'main', description: 'Application branch.' }
    CONFIGS:
      APP_REPO: "notification-service"
      APP_BRANCH: "main"
      PIPELINE_PATH: "shared"

Job with Custom Dockerfile

jobs:
  - NAME: "order-service"
    PARAMETERS:
      - { name: 'GitBranch', string: 'main', description: 'Application branch.' }
    CONFIGS:
      APP_REPO: "ecommerce-order-api"
      APP_BRANCH: "main"
      DOCKERFILE_PATH: "services/order/Dockerfile"
      DOCKER_BUILD_ARGS: "ENV"

Examples

Example 1: Simple Microservice

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v2"
    ENV: "prod"
  jobs:
    - NAME: "api-gateway"
      PARAMETERS:
        - { name: 'GitBranch', string: 'production', description: 'Production branch.' }
      CONFIGS:
        APP_REPO: "api-gateway"
        APP_BRANCH: "production"
        DOCKER_BUILD_ARGS: "ENV"

Example 2: Multi-Parameter Job

jobs:
  - NAME: "microservices-orchestrator"
    PARAMETERS:
      - { name: 'TAG', string: 'v3.2.1', description: 'Application version tag.' }
      - { name: 'Services', string: 'all', description: 'Services to deploy (comma-separated or "all").' }
      - { name: 'RollbackOnFailure', bool: true, description: 'Auto-rollback on failure.' }
    CONFIGS:
      SERVICE: "microservices-orchestrator"
      CICD_TEMPLATE_NAME: "microservices-deployment-template"
      APP_REPO: "microservices-platform"
      APP_BRANCH: "production"

Example 3: Infrastructure Job

jobs:
  - NAME: "terraform-infrastructure-deploy"
    PARAMETERS:
      - { name: 'GitBranch', string: 'main', description: 'Terraform code branch.' }
      - { name: 'TerraformAction', choices: ['plan', 'apply', 'destroy'], description: 'Terraform action.' }
      - { name: 'Region', string: 'us-east-1', description: 'AWS region.' }
    CONFIGS:
      JOB_TYPE: "freestyle"
      SERVICE: "terraform-infrastructure-deploy"
      APP_REPO: "terraform-infrastructure"
      APP_BRANCH: "main"
      SCRIPT: |
        echo "Executing Terraform ${TerraformAction} in ${Region}"
        terraform init
        terraform ${TerraformAction}

Example 4: Security Scanning Job

jobs:
  - NAME: "security-vulnerability-scan"
    PARAMETERS:
      - { name: 'ScanType', choices: ['container', 'code', 'dependency'], description: 'Scan type.' }
      - { name: 'Severity', choices: ['critical', 'high', 'medium', 'all'], description: 'Severity level.' }
    CONFIGS:
      JOB_TYPE: "freestyle"
      SERVICE: "security-vulnerability-scan"
      APP_REPO: "security-scanner"
      APP_BRANCH: "main"
      SCRIPT: |
        echo "Running ${ScanType} security scan with ${Severity} severity"
        # Security scanning logic here

Example 5: Database Operations

jobs:
  - NAME: "database-backup-job"
    PARAMETERS:
      - { name: 'DatabaseType', choices: ['postgresql', 'mysql', 'mongodb'], description: 'Database type.' }
      - { name: 'BackupType', choices: ['full', 'incremental'], description: 'Backup type.' }
    CONFIGS:
      JOB_TYPE: "freestyle"
      SERVICE: "database-backup-job"
      APP_REPO: "backup-scripts"
      APP_BRANCH: "master"
      SCRIPT: |
        echo "Performing ${BackupType} backup for ${DatabaseType}"
        # Backup logic here

Advanced Features

Custom Pipeline Templates

Use custom pipeline templates for specialized deployment patterns:

jobs:
  - NAME: "blue-green-deployment"
    PARAMETERS:
      - { name: 'ApplicationName', string: '', description: 'Application name.' }
      - { name: 'TrafficPercentage', string: '10', description: 'Traffic percentage.' }
    CONFIGS:
      SERVICE: "blue-green-deployment"
      CICD_TEMPLATE_NAME: "blue-green-deploy-template"
      APP_REPO: "deployment-automation"
      APP_BRANCH: "main"

DSL Script Jobs

Execute Groovy DSL scripts directly:

jobs:
  - NAME: "jenkins-config-sync"
    PARAMETERS:
      - { name: 'ConfigBranch', string: 'master', description: 'Config branch.' }
    CONFIGS:
      JOB_TYPE: "freestyle"
      SKIP_GIT: true
      SKIP_ENJECTED_VARS: true
      DSL_SCRIPT: |
        import io.jenkins.plugins.casc.ConfigurationAsCode;
        ConfigurationAsCode.get().configure()
        println("Configuration sync completed")

Environment Variable Injection

All CONFIGS are automatically injected as environment variables in your jobs:

jobs:
  - NAME: "my-service"
    CONFIGS:
      APP_REPO: "my-service"
      AWS_REGION: "ap-south-1"
      EKS_CLUSTER: "my-cluster"

These become available as $APP_REPO, $AWS_REGION, $EKS_CLUSTER in your pipelines.

Skip Git Checkout

For jobs that don’t need source code:

jobs:
  - NAME: "system-maintenance"
    CONFIGS:
      JOB_TYPE: "freestyle"
      SKIP_GIT: true
      SCRIPT: |
        echo "Running system maintenance"

Custom Jenkinsfile Path

Specify custom Jenkinsfile location:

jobs:
  - NAME: "custom-pipeline"
    CONFIGS:
      DEFAULT_JENKINSFILE: "custom/path/to/Jenkinsfile"
      APP_REPO: "my-repo"

Configuration Reference

YAML Structure

jobgenie:
  default:                    # Default configurations applied to all jobs
    HOME_DIR: prod            # Home directory
    GROUP: "v2"               # Version group
    ENV: "prod"               # Environment name
  jobs:                       # List of job definitions
    - NAME: "job-name"        # Job name (required)
      PARAMETERS: []          # Build parameters (optional)
      CONFIGS: {}             # Job configurations (required)

Common CONFIGS

Parameter	Description	Example
APP_REPO	Application repository name	"user-service"
APP_BRANCH	Default application branch	"production"
JOB_TYPE	Job type (pipeline or freestyle)	"freestyle"
DOCKER_BUILD_ARGS	Docker build arguments	"ENV,SERVICE"
DOCKERFILE_PATH	Custom Dockerfile path	"services/order/Dockerfile"
SSH_KEYS	SSH key path	"default:/opt/jenkins/keys/key_rsa"
SCRIPT	Shell script (freestyle only)	"echo 'Hello'"
DSL_SCRIPT	Groovy DSL script	"import jenkins.model.Jenkins"
SKIP_GIT	Skip Git SCM checkout	true or false
SKIP_ENJECTED_VARS	Skip environment variable injection	true or false
CICD_TEMPLATE_NAME	Custom pipeline template	"blue-green-deploy-template"
PIPELINE_PATH	Custom job path	"shared"
DISABLED	Disable job	true or false
KEEP_BUILDS	Number of builds to keep	50

Parameter Types

PARAMETERS:
  # String parameter
  - { name: 'GitBranch', string: 'main', description: 'Git branch.' }
  
  # Choice parameter
  - { name: 'Environment', choices: ['prod', 'staging'], description: 'Environment.' }
  
  # Boolean parameter
  - { name: 'DryRun', bool: false, description: 'Dry run mode.' }

Best Practices

1. Naming Conventions

• Use descriptive job names: user-authentication-service not user-svc
• Follow kebab-case for job names
• Use consistent naming across environments

2. YAML Organization

• One YAML file per environment
• Group related jobs together
• Use comments to document complex configurations

3. Parameter Design

• Provide clear descriptions for all parameters
• Use choice parameters for limited options
• Set sensible defaults for string parameters

4. Configuration Management

• Store sensitive data in Jenkins credentials, not YAML
• Use environment variables for environment-specific values
• Leverage default section for common configurations

5. Version Control

• Commit YAML changes via pull requests
• Review job definitions before merging
• Tag releases for production job definitions

6. Testing

• Test job definitions in non-production first
• Validate YAML syntax before committing
• Verify job creation after seed job runs

Troubleshooting

Common issues and solutions when working with JobGenie:

Jobs Not Created

Problem: Seed job runs but no jobs are created.

Solutions:

• Verify YAML file naming: must end with -jobs.yml or -jobs.yaml
• Check YAML syntax is valid
• Review seed job console output for errors
• Ensure file path matches expected structure: {org}/{project}/{env}/jobs/{file}.yml

YAML Parsing Errors

Problem: Seed job fails with YAML parsing errors.

Solutions:

• Validate YAML syntax using online validators
• Check indentation (must be spaces, not tabs)
• Verify all required fields are present
• Review console output for specific error messages

Jobs Created in Wrong Location

Problem: Jobs appear in unexpected folders.

Solutions:

• Check HOME_DIR, GROUP, and ENV in default section
• Verify PIPELINE_PATH if using custom paths
• Review job name structure (slashes create subfolders)

Parameters Not Appearing

Problem: Build parameters don’t show up in job.

Solutions:

• Verify parameter syntax is correct
• Check parameter names don’t conflict with reserved words
• Ensure PARAMETERS is a list, not a map

Environment Variables Not Available

Problem: CONFIGS values not available as environment variables.

Solutions:

• Ensure SKIP_ENJECTED_VARS is not set to true
• Check that CONFIGS values don’t contain newlines (use SCRIPT for multi-line)
• Verify job type supports environment variables

Need More Help?

For detailed troubleshooting steps and integration help, check the Getting Started Guide.

Contributing

JobGenie is part of the mCloud-Jenkins project. To contribute:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Test thoroughly
5. Submit a pull request

Support

• Maintainer: Hari Prasad (HarryTheDevOpsGuy)
• Email: HarryTheDevOpsGuy@gmail.com
• Documentation: Full Documentation
• Website: https://harrythedevopsguy.github.io/

License

Proprietary - Internal Use Only

Acknowledgments

• Jenkins Community
• JobDSL Plugin Maintainers
• All contributors and users

Documentation & Resources

📚 Documentation Pages

• Getting Started Guide - Complete integration guide for new Jenkins instances
• Configuration Reference - Detailed YAML structure and parameters
• Examples - Real-world job definition examples
• Best Practices - Recommended patterns and conventions

🔗 External Resources

• GitHub Repository: mCloud-Jenkins
• JobGenie README: Full Documentation
• Website: harrythedevopsguy.github.io

---

Need Help?

• 📖 Read the Getting Started Guide for integration instructions
• 🐛 Check the Troubleshooting section for common issues
• 📧 Contact: HarryTheDevOpsGuy@gmail.com

---

🚀 Powered by JobGenie

Transform CI/CD with YAML-driven Jenkins job automation

Visit JobGenie Homepage • Documentation • GitHub

“Soch Wahi, Approach Nai” - Same Vision, New Approach

From Zero to Hero in Minutes, Not Months 🚀

Built with ❤️ by the DevOps Team

Tags: #Jenkins #JobDSL #YAML #GitOps #CI/CD #Automation #DevOps

Related Documentation

More from Tools

Architecture Documentation - Jenkins as Code

Comprehensive architecture guide for Jenkins as Code automation framework

Best Practices Guide - Jenkins as Code

Recommended practices for using Jenkins as Code automation framework

DevOps as a Service - Automated CI/CD Management

Transforming Jenkins from a bottleneck to a self-service platform for develop...

JobGenie - Complete Job Creation Guide

Step-by-step guide to creating and configuring Jenkins jobs using JobGenie YA...

JobGenie - Complete Guide

Your friendly Jenkins job generator - Comprehensive guide to JobGenie job gen...

Quick Reference Guide - Jenkins as Code

Quick reference for common tasks and configurations in Jenkins as Code

Jenkins as Code - Complete Setup Guide

Step-by-step guide to set up Jenkins as Code with plugins, access control, an...

Jenkins as Code - Enterprise CI/CD Automation

Complete Jenkins automation framework with Infrastructure as Code, Configurat...

BG Deployer

Automated blue-green deployment for zero-downtime AWS releases

DevOps Tools & Utilities | Hari Prasad

Custom-built DevOps tools for automation, monitoring, deployment, and security

JobGenie Getting Started

Your DevOps Superpower Unleashed - Transform CI/CD with YAML-driven Jenkins j...

mCert

SSL certificate monitoring with Slack/email alerts & Telegram

mTracker

Real-time Linux user activity monitoring with Slack notifications

mWatcher

Server health monitoring for CPU, memory, disk with alerting

Sample DevOps Tool Documentation

A comprehensive guide to using our sample DevOps tool for automation and moni...

Typography Demo

Demonstration of enhanced typography features in the documentation template

Related by Tags

Architecture Documentation - Jenkins as Code

Comprehensive architecture guide for Jenkins as Code automation framework

#Jenkins #JobDSL

Best Practices Guide - Jenkins as Code

Recommended practices for using Jenkins as Code automation framework

#Jenkins #CI/CD

Related Blog Posts

JobGenie: Transform Jenkins Job Creation with Jobs as Code

Learn how to integrate JobGenie into your existing Jenkins instance to create jobs as code using ...

Dec 14, 2025

OpenResty Production Setup: Supercharge with Lua-Based Metrics and Monitoring

Complete guide to deploying production-ready OpenResty with advanced Lua-based metrics collection...

Oct 16, 2024

KEDA on EKS: Complete Guide to Event-Driven Autoscaling with Real-World Examples

Master KEDA implementation on Amazon EKS with comprehensive examples for multiple scaling scenari...

Oct 15, 2024

Related Tools & Projects

BG Deployer

Automated blue-green deployment for zero-downtime AWS releases

AWS Blue-Green

mCert

SSL certificate monitoring with Slack/email alerts & Telegram

SSL Telegram

mTracker

Real-time Linux user activity monitoring with Slack notifications

Linux Monitoring

mWatcher

Server health monitoring for CPU, memory, disk with alerting

Health Alerts

gCrypt

Git-crypt wrapper for secure file encryption & access management

Security Git-crypt

Interactive Tools

AWS VPC Designer, EKS Cost Calculator, and more utilities

Tools Calculator

External Resources

Official Documentation

AWS Documentation

Complete AWS service documentation

Community Resources

GitHub Profile

Open source projects and repositories

LinkedIn

Professional network and updates

Dev.to

Technical articles and tutorials

Contact

Get in touch for questions or collaboration

Quick Actions

All Docs

Browse all documentation

Blog

Latest articles

Tools

DevOps utilities

Portfolio

My work & projects

Found this helpful?

Help us improve this documentation by sharing your feedback or suggesting improvements.

Report Issue Start Discussion