Home Portfolio Blog Courses

JobGenie - Complete Guide

Your friendly Jenkins job generator - Comprehensive guide to JobGenie job generation engine

Updated Jan 15, 2024
10 min read
Intermediate
tools

JobGenie Logo

JobGenie - Technical Reference Guide

Your friendly Jenkins job generator - Complete technical reference for JobGenie job generation engine

Powered by JobGenie - Your DevOps Superpower Unleashed

Visit JobGenie Homepage β€’ Examples

πŸ“š Related Guides:

πŸ“‹ Table of Contents

🎯 Overview

JobGenie is a powerful Jenkins job generation engine that automatically detects YAML job definition files and generates Jenkins jobs. It eliminates the need for manual job creation and provides a consistent, scalable approach to CI/CD pipeline management.

πŸ’‘ Key Philosophy: β€œSoch Wahi, Approach Nai” - Same Vision, New Approach

No complex logic needed! Just define jobs in simple YAML files, and JobGenie automatically creates them in Jenkins.

How JobGenie Works

JobGenie uses an auto-discovery engine that:

  1. Scans Repository: Recursively searches for files ending with -jobs.yml or -jobs.yaml
  2. Parses YAML: Uses SnakeYAML library to parse job definitions
  3. Extracts Metadata: Identifies organization, project, and environment from file path
  4. Generates Jobs: Uses JobDSL to create/update Jenkins jobs automatically
  5. Manages Lifecycle: Deletes jobs that are removed from YAML files

File Path Convention

Job definition files must follow this naming pattern:

{organization}/{project}/{environment}/jobs/{project}-{environment}-jobs.yml

Examples:

  • amazon/mcloud/prod/jobs/mcloud-prod-jobs.yml
  • amazon/mcloud/nonprod/jobs/mcloud-nonprod-jobs.yml

Quick Start

New to JobGenie? Start with the Job Creation Guide for step-by-step instructions on creating your first job.

Need setup help? See the Setup Guide for Jenkins installation and JobGenie integration.

Key Benefits

  • βœ… Zero Manual Configuration: Define jobs in code, not UI
  • βœ… Consistency: Same structure across all jobs
  • βœ… Scalability: Generate hundreds of jobs from simple configs
  • βœ… Maintainability: Version-controlled job definitions
  • βœ… Flexibility: Support for various job types and patterns

🧠 Core Concepts

1. YAML Structure

JobGenie uses YAML files with a specific structure:

jobgenie:
  default:                    # Default configurations (applied to all jobs)
    HOME_DIR: prod            # Home directory (prod/nonprod)
    GROUP: "v4"               # Version group
    ENV: "prod"              # Environment name
  jobs:                       # List of job definitions
    - NAME: "job-name"        # Job name (required)
      PARAMETERS: []          # Build parameters (optional)
      CONFIGS: {}             # Job configurations (required)

2. Configuration Hierarchy

Configurations are merged in this order (later overrides earlier):

  1. default section: Global defaults for all jobs in the file
  2. CONFIGS section: Individual job-specific configurations
  3. Environment variables: Injected from build parameters

3. Variable Resolution

Variables are resolved using this precedence:

  • Job-specific CONFIGS (highest priority)
  • Default section values
  • Environment variables from parameters

πŸ“ Configuration Structure

Basic YAML Structure

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "user-authentication-service"
      PARAMETERS:
        - { name: 'GitBranch', string: 'production', description: 'Production application git branch.' }
        - { name: 'DeployVersion', string: 'latest', description: 'Application version to deploy.' }
      CONFIGS:
        APP_REPO: "user-auth-service"
        APP_BRANCH: "production"
        DOCKER_BUILD_ARGS: "ENV,SERVICE"
        SSH_KEYS: "default:/opt/jenkins/keys/prod_key_rsa"

Required CONFIGS

For Standard Pipeline Jobs

CONFIGS:
  APP_REPO: "repository-name"           # Git repository name
  APP_BRANCH: "branch-name"            # Git branch
  # Optional: DOCKER_BUILD_ARGS, SSH_KEYS, etc.

For Freestyle Jobs

CONFIGS:
  JOB_TYPE: "freestyle"                # Required for freestyle jobs
  SERVICE: "service-name"              # Service identifier
  APP_REPO: "repository-name"         # Git repository
  APP_BRANCH: "branch-name"           # Git branch
  SCRIPT: |                           # Shell script to execute
    echo "Your script here"

For Seed Jobs

CONFIGS:
  SEED_JOB: true                      # Marks this as a seed job
  CONFIG_BRANCH: "${GitBranch}"       # Config repository branch
  JENKINSFILE: "JobGenie/Jenkinsfile" # Jenkinsfile path

Common CONFIGS Options

Docker Build Configuration

CONFIGS:
  DOCKER_BUILD_ARGS: "ENV,SERVICE"     # Docker build arguments (comma-separated)
  DOCKERFILE_PATH: "services/order/Dockerfile"  # Custom Dockerfile path

Git Configuration

CONFIGS:
  SSH_KEYS: "default:/opt/jenkins/keys/prod_key_rsa"  # SSH key path
  CONFIG_REPO: "git@github.com:User/Repo.git"         # Config repository
  CONFIG_BRANCH: "${GitBranch}"                       # Config branch

Job Behavior

CONFIGS:
  SKIP_GIT: true                      # Skip Git SCM checkout
  SKIP_ENJECTED_VARS: true           # Skip environment variable injection
  DISABLED: true                      # Create job in disabled state
  PIPELINE_PATH: "shared"             # Custom job path
  JENKINSFILE_DIR: "default"          # Jenkinsfile directory

Script Execution (Freestyle)

CONFIGS:
  SCRIPT: |                           # Multi-line shell script
    echo "Running migrations..."
    # Your script logic here
    echo "Completed"

DSL Script Execution

CONFIGS:
  DSL_SCRIPT: |                       # Groovy DSL script
    import io.jenkins.plugins.casc.ConfigurationAsCode;
    ConfigurationAsCode.get().configure()

🎨 Job Types

1. Pipeline Jobs (Default)

Purpose: Standard Jenkins pipeline jobs using Jenkinsfiles

Configuration:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "user-authentication-service"
      PARAMETERS:
        - { name: 'GitBranch', string: 'production', description: 'Production application git branch.' }
      CONFIGS:
        APP_REPO: "user-auth-service"
        APP_BRANCH: "production"
        DOCKER_BUILD_ARGS: "ENV,SERVICE"
        SSH_KEYS: "default:/opt/jenkins/keys/prod_key_rsa"

Generated Job Location:

  • {organization}/{project}/{environment}/deploy/{GROUP}/{ENV}/{JOB_NAME}
  • Example: amazon/mcloud/prod/deploy/v4/prod/user-authentication-service

2. Freestyle Jobs

Purpose: Execute shell scripts directly without Jenkinsfile

Configuration:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "database-migration-runner"
      PARAMETERS:
        - { name: 'GitBranch', string: 'master', description: 'Migration scripts git branch.' }
      CONFIGS:
        JOB_TYPE: "freestyle"
        SERVICE: "database-migration-runner"
        APP_REPO: "database-migrations"
        APP_BRANCH: "master"
        SCRIPT: |
          echo "Running database migrations..."
          # Your migration logic here

Key Difference: Set JOB_TYPE: "freestyle" and provide SCRIPT instead of using Jenkinsfile

3. Seed Jobs

Purpose: Generate other jobs (JobGenie seed jobs)

Configuration:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "0-poc-DevOps"
      PARAMETERS:
        - { name: 'GitBranch', string: 'master', description: 'Dev application git branch.' }
      CONFIGS:
        SEED_JOB: true
        CONFIG_BRANCH: "${GitBranch}"
        JENKINSFILE: "JobGenie/Jenkinsfile"

Key Difference: Set SEED_JOB: true to mark as seed job

4. Custom Template Jobs

Purpose: Jobs using custom pipeline templates

Configuration:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "microservices-orchestrator"
      PARAMETERS:
        - { name: 'TAG', string: 'v3.2.1', description: 'Application version tag.' }
      CONFIGS:
        SERVICE: "microservices-orchestrator"
        CICD_TEMPLATE_NAME: "microservices-deployment-template"
        APP_REPO: "microservices-platform"
        APP_BRANCH: "production"

Key Difference: Use CICD_TEMPLATE_NAME to specify custom template

πŸš€ Advanced Features

1. Custom Dockerfile Path

Specify custom Dockerfile location:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "order-processing-api"
      CONFIGS:
        APP_REPO: "ecommerce-order-api"
        APP_BRANCH: "main"
        DOCKERFILE_PATH: "services/order/Dockerfile"
        DOCKER_BUILD_ARGS: "ENV"

2. Custom Pipeline Path

Create jobs in custom folder structure:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "notification-service"
      CONFIGS:
        APP_REPO: "notification-engine"
        APP_BRANCH: "master"
        PIPELINE_PATH: "shared"

Result: Job created at {organization}/{project}/{environment}/shared/{JOB_NAME}

3. Disabled Jobs

Create jobs in disabled state:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "user-authentication-service"
      CONFIGS:
        APP_REPO: "user-auth-service"
        APP_BRANCH: "production"
        DISABLED: true  # Job created but disabled

4. Jobs Without Git Checkout

Skip Git SCM checkout for jobs that don’t need source code:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "jenkins-configuration-sync"
      CONFIGS:
        JOB_TYPE: "freestyle"
        SKIP_GIT: true
        SKIP_ENJECTED_VARS: true
        DSL_SCRIPT: |
          import io.jenkins.plugins.casc.ConfigurationAsCode;
          ConfigurationAsCode.get().configure()

5. DSL Script Execution

Execute Groovy DSL scripts directly:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "jenkins-configuration-sync"
      CONFIGS:
        JOB_TYPE: "freestyle"
        SKIP_GIT: true
        SKIP_ENJECTED_VARS: true
        DSL_SCRIPT: |
          import io.jenkins.plugins.casc.ConfigurationAsCode;
          import jenkins.model.Jenkins;
          ConfigurationAsCode.get().configure()
          println("Configuration sync completed")

πŸ“š Examples

Example 1: Simple Pipeline Job

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "user-authentication-service"
      PARAMETERS:
        - { name: 'GitBranch', string: 'production', description: 'Production application git branch.' }
        - { name: 'DeployVersion', string: 'latest', description: 'Application version to deploy.' }
      CONFIGS:
        APP_REPO: "user-auth-service"
        APP_BRANCH: "production"
        DOCKER_BUILD_ARGS: "ENV,SERVICE"
        SSH_KEYS: "default:/opt/jenkins/keys/prod_key_rsa"

Generated Job Location:

  • amazon/mcloud/prod/deploy/v4/prod/user-authentication-service

Example 2: Freestyle Job with Script

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "database-migration-runner"
      PARAMETERS:
        - { name: 'GitBranch', string: 'master', description: 'Migration scripts git branch.' }
        - { name: 'MigrationVersion', string: '', description: 'Specific migration version to run.' }
        - { name: 'DryRun', bool: false, description: 'Perform dry run without applying changes.' }
      CONFIGS:
        JOB_TYPE: "freestyle"
        SERVICE: "database-migration-runner"
        APP_REPO: "database-migrations"
        APP_BRANCH: "master"
        SCRIPT: |
          echo "Running database migrations..."
          # Migration logic here
          echo "Migrations completed successfully"

Example 3: Seed Job

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "0-poc-DevOps"
      PARAMETERS:
        - { name: 'GitBranch', string: 'master', description: 'Dev application git branch.' }
      CONFIGS:
        SEED_JOB: true
        CONFIG_BRANCH: "${GitBranch}"
        JENKINSFILE: "JobGenie/Jenkinsfile"

Example 4: Job with Custom Dockerfile

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "order-processing-api"
      PARAMETERS:
        - { name: 'GitBranch', string: 'main', description: 'Application git branch.' }
      CONFIGS:
        APP_REPO: "ecommerce-order-api"
        APP_BRANCH: "main"
        DOCKER_BUILD_ARGS: "ENV"
        DOCKERFILE_PATH: "services/order/Dockerfile"
        SSH_KEYS: "default:/opt/jenkins/keys/prod_key_rsa"

Example 5: Job with Choice Parameters

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "inventory-management-service"
      PARAMETERS:
        - { name: 'GitBranch', string: 'release/v2.0', description: 'Application git branch.' }
        - { name: 'Environment', choices: ['prod', 'prod-dr'], description: 'Target deployment environment.' }
      CONFIGS:
        APP_REPO: "inventory-service"
        APP_BRANCH: "release/v2.0"
        DOCKER_BUILD_ARGS: "ENV,Environment"
        SSH_KEYS: "default:/opt/jenkins/keys/prod_key_rsa"

Example 6: Infrastructure Job (Terraform)

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "terraform-infrastructure-deploy"
      PARAMETERS:
        - { name: 'GitBranch', string: 'main', description: 'Terraform code git branch.' }
        - { name: 'TerraformAction', choices: ['plan', 'apply', 'destroy'], description: 'Terraform action to execute.' }
        - { name: 'Region', string: 'us-east-1', description: 'AWS region for deployment.' }
      CONFIGS:
        JOB_TYPE: "freestyle"
        SERVICE: "terraform-infrastructure-deploy"
        APP_REPO: "terraform-infrastructure"
        APP_BRANCH: "main"
        SCRIPT: |
          echo "Executing Terraform ${TerraformAction} in ${Region}"
          terraform init
          terraform ${TerraformAction}

Example 7: Security Scanning Job

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "security-vulnerability-scan"
      PARAMETERS:
        - { name: 'ScanType', choices: ['container', 'code', 'dependency'], description: 'Type of security scan to perform.' }
        - { name: 'Severity', choices: ['critical', 'high', 'medium', 'all'], description: 'Minimum severity level to report.' }
      CONFIGS:
        JOB_TYPE: "freestyle"
        SERVICE: "security-vulnerability-scan"
        APP_REPO: "security-scanner"
        APP_BRANCH: "main"
        SCRIPT: |
          echo "Running ${ScanType} security scan with ${Severity} severity"
          # Security scanning logic here

Example 8: Custom Template Job

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "microservices-orchestrator"
      PARAMETERS:
        - { name: 'TAG', string: 'v3.2.1', description: 'Application version tag.' }
        - { name: 'Services', string: 'all', description: 'Comma-separated list of services to deploy (or "all").' }
        - { name: 'RollbackOnFailure', bool: true, description: 'Automatically rollback on deployment failure.' }
      CONFIGS:
        SERVICE: "microservices-orchestrator"
        CICD_TEMPLATE_NAME: "microservices-deployment-template"
        APP_REPO: "microservices-platform"
        APP_BRANCH: "production"
    [
        APP_REPO: 'complex-app',
        APP_BRANCH: 'release/v2.0',
        appNames: ['api', 'worker', 'scheduler'],
        BUILD: [
            NAME: 'complex-app'
        ],
        DOCKER_BUILD_ARGS: 'ENV,VERSION'
    ]
] ] ```

Generated:

  • Build job: stage-complex-app-build
  • Deploy jobs: stage-api-deploy, stage-worker-deploy, stage-scheduler-deploy

Example 9: Compliance Audit Job

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "compliance-audit"
      PARAMETERS:
        - { name: 'AuditType', choices: ['SOC2', 'PCI-DSS', 'GDPR', 'HIPAA'], description: 'Compliance standard to audit.' }
        - { name: 'GenerateReport', bool: true, description: 'Generate compliance audit report.' }
      CONFIGS:
        JOB_TYPE: "freestyle"
        SERVICE: "compliance-audit"
        APP_REPO: "compliance-tools"
        APP_BRANCH: "master"
        SCRIPT: |
          echo "Running ${AuditType} compliance audit"
          # Compliance audit logic here

Example 10: Blue-Green Deployment Manager

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "blue-green-deployment-manager"
      PARAMETERS:
        - { name: 'ApplicationName', string: '', description: 'Application name for blue-green deployment.' }
        - { name: 'TrafficPercentage', string: '10', description: 'Percentage of traffic to route to new version initially.' }
        - { name: 'HealthCheckEndpoint', string: '/health', description: 'Health check endpoint path.' }
      CONFIGS:
        SERVICE: "blue-green-deployment-manager"
        CICD_TEMPLATE_NAME: "blue-green-deploy-template"
        APP_REPO: "deployment-automation"
        APP_BRANCH: "main"

βœ… Best Practices

1. YAML Organization

βœ… DO:

# Group related jobs together in one file
jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "service-1"
      CONFIGS:
        APP_REPO: "service-1"
    - NAME: "service-2"
      CONFIGS:
        APP_REPO: "service-2"

❌ DON’T:

# Don't create separate files for each job
# Instead, group related jobs in one file per environment

2. Naming Conventions

βœ… DO:

  • Use descriptive job names: user-authentication-service not user-svc
  • Follow kebab-case for job names
  • Use consistent naming across environments

❌ DON’T:

  • Use ambiguous names
  • Mix naming styles
  • Create unnecessarily long names

3. Default Configuration

βœ… DO:

# Set sensible defaults in default section
jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "my-service"
      CONFIGS:
        APP_REPO: "my-service"
        # ENV and GROUP inherited from default

❌ DON’T:

// Don't hardcode environment-specific values in defaults
def defaultConfigs = [
    ENV: 'prod'  // Wrong - should be per-project
]

4. Job Parameters

βœ… DO:

  • Provide clear descriptions
  • Use appropriate parameter types
  • Set sensible defaults

❌ DON’T:

  • Create too many parameters
  • Use unclear descriptions
  • Mix parameter types unnecessarily

5. Version Control

βœ… DO:

  • Commit all JobGenie changes
  • Use feature branches
  • Review changes before merging

❌ DON’T:

  • Make changes directly in production
  • Skip code review
  • Delete old configurations without migration

πŸ”§ Troubleshooting

Issue 1: Jobs Not Generated

Symptoms: Seed job runs but no jobs appear

Solutions:

  1. Check createjob flag: globalConfigs.createjob = true
  2. Verify JobGenie map structure
  3. Check Jenkins console logs
  4. Verify file path in seed job

Issue 2: Wrong Job Path

Symptoms: Jobs created in unexpected location

Solutions:

  1. Check DEPLOY_HOME variable
  2. Verify JOB_SUFFIX_MKDIR calculation
  3. Review folder creation logic

Issue 3: Missing Parameters

Symptoms: Expected parameters not appearing

Solutions:

  1. Verify JOB_PARAMS structure
  2. Check parameter type (string, choices, bool)
  3. Review pipelineGenerator function

Issue 4: Template Not Found

Symptoms: Jenkinsfile not found error

Solutions:

  1. Verify DEFAULT_JENKINSFILE path
  2. Check CONFIG_REPO and CONFIG_BRANCH
  3. Ensure Jenkinsfile exists in repository

Issue 5: Variable Resolution Issues

Symptoms: Variables not resolving correctly

Solutions:

  1. Check variable precedence
  2. Verify variable names (case-sensitive)
  3. Review configuration merging logic

πŸ“– Reference

Variable Reference

Variable Type Required Description
APP_REPO String Yes* Git repository name
APP_BRANCH String Yes* Git branch
appNames List Yes* List of service names
SERVICE String Yes** Service identifier
CICD_TEMPLATE_NAME String No Pipeline template name
DOCKER_BUILD_ARGS String No Docker build arguments
SSH_KEYS String No SSH key path
JOB_PARAMS List No Custom job parameters

*Required for standard application jobs **Required for custom jobs

Parameter Types

// String parameter
[ name: 'BRANCH', string: 'master', description: 'Git branch' ]

// Choice parameter
[ name: 'ENV', choices: ['dev', 'prod'], description: 'Environment' ]

// Boolean parameter
[ name: 'DRY_RUN', bool: false, description: 'Dry run mode' ]

// Dynamic choice parameter
[ name: 'ImageTag', choiceType: 'SINGLE_SELECT', arChoiceGroovy: '...' ]

πŸ“š Additional Resources

🧭 Navigation

Related Topics:


JobGenie Logo

πŸš€ Powered by JobGenie

Transform CI/CD with YAML-driven Jenkins job automation

Visit JobGenie Homepage β€’ Documentation β€’ Examples

β€œSoch Wahi, Approach Nai” - Same Vision, New Approach

Maintained by the DevOps Team

Last Updated: January 15, 2024
Version: 2.0.0

Related Documentation

More from Tools

Architecture Documentation - Jenkins as Code

Comprehensive architecture guide for Jenkins as Code automation framework

Best Practices Guide - Jenkins as Code

Recommended practices for using Jenkins as Code automation framework

DevOps as a Service - Automated CI/CD Management

Transforming Jenkins from a bottleneck to a self-service platform for develop...

JobGenie - Complete Job Creation Guide

Step-by-step guide to creating and configuring Jenkins jobs using JobGenie YA...

Quick Reference Guide - Jenkins as Code

Quick reference for common tasks and configurations in Jenkins as Code

Jenkins as Code - Complete Setup Guide

Step-by-step guide to set up Jenkins as Code with plugins, access control, an...

Jenkins as Code - Enterprise CI/CD Automation

Complete Jenkins automation framework with Infrastructure as Code, Configurat...

BG Deployer

Automated blue-green deployment for zero-downtime AWS releases

DevOps Tools & Utilities | Hari Prasad

Custom-built DevOps tools for automation, monitoring, deployment, and security

JobGenie Getting Started

Your DevOps Superpower Unleashed - Transform CI/CD with YAML-driven Jenkins j...

JobGenie

Your DevOps Superpower Unleashed - Transform CI/CD with YAML-driven Jenkins j...

mCert

SSL certificate monitoring with Slack/email alerts & Telegram

mTracker

Real-time Linux user activity monitoring with Slack notifications

mWatcher

Server health monitoring for CPU, memory, disk with alerting

Sample DevOps Tool Documentation

A comprehensive guide to using our sample DevOps tool for automation and moni...

Typography Demo

Demonstration of enhanced typography features in the documentation template

Found this helpful?

Help us improve this documentation by sharing your feedback or suggesting improvements.