JobGenie: Transform Jenkins Job Creation with Jobs as Code

Are you tired of manually creating Jenkins jobs through the UI? Do you want to version control your CI/CD pipeline configurations? JobGenie is the solution that transforms Jenkins job creation from manual configuration to code-driven automation using simple YAML definitions.

In this comprehensive guide, we’ll explore what JobGenie is, how to integrate it into your existing Jenkins instance, and create your first “Hello World” job using JobGenie.

What is JobGenie?

JobGenie is a powerful Jenkins job generation engine that automatically detects YAML job definition files and generates Jenkins jobs dynamically. It eliminates the need for manual job creation and provides a consistent, scalable approach to CI/CD pipeline management.

Key Features

• ✅ YAML-Based Configuration: Define jobs using simple, readable YAML files
• ✅ Auto-Discovery: Automatically finds and processes job definition files
• ✅ Zero Manual Configuration: No need to click through Jenkins UI
• ✅ Version Controlled: All job definitions stored in Git
• ✅ Consistent Structure: Same job structure across all environments
• ✅ Scalable: Generate hundreds of jobs from simple configurations
• ✅ Self-Service: Teams can create their own jobs via Git PRs

How JobGenie Works

JobGenie uses an auto-discovery engine that:

1. Scans Repository: Recursively searches for files ending with -jobs.yml or -jobs.yaml
2. Parses YAML: Uses SnakeYAML library to parse job definitions
3. Extracts Metadata: Identifies organization, project, and environment from file path
4. Generates Jobs: Uses JobDSL to create/update Jenkins jobs automatically
5. Manages Lifecycle: Deletes jobs that are removed from YAML files

The Problem It Solves

Traditional Jenkins job creation has several challenges:

• Manual Bottleneck: Every new project requires DevOps intervention
• Inconsistent Configurations: Different developers create jobs differently
• No Version Control: Job configurations exist only in Jenkins UI
• Difficult to Scale: Creating hundreds of jobs manually is time-consuming
• Error-Prone: Manual configuration leads to mistakes and inconsistencies

JobGenie solves all these problems by treating jobs as code.

Benefits of Jobs as Code

1. Version Control & Audit Trail

All job definitions are stored in Git, providing:

• Complete history of changes
• Easy rollback to previous configurations
• Full audit trail for compliance
• Code review process for job changes

2. Consistency Across Environments

# Same structure for dev, staging, and production
jobgenie:
  default:
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "my-service"
      CONFIGS:
        APP_REPO: "my-service"
        APP_BRANCH: "main"

3. Self-Service Onboarding

Development teams can create their own CI/CD pipelines:

1. Create a YAML file in the repository
2. Submit a Git PR
3. JobGenie automatically generates the job
4. No DevOps intervention required

4. Rapid Scaling

Generate hundreds of jobs in minutes instead of days:

jobgenie:
  default:
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "service-1"
      CONFIGS:
        APP_REPO: "monorepo"
    - NAME: "service-2"
      CONFIGS:
        APP_REPO: "monorepo"
    # ... 50+ more services

Integrating JobGenie into Existing Jenkins

Prerequisites

Before integrating JobGenie, ensure you have:

• ✅ Jenkins instance (version 2.300+ recommended)
• ✅ Git repository access
• ✅ Admin access to Jenkins
• ✅ Basic understanding of YAML syntax

Step 1: Install Required Plugins

JobGenie requires the following Jenkins plugins:

Core Plugins (Required)

# Essential plugins for JobGenie
plugins:
  - job-dsl                    # Job generation engine
  - configuration-as-code      # System configuration management
  - workflow-aggregator       # Pipeline support
  - git                       # Git integration
  - credentials-binding       # Secure credential management
  - role-strategy             # Role-based access control (optional but recommended)

Installation Methods

Via Jenkins UI:

1. Navigate to Manage Jenkins → Manage Plugins → Available
2. Search for each plugin
3. Select and install: job-dsl, configuration-as-code, workflow-aggregator, git
4. Restart Jenkins if prompted

Via Configuration as Code:

# jenkins.yaml
jenkins:
  plugins:
    - job-dsl
    - configuration-as-code
    - workflow-aggregator
    - git

Step 2: Set Up Repository Structure

Create the following directory structure in your Git repository:

JobGenie-Pipelines/
├── {organization}/
│   ├── {project}/
│   │   ├── {environment}/
│   │   │   └── jobs/
│   │   │       └── {project}-{environment}-jobs.yml

Example Structure:

JobGenie-Pipelines/
├── amazon/
│   ├── mcloud/
│   │   ├── prod/
│   │   │   └── jobs/
│   │   │       └── mcloud-prod-jobs.yml
│   │   └── nonprod/
│   │       └── jobs/
│   │           └── mcloud-nonprod-jobs.yml
└── global/
    └── common/
        └── prod/
            └── jobs/
                └── common-prod-jobs.yml

Step 3: Configure Jenkins Credentials

Set up Git credentials in Jenkins:

1. Navigate to Manage Jenkins → Credentials → System → Global credentials
2. Add SSH credentials:
   • Kind: SSH Username with private key
   • ID: jenkins_repo_key (or your preferred ID)
   • Username: git
   • Private Key: Upload your SSH private key

Step 4: Create Seed Job

The seed job is a Pipeline job that uses the JobGenie/Jenkinsfile from your repository. This Jenkinsfile automatically checks out the repository, loads shared libraries, and executes the JobGenie engine.

Create Seed Job via Jenkins UI

1. Navigate to Jenkins Dashboard
2. Click “New Item”
3. Enter Item Name: 0-JobGenie-Generator (or your preferred name)
4. Select: Pipeline (not Freestyle)
5. Click OK

Configure Seed Job

General Settings:

• Description:

  JobGenie Seed Job - Automatically generates Jenkins jobs from YAML definitions.
  Scans repository for files matching *-jobs.yml pattern and creates jobs dynamically.
  

Pipeline Configuration:

• Definition: Pipeline script from SCM
• SCM: Git
• Repository URL: Your JobGenie-Pipelines repository URL
  • Example: git@github.com:YOUR_USERNAME/JobGenie-Pipelines.git
  • Or: https://github.com/YOUR_USERNAME/JobGenie-Pipelines.git
• Credentials: Select your SSH credential (e.g., jenkins_repo_key)
• Branches to build: */master (or your default branch: */main)
• Script Path: JobGenie/Jenkinsfile
• Lightweight checkout: Unchecked (we need full workspace)

Build Triggers (Optional):

• Poll SCM: H/5 * * * * (every 5 minutes)
  • This will automatically check for changes and regenerate jobs
• Or trigger manually when needed

Step 5: Update JobGenie Configuration (Optional)

If you need to customize the JobGenie engine, edit JobGenie/pipelines/jobGenie.groovy:

// Update these variables in jobGenie.groovy
defaultConfigs += [
  CONFIG_REPO: "https://github.com/YOUR_USERNAME/JobGenie-Pipelines.git",
  CONFIG_BRANCH: "master",  // or "main"
  JENKINS_GIT_KEY: 'jenkins_repo_key',  // Your credential ID
  GROUP: "v2",
  ENV: "stage",
  JENKINSFILE_DIR: "default",
]

Step 6: Run Seed Job

1. Navigate to the seed job: 0-JobGenie-Generator
2. Click Build Now
3. Check console output for any errors
4. Verify jobs are created in Jenkins

Creating Your First Hello World Job

Now let’s create a simple “Hello World” job using JobGenie.

Step 1: Create Job Definition File

Create the following file structure:

JobGenie-Pipelines/
└── demo/
    └── hello-world/
        └── dev/
            └── jobs/
                └── hello-world-dev-jobs.yml

Step 2: Write Hello World YAML

Edit hello-world-dev-jobs.yml:

jobgenie:
  default:
    HOME_DIR: nonprod
    GROUP: "v1"
    ENV: "dev"
  jobs:
    - NAME: "hello-world"
      PARAMETERS:
        - { name: 'GitBranch', string: 'main', description: 'Git branch to build.' }
        - { name: 'Message', string: 'Hello from JobGenie!', description: 'Custom message to display.' }
      CONFIGS:
        APP_REPO: "hello-world-app"
        APP_BRANCH: "main"
        SERVICE: "hello-world"

Step 3: Enhanced Hello World with Pipeline

For a more complete example, let’s create a job that actually runs a pipeline:

jobgenie:
  default:
    HOME_DIR: nonprod
    GROUP: "v1"
    ENV: "dev"
  jobs:
    - NAME: "hello-world"
      PARAMETERS:
        - { name: 'GitBranch', string: 'main', description: 'Git branch to build.' }
        - { name: 'Message', string: 'Hello from JobGenie!', description: 'Custom message to display.' }
      CONFIGS:
        APP_REPO: "hello-world-app"
        APP_BRANCH: "main"
        SERVICE: "hello-world"
        DOCKER_BUILD_ARGS: "ENV"

Step 4: Understanding Generated Jobs

When JobGenie processes your YAML file, it generates Jenkins jobs using the shared library templates. The actual pipeline scripts are loaded from:

• Jenkinsfile Location: {organization}/{project}/{environment}/jenkinsfiles/{JENKINSFILE_DIR}/Jenkinsfile
• Default Template: Uses the default Jenkinsfile template
• Custom Templates: Specify JENKINSFILE_DIR in your YAML to use different templates

The generated jobs will:

1. Checkout your application repository (APP_REPO)
2. Use the branch specified in APP_BRANCH or build parameter
3. Execute the pipeline defined in the Jenkinsfile template
4. Support all parameters defined in your YAML file

Note: The actual pipeline execution logic is defined in the Jenkinsfile templates in your repository, not in the JobGenie engine itself. This allows you to customize pipelines per project/environment.

Step 5: Commit and Push

# Navigate to your repository
cd JobGenie-Pipelines

# Add your job definition file
git add amazon/demo/nonprod/jobs/demo-nonprod-jobs.yml

# Commit
git commit -m "Add Hello World job using JobGenie"

# Push
git push origin master  # or 'main' depending on your default branch

Step 6: Run Seed Job

1. Navigate to 0-JobGenie-Generator
2. Click Build Now
3. Wait for build to complete
4. Check console output

Step 7: Verify Job Creation

1. Navigate to Jenkins dashboard
2. Look for jobs created under the folder structure:
   • amazon/demo/nonprod/deploy/v1/dev/hello-world
   • Or check the console output of the seed job for exact paths
3. Click on the generated job
4. Click Build with Parameters (if parameters are defined)
5. Set parameters:
   • GitBranch: main
   • Message: Hello from JobGenie! (if defined)
6. Click Build
7. Watch the pipeline execute!

Note: The exact job path depends on:

• Organization: amazon
• Project: demo
• Environment: nonprod
• GROUP: v1 (from default section)
• ENV: dev (from default section)
• Job Name: hello-world

Final path: amazon/demo/nonprod/deploy/v1/dev/hello-world

Expected Output

When you run the hello-world job, you should see:

==========================================
  Hello from JobGenie!
==========================================
Service: hello-world
Repository: hello-world-app
Branch: main
Environment: dev
Build Number: 1
==========================================

Understanding the YAML Structure

Basic Structure

jobgenie:
  default:                    # Default configurations (applied to all jobs)
    HOME_DIR: nonprod         # Home directory (prod/nonprod)
    GROUP: "v1"              # Version group
    ENV: "dev"               # Environment name
  jobs:                       # List of job definitions
    - NAME: "job-name"       # Job name (required)
      PARAMETERS: []          # Build parameters (optional)
      CONFIGS: {}             # Job configurations (required)

Configuration Hierarchy

Configurations are merged in this order (later overrides earlier):

1. default section: Default values for all jobs in the file
2. CONFIGS section: Individual job-specific configurations
3. Environment variables: Injected from build parameters

Required Fields

• NAME: Job name (required)
• CONFIGS: Job configuration map (required)
  • APP_REPO: Git repository name (required)
  • APP_BRANCH: Git branch name (required)

Optional Fields

• PARAMETERS: List of build parameters
• HOME_DIR: Home directory (prod/nonprod)
• GROUP: Version group (e.g., “v1”, “v2”, “v4”)
• ENV: Environment name (dev, stage, prod)
• SERVICE: Service name
• DOCKER_BUILD_ARGS: Docker build arguments

Advanced Examples

Multiple Jobs in One File

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "user-api"
      CONFIGS:
        APP_REPO: "user-service"
        APP_BRANCH: "production"
    - NAME: "payment-api"
      CONFIGS:
        APP_REPO: "payment-service"
        APP_BRANCH: "production"
    - NAME: "order-api"
      CONFIGS:
        APP_REPO: "order-service"
        APP_BRANCH: "production"

Jobs with Parameters

jobgenie:
  default:
    HOME_DIR: nonprod
    GROUP: "v4"
    ENV: "stage"
  jobs:
    - NAME: "deploy-service"
      PARAMETERS:
        - { name: 'GitBranch', string: 'develop', description: 'Application branch.' }
        - { name: 'Environment', choices: ['stage', 'qa'], description: 'Target environment.' }
        - { name: 'SkipTests', bool: false, description: 'Skip test execution.' }
      CONFIGS:
        APP_REPO: "my-service"
        APP_BRANCH: "develop"
        SERVICE: "deploy-service"

Environment-Specific Configuration

# File: demo/myapp/dev/jobs/myapp-dev-jobs.yml
jobgenie:
  default:
    HOME_DIR: nonprod
    GROUP: "v1"
    ENV: "dev"
  jobs:
    - NAME: "my-service"
      CONFIGS:
        APP_REPO: "my-service"
        APP_BRANCH: "develop"

# File: demo/myapp/prod/jobs/myapp-prod-jobs.yml
jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "my-service"
      CONFIGS:
        APP_REPO: "my-service"
        APP_BRANCH: "production"

Best Practices

1. Organize by Environment

Keep separate files for each environment:

amazon/
└── mcloud/
    ├── prod/
    │   └── jobs/
    │       └── mcloud-prod-jobs.yml
    └── nonprod/
        └── jobs/
            └── mcloud-nonprod-jobs.yml

2. Use Consistent Naming

# ✅ Good
- NAME: "user-service"
- NAME: "payment-service"

# ❌ Bad
- NAME: "UserService"      # PascalCase
- NAME: "user_service"     # snake_case
- NAME: "userService"      # camelCase

3. Group Related Jobs

Put related services in the same file:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    # All payment-related services together
    - NAME: "payment-api"
    - NAME: "payment-processor"
    - NAME: "payment-gateway"

4. Document Custom Variables

jobgenie:
  default:
    # HOME_DIR: prod/nonprod - determines deployment home
    # GROUP: version group (v1, v2, v4)
    # ENV: environment name (dev, stage, prod)
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"

5. Version Control Everything

• Commit all YAML files to Git
• Use pull requests for job changes
• Review job definitions before merging
• Tag releases for job configurations

Troubleshooting

Jobs Not Generated

Problem: Seed job runs but no jobs appear.

Solutions:

1. Check seed job console output for errors
2. Verify YAML syntax is correct (use online YAML validator)
3. Check file path matches convention: {org}/{project}/{env}/jobs/{project}-{env}-jobs.yml
4. Ensure file name ends with -jobs.yml or -jobs.yaml
5. Verify shared libraries repository is accessible (mCloud-Jenkins)
6. Check that createjob: true in jobGenie.groovy globalConfigs
7. Verify Git credentials have access to both repositories

YAML Parsing Errors

Problem: YAML syntax errors.

Solutions:

1. Validate YAML syntax using online validator
2. Check indentation (use spaces, not tabs)
3. Ensure proper structure: jobgenie.default and jobgenie.jobs
4. Verify all required fields are present

Jobs Created in Wrong Location

Problem: Jobs appear in unexpected folders.

Solutions:

1. Verify YAML file path structure: {org}/{project}/{env}/jobs/{project}-{env}-jobs.yml
2. Check default section has correct GROUP and ENV values
3. Job location follows: {org}/{project}/{env}/deploy/{GROUP}/{ENV}/{JOB_NAME}
4. Review seed job console output to see detected paths
5. Ensure HOME_DIR matches environment (prod/nonprod)

Parameters Not Showing

Problem: Build parameters don’t appear in job.

Solutions:

1. Verify PARAMETERS array structure in YAML
2. Check parameter types: string, choices, bool
3. Ensure correct syntax: { name: 'ParamName', string: 'default', description: '...' }
4. For choices: use choices: ['option1', 'option2'] (not choice)
5. Re-run seed job after fixing YAML
6. Check that Jenkinsfile template supports parameters

Seed Job Fails to Run

Problem: Seed job fails immediately or can’t checkout repository.

Solutions:

1. Verify Git repository URL is correct in seed job configuration
2. Check SSH credentials (jenkins_repo_key) are valid and have repository access
3. Ensure JobGenie/Jenkinsfile exists in repository root
4. Verify required plugins are installed: job-dsl, workflow-aggregator, git
5. Check Jenkins has access to shared libraries repository (mCloud-Jenkins)
6. Review console output for specific error messages
7. Verify branch name matches your repository (master vs main)
8. Ensure Jenkins node has label linux-slave (or update Jenkinsfile label)

Shared Libraries Access Issues

Problem: Seed job fails with “Class not found” or shared library errors.

Solutions:

1. Verify mCloud-Jenkins repository is accessible
2. Check credentials have access to shared libraries repository
3. Contact HarryTheDevOpsGuy@gmail.com for access if needed
4. Verify additionalClasspath: 'mCloud-Jenkins/sharedlibs/src' path is correct
5. Check that CommonUtils class exists in shared libraries

Real-World Use Cases

Microservices Architecture

Generate jobs for 50+ microservices:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "api-gateway"
    - NAME: "user-service"
    - NAME: "order-service"
    - NAME: "payment-service"
    # ... 46 more services

Multi-Environment Deployment

Same service, different environments:

# dev/jobs/myapp-dev-jobs.yml
jobgenie:
  default:
    ENV: "dev"
    APP_BRANCH: "develop"

# prod/jobs/myapp-prod-jobs.yml
jobgenie:
  default:
    ENV: "prod"
    APP_BRANCH: "production"

Custom Pipeline Templates

Use different templates for different job types:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v4"
    ENV: "prod"
  jobs:
    - NAME: "data-pipeline"
      CONFIGS:
        SERVICE: "data-pipeline"
        CICD_TEMPLATE_NAME: "data-processing"
        APP_REPO: "data-pipeline-app"

Conclusion

JobGenie transforms Jenkins job management from a manual, error-prone process into a code-driven, scalable solution. By integrating JobGenie into your existing Jenkins instance, you can:

• ✅ Reduce onboarding time from days to hours
• ✅ Eliminate manual configuration errors
• ✅ Enable self-service job creation for teams
• ✅ Maintain consistency across all environments
• ✅ Scale effortlessly to hundreds of jobs

Next Steps

1. Set up JobGenie in your Jenkins instance
2. Create your first job using the Hello World example
3. Migrate existing jobs to YAML definitions
4. Train your team on JobGenie usage
5. Establish best practices for your organization

Additional Resources

• JobGenie Complete Guide - Comprehensive job creation guide
• Jenkins as Code Setup - Complete setup instructions
• Architecture Documentation - System architecture overview
• Best Practices - Recommended patterns

Get Help

Need professional assistance with JobGenie integration? Contact for:

• Jenkins as Code implementation
• JobGenie integration and customization
• Team training and knowledge transfer
• CI/CD pipeline development

Happy Automating! 🚀