JobGenie Getting Started

Your DevOps Superpower Unleashed - Transform CI/CD with YAML-driven Jenkins job automation

Updated Jan 15, 2025

15 min read

Intermediate

tools

JobGenie Getting Started Guide

“Soch Wahi, Approach Nai” - Redefining CI/CD Automation

Welcome to JobGenie! This guide will walk you through integrating JobGenie into a new Jenkins instance and getting your first YAML-driven jobs up and running.

Overview

JobGenie is a robust solution for managing Jenkins as code, transforming complex CI/CD pipeline creation into simple YAML definitions. It combines powerful tools and utilities to deliver production-grade automation from zero to deployment.

Core Components

JobGenie leverages two main components:

Ansible: Automates Jenkins installation and configuration according to project needs. The Ansible solution is fully flexible and can configure any aspect of a Linux server running Jenkins.
Jenkins Plugins: Utilizes essential plugins to supercharge Jenkins configuration and job creation:
- JobDSL Plugin: Generates jobs programmatically from code
- Configuration as Code (JCasC): Manages Jenkins system configuration as YAML
- Role Strategy Plugin: Implements role-based access control
- Pipeline Plugin: Enables declarative and scripted pipelines
- Git Plugin: Integrates Git repositories for source control

Key Benefits

✅ Zero Manual Configuration: Define jobs in YAML, JobGenie creates them automatically
✅ GitOps Native: Full version control and audit trail for all configurations
✅ Scalable: Handle hundreds of jobs with minimal effort
✅ Consistent: Single source of truth eliminates configuration drift
✅ Auditable: Complete Git history of all changes

Prerequisites

System Requirements

Server Requirements

OS: Amazon Linux 2023 / Amazon Linux 2 (ARM64 or x86_64) or any Linux distribution
CPU: Minimum 2 cores, recommended 4+ cores
RAM: Minimum 4GB, recommended 8GB+
Disk: Minimum 50GB, recommended 100GB+
Network: Internet access for plugin downloads and Git operations

Software Requirements

Jenkins: Version 2.528.2 or higher (LTS recommended)
Java: JDK 11 or JDK 17 (required by Jenkins)
Git: Latest version for repository access
Ansible: 2.9 or higher (for automated setup)
Python: 3.8 or higher (for Ansible)

Access Requirements

SSH access to Jenkins server with sudo/root privileges
Git repository access (SSH keys or HTTPS credentials)
Jenkins admin credentials

Required Jenkins Plugins

The following plugins must be installed in your Jenkins instance:

Required Plugins:
  - job-dsl                    # Job generation engine
  - configuration-as-code      # System configuration management
  - role-strategy             # Role-based access control
  - workflow-aggregator       # Pipeline support
  - git                       # Git integration
  - credentials-binding        # Secure credential management
  - build-user-vars-plugin    # Build user variables

Optional but Recommended:
  - docker-slaves            # Docker-based build agents
  - docker-workflow          # Docker pipeline support
  - ansicolor                # Colored console output
  - favorite                 # Favorite jobs
  - aws-java-sdk-secretsmanager  # AWS Secrets Manager integration

Integration Methods

There are two ways to integrate JobGenie into a new Jenkins instance:

Automated Setup with Ansible (Recommended for production)
Manual Integration (For quick setup or existing Jenkins)

Choose the method that best fits your environment and requirements.

Method 1: Automated Setup with Ansible

This method uses Ansible playbooks to automate the entire Jenkins setup, including JobGenie integration.

Step 1: Clone Required Repositories

# Clone the infrastructure repository (contains Ansible automation)
git clone https://github.com/HarryTheDevOpsGuy/mCloud-infra.git
cd mCloud-infra/ansible

# Clone the Jenkins configuration repository
cd ..
git clone https://github.com/HarryTheDevOpsGuy/mCloud-Jenkins.git
cd mCloud-Jenkins

Step 2: Configure Ansible Variables

Edit the Ansible group variables file to configure your Jenkins instance:

cd ../mCloud-infra/ansible
vim group_vars/packer_al2023_aarch64_devops_jenkins.yml

Essential Configuration

# Jenkins Version
jenkins_version: 2.528.2
jenkins_home: "/var/lib/jenkins"

# Jenkins Plugins (includes JobGenie requirements)
jenkins_plugins:
  - docker-slaves
  - docker-workflow
  - ansicolor
  - google-login
  - aws-java-sdk-secretsmanager
  - favorite
  - job-dsl                    # Required for JobGenie
  - configuration-as-code     # Required for JobGenie
  - role-strategy             # Required for JobGenie
  - workflow-aggregator       # Required for pipelines
  - git                       # Required for Git integration
  - credentials-binding       # Required for secure credentials

# Jenkins URL Configuration
jenkins_location:
  url: "https://jenkins.example.com/"
  adminAddress: "jenkins-admin@example.com"

# Security Configuration
jenkins_securityRealm:
  local:
    allowsSignup: false
    enableCaptcha: false
    users:
      - id: "admin"
        name: "admin"
        password: "${ADMIN_PASSWORD}"  # Use environment variable or Ansible Vault

# JobGenie Seed Job Configuration
jenkins_onboarding:
  global:
    common:
      mkdir: ["common/manage-infra"]
      jobs: []
        
  amazon:
    mcloud:
      jobs: []
      env: ["nonprod", "prod"]

# Access Control Configuration
overall_read_users:
  amazon:
    mcloud:
      - user: "dev-user"
    qa:
      - user: "qa-user"

overall_admin_users:
  devops_managers:
    - user: "hari_25585"
    - user: "admin"

Step 3: Configure Global Libraries

Set up shared libraries for JobGenie:

jenkins_glabal_libraries:
  - name: "sharedPipelineUtils"
    branch: "master"
    path: "sharedlibs"
    creds: "jenkins_repo_key"
    repo: "git@github.com:HarryTheDevOpsGuy/mCloud-Jenkins.git"

Step 4: Run Ansible Playbook

Execute the Ansible playbook to install and configure Jenkins:

# Test connectivity first (dry-run)
ansible-playbook packer.yml \
  -e "target_host=packer_al2023_aarch64_devops_jenkins" \
  --check

# Run full deployment
ansible-playbook packer.yml \
  -e "target_host=packer_al2023_aarch64_devops_jenkins" \
  -t deploy,monitoring

Playbook Tags

deploy: Jenkins installation and configuration
monitoring: Monitoring stack setup
install_stacks: Install specific stacks only

Step 5: Post-Deployment Configuration

After the Ansible playbook completes:

Access Jenkins: Navigate to https://jenkins.example.com/
Login: Use admin credentials configured in Ansible variables
Verify Configuration: Check that Configuration as Code is loaded
Verify Plugins: Ensure all required plugins are installed
Verify Seed Jobs: Check that JobGenie seed jobs exist

Step 6: Configure Git Credentials

Add SSH key credential for Git repository access:

Navigate to: Manage Jenkins → Credentials → System → Global credentials
Click Add Credentials
Configure:
- Kind: SSH Username with private key
- ID: jenkins_repo_key
- Username: git
- Private Key: Upload SSH private key or paste content
Click OK

Method 2: Manual Integration

If you have an existing Jenkins instance or prefer manual setup, follow these steps.

Step 1: Install Required Plugins

Navigate to: Manage Jenkins → Manage Plugins → Available
Search and install the following plugins:
- job-dsl
- configuration-as-code
- role-strategy
- workflow-aggregator
- git
- credentials-binding
- build-user-vars-plugin
Click Install without restart or Download now and install after restart
Restart Jenkins if prompted

Step 2: Configure Jenkins System Settings

Configure Git Settings

Navigate to: Manage Jenkins → Configure System
Under Git, configure:
- Git executable: /usr/bin/git (or your Git path)
- Create account: Configure if needed
Click Save

Configure Global Libraries

Navigate to: Manage Jenkins → Configure System → Global Pipeline Libraries
Click Add
Configure:
- Name: sharedPipelineUtils
- Default version: master
- Retrieval method: Modern SCM
- Source Code Management: Git
  - Repository URL: git@github.com:HarryTheDevOpsGuy/mCloud-Jenkins.git
  - Credentials: Select jenkins_repo_key (create if needed)
  - Library Path: sharedlibs
Click Save

Step 3: Configure Git Credentials

Navigate to: Manage Jenkins → Credentials → System → Global credentials
Click Add Credentials
Configure:
- Kind: SSH Username with private key
- ID: jenkins_repo_key
- Username: git
- Private Key: Upload or paste SSH private key
Click OK

Step 4: Create JobGenie Seed Job

Navigate to: New Item
Enter name: 0-JobGenie-Generator
Select: Pipeline
Click OK

Configure Seed Job

General Configuration:

Check This project is parameterized
Add String Parameter:
- Name: GitBranch
- Default Value: master
- Description: Config repository branch.

Pipeline Configuration:

Definition: Pipeline script from SCM
SCM: Git
Repository URL: https://github.com/HarryTheDevOpsGuy/mCloud-Jenkins.git
Credentials: Select jenkins_repo_key
Branch: ${GitBranch}
Script Path: JobGenie/pipelines/jobGenie.groovy

Pipeline Script (Alternative - Direct Script):

node {
  label "linux-slave"
  checkout scm

  def allEnvVars = [ message: 'Hello from pipeline', credentials: 'SECRET', TECHTEAM: 'mcloud' ] + env.getEnvironment()
  ['BUILD_NUMBER', 'BUILD_DISPLAY_NAME', 'JOB_NAME', 'JOB_BASE_NAME', 'WORKSPACE', 'BUILD_URL'].each { var -> 
    if (env."$var") { allEnvVars[var] = env."$var" }
  }

  dir('mCloud-Jenkins') {
    git url: 'https://github.com/HarryTheDevOpsGuy/mCloud-Jenkins.git', 
        branch: "${GitBranch}", 
        credentialsId: 'jenkins_repo_key'
  }

  jobDsl targets: ["JobGenie/pipelines/*.groovy"].join('\n'),
    additionalClasspath: 'mCloud-Jenkins/sharedlibs/src',
    removedJobAction: 'DELETE',
    removedViewAction: 'DELETE',
    additionalParameters: allEnvVars
}

Click Save

Step 5: Run Seed Job

Navigate to the 0-JobGenie-Generator job
Click Build with Parameters
Set GitBranch to master (or your config branch)
Click Build
Monitor the build console output for job generation

Directory Structure

JobGenie follows a well-organized directory structure designed to manage multiple organizations’ infrastructure, especially Kubernetes infrastructure.

Standard Structure

{organization}/{project}/{environment}/
├── jobs/
│   └── {project}-{environment}-jobs.yml
├── jenkinsfiles/
│   ├── default/
│   └── sample/
└── helm-values/
    ├── apps/
    └── infra/

Job File Path Convention

{organization}/{project}/{environment}/jobs/{project}-{environment}-jobs.yml

Examples:

amazon/mcloud/prod/jobs/mcloud-prod-jobs.yml
amazon/mcloud/nonprod/jobs/mcloud-nonprod-jobs.yml
global/common/nonprod/jobs/common-nonprod-jobs.yml

Create Directory Structure

You can create the standard directory structure using:

# Create organization-specific structure
mkdir -p amazon/{mcloud,xcloud}/{nonprod,prod}/{jenkinsfiles/{default,sample},jobs,helm-values/{apps,infra}}

# Create global/common structure
mkdir -p global/common/{nonprod,prod}/{jenkinsfiles/{default,sample},jobs,helm-values/{apps,infra}}

Directory Structure Benefits

Organized: Clear separation by organization, project, and environment
Scalable: Easy to add new projects and environments
Maintainable: Consistent structure across all projects
Kubernetes-Ready: Designed for managing Kubernetes infrastructure

Creating Your First Job

Now that JobGenie is integrated, let’s create your first YAML-driven job.

Step 1: Create Job Definition File

Create a YAML file following the naming convention:

cd mCloud-Jenkins
mkdir -p amazon/myproject/prod/jobs
vim amazon/myproject/prod/jobs/myproject-prod-jobs.yml

Step 2: Define Your Job

Add the following content to your YAML file:

jobgenie:
  default:
    HOME_DIR: prod
    GROUP: "v2"
    ENV: "prod"
  jobs:
    - NAME: "hello-world-service"
      PARAMETERS:
        - { name: 'GitBranch', string: 'main', description: 'Application git branch.' }
      CONFIGS:
        APP_REPO: "hello-world"
        APP_BRANCH: "main"
        DOCKER_BUILD_ARGS: "ENV,SERVICE"

Step 3: Commit and Push

git add amazon/myproject/prod/jobs/myproject-prod-jobs.yml
git commit -m "Add hello-world-service job definition"
git push origin main

Step 4: Run JobGenie Seed Job

Navigate to: amazon/common/0-JobGenie-Generator (or your seed job)
Click Build with Parameters
Set GitBranch to your branch (e.g., main)
Click Build
JobGenie will scan, parse, and create your jobs automatically

Step 5: Verify Job Creation

Check Jenkins for the newly created job:

Path: amazon/myproject/prod/deploy/v2/prod/hello-world-service
Status: Job should be ready to build immediately
Configuration: Verify parameters and pipeline configuration

Verification and Testing

Verify JobGenie Integration

Check Seed Job: Verify seed job exists and runs successfully
Check Console Output: Review seed job console for any errors
Verify Job Generation: Confirm jobs are created in expected locations
Test Job Execution: Run a generated job to ensure it works

Common Verification Steps

# Check Jenkins logs
sudo tail -f /var/lib/jenkins/logs/jenkins.log

# Verify plugins are installed
curl -s http://localhost:8080/pluginManager/api/json?depth=1 | jq '.plugins[] | select(.shortName | contains("job-dsl"))'

# Check JobDSL script approval
# Navigate to: Manage Jenkins → In-process Script Approval

Test Job Definition

Before committing, validate your YAML syntax:

# Install yamllint (if not installed)
pip install yamllint

# Validate YAML syntax
yamllint amazon/myproject/prod/jobs/myproject-prod-jobs.yml

Next Steps

Now that JobGenie is integrated, explore these advanced features:

Multi-Environment Jobs: Create jobs for dev, staging, and prod
Parameterized Builds: Add choice parameters, boolean parameters
Custom Templates: Use custom pipeline templates
Freestyle Jobs: Create freestyle jobs for specialized tasks
Advanced Configuration: Explore environment variable injection, custom paths

Troubleshooting

Issue: Seed Job Fails to Run

Symptoms: Seed job fails with errors

Solutions:

Verify all required plugins are installed
Check Git credentials are configured correctly
Verify repository URL and branch are correct
Review console output for specific error messages
Ensure JobDSL script approval is granted

Issue: Jobs Not Created

Symptoms: Seed job runs successfully but no jobs are created

Solutions:

Verify YAML file naming: must end with -jobs.yml or -jobs.yaml
Check YAML syntax is valid (use yamllint)
Verify file path matches expected structure
Review seed job console output for parsing errors
Ensure jobgenie: key is present at root level

Issue: YAML Parsing Errors

Symptoms: Seed job fails with YAML parsing errors

Solutions:

Validate YAML syntax using online validators or yamllint
Check indentation (must be spaces, not tabs)
Verify all required fields are present (default, jobs)
Ensure job names don’t contain invalid characters
Review console output for specific line numbers

Issue: Jobs Created in Wrong Location

Symptoms: Jobs appear in unexpected folders

Solutions:

Check HOME_DIR, GROUP, and ENV in default section
Verify PIPELINE_PATH if using custom paths
Review job name structure (slashes create subfolders)
Ensure environment variables are set correctly

Issue: Git Credentials Not Working

Symptoms: Seed job fails to checkout repository

Solutions:

Verify SSH key is correctly configured
Test SSH access manually: ssh -T git@github.com
Check credential ID matches configuration
Ensure private key format is correct (PEM format)
Verify user has repository access

Getting Help

If you encounter issues not covered here:

Check the Full Documentation
Review seed job console output
Check Jenkins system logs
Contact the maintainer: HarryTheDevOpsGuy@gmail.com

Summary

You’ve successfully integrated JobGenie into your Jenkins instance! Here’s what you’ve accomplished:

✅ Installed Required Plugins: JobDSL, Configuration as Code, Role Strategy
✅ Configured Git Credentials: Set up repository access
✅ Created Seed Job: JobGenie generator job is ready
✅ Created Directory Structure: Organized structure for job definitions
✅ Created First Job: YAML-driven job definition

Next: Start creating more job definitions and explore advanced features!

**Need Help?** Check out the [JobGenie Full Documentation](/docs/tools/jobgenie/) for more detailed information, examples, and best practices.

Built with ❤️ by the DevOps Team

“Soch Wahi, Approach Nai” - Same Vision, New Approach

From Zero to Hero in Minutes, Not Months 🚀

Tags: #Jenkins #JobDSL #YAML #GitOps #CI/CD #Automation #DevOps

Related Tools & Projects

BG Deployer

Automated blue-green deployment for zero-downtime AWS releases

AWS Blue-Green

mCert

SSL certificate monitoring with Slack/email alerts & Telegram

SSL Telegram

mTracker

Real-time Linux user activity monitoring with Slack notifications

Linux Monitoring

mWatcher

Server health monitoring for CPU, memory, disk with alerting

Health Alerts

gCrypt

Git-crypt wrapper for secure file encryption & access management

Security Git-crypt

Interactive Tools

AWS VPC Designer, EKS Cost Calculator, and more utilities

Tools Calculator

External Resources

Official Documentation

AWS Documentation

Complete AWS service documentation

Community Resources

GitHub Profile

Open source projects and repositories

Professional network and updates

Dev.to

Technical articles and tutorials

Contact

Get in touch for questions or collaboration

Quick Actions

All Docs

Browse all documentation

Blog

Tools

DevOps utilities

Portfolio

My work & projects

Found this helpful?

Help us improve this documentation by sharing your feedback or suggesting improvements.

Report Issue Start Discussion

JobGenie Getting Started Guide

📋 Table of Contents

Overview

Core Components

Key Benefits

Prerequisites

System Requirements

Server Requirements

Software Requirements

Access Requirements

Required Jenkins Plugins

Integration Methods

Method 1: Automated Setup with Ansible

Step 1: Clone Required Repositories

Step 2: Configure Ansible Variables

Essential Configuration

Step 3: Configure Global Libraries

Step 4: Run Ansible Playbook

Playbook Tags

Step 5: Post-Deployment Configuration

Step 6: Configure Git Credentials

Method 2: Manual Integration

Step 1: Install Required Plugins

Step 2: Configure Jenkins System Settings

Configure Git Settings

Configure Global Libraries

Step 3: Configure Git Credentials

Step 4: Create JobGenie Seed Job

Configure Seed Job

Step 5: Run Seed Job

Directory Structure

Standard Structure

Job File Path Convention

Create Directory Structure

Directory Structure Benefits

Creating Your First Job

Step 1: Create Job Definition File

Step 2: Define Your Job

Step 3: Commit and Push

Step 4: Run JobGenie Seed Job

Step 5: Verify Job Creation

Verification and Testing

Verify JobGenie Integration

Common Verification Steps

Test Job Definition

Next Steps

Recommended Reading

Troubleshooting

Issue: Seed Job Fails to Run

Issue: Jobs Not Created

Issue: YAML Parsing Errors

Issue: Jobs Created in Wrong Location

Issue: Git Credentials Not Working

Getting Help

Summary

Related Documentation

More from Tools

BG Deployer

mCert

Related by Tags

Related Blog Posts

JobGenie: Transform Jenkins Job Creation with Jobs as Code

OpenResty Production Setup: Supercharge with Lua-Based Metrics and Monitoring

KEDA on EKS: Complete Guide to Event-Driven Autoscaling with Real-World Examples

Related Tools & Projects

BG Deployer

mCert

mTracker

mWatcher

gCrypt

Interactive Tools

External Resources

Official Documentation

AWS Documentation

Community Resources

GitHub Profile

LinkedIn

Dev.to

Contact

Quick Actions