Secrets management is a critical component of any DevOps workflow. Exposing API keys, passwords, or certificates can lead to devastating security breaches, compliance violations, and financial losses. In this comprehensive guide, we’ll explore best practices and tools for securing secrets throughout your DevOps lifecycle.
Why Secure Secrets Matter
The consequences of poor secrets management:
- Data Breaches: Exposed credentials can lead to unauthorized access
- Financial Loss: Cryptojacking, resource abuse, regulatory fines
- Compliance Violations: GDPR, HIPAA, PCI-DSS require secure secret handling
- Reputation Damage: Public breaches erode customer trust
- Supply Chain Attacks: Compromised CI/CD pipelines affect downstream systems
Real-world incident: In 2020, a major breach exposed 100,000+ repositories with hardcoded credentials on GitHub. Don’t be a statistic!
Common Anti-Patterns (Don’t Do This!)
# ❌ NEVER: Hardcode secrets in code
DATABASE_PASSWORD="SuperSecret123"
API_KEY="sk-1234567890abcdef"
# ❌ NEVER: Commit secrets to version control
git add .env
git commit -m "Added API keys"
# ❌ NEVER: Pass secrets as plain text arguments
docker run -e PASSWORD=secret123 myapp
# ❌ NEVER: Log secrets
echo "Connecting with password: $PASSWORD"
# ❌ NEVER: Store secrets in Docker images
RUN echo "API_KEY=abc123" >> /app/.env
Secrets Management Tools Comparison
| Tool | Type | Use Case | Pros | Cons |
|---|---|---|---|---|
| HashiCorp Vault | Universal | Enterprise-grade secret management | Feature-rich, audit logs, dynamic secrets | Complex setup |
| Azure Key Vault | Cloud-native | Azure workloads | Deep Azure integration, managed service | Azure-specific |
| AWS Secrets Manager | Cloud-native | AWS workloads | Auto-rotation, versioning | AWS-specific, cost |
| GCP Secret Manager | Cloud-native | GCP workloads | Simple, scalable | GCP-specific |
| Kubernetes Secrets | Platform-specific | K8s clusters | Native integration | Base64 only, limited features |
| SOPS | File encryption | Git-stored secrets | Git-friendly, multiple backends | Requires careful key management |
HashiCorp Vault: Deep Dive
Installation & Setup
# Install Vault
wget https://releases.hashicorp.com/vault/1.15.0/vault_1.15.0_linux_amd64.zip
unzip vault_1.15.0_linux_amd64.zip
sudo mv vault /usr/local/bin/
# Start Vault in dev mode (for testing)
vault server -dev -dev-root-token-id="root"
# Set environment variables
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN="root"
# Check status
vault status
Production Deployment on Kubernetes
# vault-deployment.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: vault
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault
namespace: vault
---
apiVersion: v1
kind: ConfigMap
metadata:
name: vault-config
namespace: vault
data:
vault.hcl: |
ui = true
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 0
tls_cert_file = "/vault/tls/tls.crt"
tls_key_file = "/vault/tls/tls.key"
}
storage "file" {
path = "/vault/data"
}
api_addr = "https://vault.vault.svc.cluster.local:8200"
cluster_addr = "https://vault.vault.svc.cluster.local:8201"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: vault
namespace: vault
spec:
serviceName: vault
replicas: 3
selector:
matchLabels:
app: vault
template:
metadata:
labels:
app: vault
spec:
serviceAccountName: vault
containers:
- name: vault
image: hashicorp/vault:1.15.0
args:
- server
- -config=/vault/config/vault.hcl
ports:
- containerPort: 8200
name: vault-port
- containerPort: 8201
name: cluster-port
env:
- name: VAULT_ADDR
value: "https://127.0.0.1:8200"
- name: VAULT_API_ADDR
value: "https://vault.vault.svc.cluster.local:8200"
volumeMounts:
- name: vault-config
mountPath: /vault/config
- name: vault-data
mountPath: /vault/data
- name: vault-tls
mountPath: /vault/tls
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
livenessProbe:
httpGet:
path: /v1/sys/health?standbyok=true
port: 8200
scheme: HTTPS
initialDelaySeconds: 60
periodSeconds: 10
readinessProbe:
httpGet:
path: /v1/sys/health?standbyok=true
port: 8200
scheme: HTTPS
initialDelaySeconds: 30
periodSeconds: 10
volumes:
- name: vault-config
configMap:
name: vault-config
- name: vault-tls
secret:
secretName: vault-tls
volumeClaimTemplates:
- metadata:
name: vault-data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
name: vault
namespace: vault
spec:
type: ClusterIP
selector:
app: vault
ports:
- port: 8200
targetPort: 8200
name: vault-port
- port: 8201
targetPort: 8201
name: cluster-port
Working with Vault
# Initialize Vault
vault operator init -key-shares=5 -key-threshold=3
# Unseal Vault (repeat with 3 different keys)
vault operator unseal <unseal-key-1>
vault operator unseal <unseal-key-2>
vault operator unseal <unseal-key-3>
# Login
vault login <root-token>
# Enable secrets engine
vault secrets enable -path=secret kv-v2
# Store secrets
vault kv put secret/database/config \
username="dbadmin" \
password="SuperSecurePassword123!" \
host="db.example.com" \
port="5432"
# Read secrets
vault kv get secret/database/config
vault kv get -field=password secret/database/config
# Create a policy
vault policy write myapp-policy - <<EOF
path "secret/data/myapp/*" {
capabilities = ["read", "list"]
}
EOF
# Create token with policy
vault token create -policy=myapp-policy -ttl=1h
Dynamic Secrets for Databases
# Enable database secrets engine
vault secrets enable database
# Configure PostgreSQL
vault write database/config/postgresql \
plugin_name=postgresql-database-plugin \
allowed_roles="readonly" \
connection_url="postgresql://:@postgres:5432/mydb?sslmode=disable" \
username="vault" \
password="vaultpass"
# Create role
vault write database/roles/readonly \
db_name=postgresql \
creation_statements="CREATE ROLE \"\" WITH LOGIN PASSWORD '' VALID UNTIL '' IN ROLE readonly;" \
default_ttl="1h" \
max_ttl="24h"
# Generate dynamic credentials
vault read database/creds/readonly
# Output:
# Key Value
# --- -----
# lease_id database/creds/readonly/abc123
# lease_duration 1h
# username v-root-readonly-xyz789
# password A1a-randompassword
Azure Key Vault Integration
Setup & Configuration
# Install Azure CLI
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
# Login
az login
# Create Key Vault
az keyvault create \
--name "myapp-vault" \
--resource-group "myResourceGroup" \
--location "eastus" \
--enable-rbac-authorization false
# Add secrets
az keyvault secret set \
--vault-name "myapp-vault" \
--name "DatabasePassword" \
--value "SuperSecure123!"
az keyvault secret set \
--vault-name "myapp-vault" \
--name "APIKey" \
--value "sk-1234567890abcdef"
# List secrets
az keyvault secret list --vault-name "myapp-vault" --output table
# Retrieve secret
az keyvault secret show \
--vault-name "myapp-vault" \
--name "DatabasePassword" \
--query "value" -o tsv
Azure Pipeline Integration
# azure-pipelines.yml
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
variables:
- group: production-variables # Variable group linked to Key Vault
steps:
- task: AzureKeyVault@2
displayName: 'Fetch secrets from Azure Key Vault'
inputs:
azureSubscription: 'Azure-Service-Connection'
KeyVaultName: 'myapp-vault'
SecretsFilter: 'DatabasePassword,APIKey,SSHPrivateKey'
RunAsPreJob: true
- task: Bash@3
displayName: 'Deploy application with secrets'
inputs:
targetType: 'inline'
script: |
# Secrets are now available as environment variables
echo "Database Password length: ${#DATABASEPASSWORD}"
# Use secrets in deployment
kubectl create secret generic app-secrets \
--from-literal=db-password="$(DatabasePassword)" \
--from-literal=api-key="$(APIKey)" \
--dry-run=client -o yaml | kubectl apply -f -
# Never log actual secret values!
env:
DATABASEPASSWORD: $(DatabasePassword)
APIKEY: $(APIKey)
Managed Identity Integration
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
template:
metadata:
labels:
aadpodidbinding: myapp-identity
spec:
containers:
- name: myapp
image: myapp:latest
env:
- name: KEYVAULT_NAME
value: "myapp-vault"
- name: AZURE_CLIENT_ID
value: "<managed-identity-client-id>"
Application code:
# app.py
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
import os
# Authenticate using Managed Identity
credential = DefaultAzureCredential()
vault_url = f"https://{os.environ['KEYVAULT_NAME']}.vault.azure.net"
client = SecretClient(vault_url=vault_url, credential=credential)
# Retrieve secrets
db_password = client.get_secret("DatabasePassword").value
api_key = client.get_secret("APIKey").value
# Use secrets (never log them!)
print("Successfully retrieved secrets")
Kubernetes Secrets Best Practices
Using External Secrets Operator
# Install External Secrets Operator
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets-system \
--create-namespace
# secret-store.yaml
---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: azure-keyvault-store
namespace: production
spec:
provider:
azurekv:
vaultUrl: "https://myapp-vault.vault.azure.net"
authType: ManagedIdentity
identityId: "<managed-identity-id>"
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: app-secrets
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: azure-keyvault-store
kind: SecretStore
target:
name: app-secrets
creationPolicy: Owner
data:
- secretKey: database-password
remoteRef:
key: DatabasePassword
- secretKey: api-key
remoteRef:
key: APIKey
Encrypting Secrets at Rest
# Create encryption key
head -c 32 /dev/urandom | base64
# Update API server with encryption config
cat <<EOF > /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded-secret>
- identity: {}
EOF
# Add to kube-apiserver manifest
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml
SOPS: Encrypted Files in Git
# Install SOPS
wget https://github.com/mozilla/sops/releases/download/v3.8.0/sops_3.8.0_amd64.deb
sudo dpkg -i sops_3.8.0_amd64.deb
# Configure with age (simple alternative to GPG)
age-keygen -o key.txt
export SOPS_AGE_KEY_FILE=key.txt
# Encrypt file
sops -e secrets.yaml > secrets.enc.yaml
# Decrypt file
sops -d secrets.enc.yaml
# Edit encrypted file
sops secrets.enc.yaml
Example .sops.yaml:
creation_rules:
- path_regex: \.production\.yaml$
age: 'age1...'
- path_regex: \.development\.yaml$
age: 'age2...'
Encrypted secrets file:
# secrets.enc.yaml
database:
password: ENC[AES256_GCM,data:...,iv:...,tag:...,type:str]
username: admin # Unencrypted values possible
api:
key: ENC[AES256_GCM,data:...,iv:...,tag:...,type:str]
sops:
age:
- recipient: age1...
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
...
-----END AGE ENCRYPTED FILE-----
GitHub Actions Secrets
# .github/workflows/deploy.yml
name: Deploy
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v2
with:
aws-access-key-id: $
aws-secret-access-key: $
aws-region: us-east-1
- name: Login to ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
- name: Build and push Docker image
env:
ECR_REGISTRY: $
IMAGE_TAG: $
run: |
docker build -t $ECR_REGISTRY/myapp:$IMAGE_TAG .
docker push $ECR_REGISTRY/myapp:$IMAGE_TAG
- name: Deploy to Kubernetes
env:
KUBE_CONFIG: $
run: |
echo "$KUBE_CONFIG" | base64 -d > kubeconfig
export KUBECONFIG=kubeconfig
kubectl set image deployment/myapp myapp=$ECR_REGISTRY/myapp:$IMAGE_TAG
Security Best Practices Checklist
✅ Never commit secrets to version control
✅ Use .gitignore for sensitive files (.env, *.key, etc.)
✅ Rotate secrets regularly (automated with dynamic secrets)
✅ Implement least-privilege access (grant minimum required permissions)
✅ Audit secret access (enable logging on all secret stores)
✅ Encrypt secrets at rest and in transit
✅ Use short-lived credentials (TTL-based tokens)
✅ Separate secrets per environment (dev, staging, prod)
✅ Implement secret scanning (detect committed secrets)
✅ Use managed identities (avoid long-lived credentials)
✅ Monitor for secret exposure (alerts on suspicious access)
✅ Have a secrets incident response plan
Secret Scanning Tools
# Gitleaks - Scan for secrets in git history
docker run -v $(pwd):/path zricethezav/gitleaks:latest \
detect --source="/path" --verbose
# TruffleHog - Find high entropy strings
docker run --rm -v $(pwd):/repo trufflesecurity/trufflehog:latest \
filesystem /repo
# git-secrets - Prevent commits with secrets
git secrets --install
git secrets --register-aws
git secrets --scan
Incident Response: What If Secrets Are Leaked?
- Immediate Actions (within minutes)
# Revoke/rotate compromised secrets immediately vault token revoke <token> az keyvault secret set-attributes --enabled false --name SecretName # Block access if needed vault policy write compromised-deny - <<EOF path "*" { capabilities = ["deny"] } EOF - Investigation (within hours)
- Check audit logs for unauthorized access
- Identify scope of exposure
- Review recent deployments
- Remediation (within 24 hours)
- Rotate all potentially affected secrets
- Update applications with new secrets
- Patch vulnerability that led to exposure
- Post-Incident (within 1 week)
- Document incident timeline
- Update security procedures
- Implement additional controls
Monitoring & Auditing
# Vault metrics in Prometheus
vault_token_creation_count
vault_secret_kv_count
vault_audit_log_request_failure
# Alerting rules
- alert: VaultHighFailureRate
expr: rate(vault_audit_log_request_failure[5m]) > 0.05
for: 5m
annotations:
summary: "High failure rate on Vault audit logs"
Conclusion
Securing secrets is not optional—it’s a fundamental requirement for any modern DevOps practice. By using proper tools like HashiCorp Vault, cloud key management services, and following best practices, you can significantly reduce the risk of secret exposure and maintain compliance with security standards.
Remember: The best secret is one that never existed. Use dynamic credentials and managed identities whenever possible.
Additional Resources
- HashiCorp Vault Documentation
- Azure Key Vault Best Practices
- OWASP Secrets Management Cheat Sheet
- CIS Kubernetes Benchmark
How do you manage secrets in your organization? Share your experiences in the comments!