Container security is critical in modern cloud-native environments. This comprehensive guide covers security best practices across the entire container lifecycle—from build to runtime—for both Docker and Kubernetes.
Container Security Landscape
The Four Pillars of Container Security
- Image Security: Secure base images and dependencies
- Build Security: Secure CI/CD pipelines
- Registry Security: Secure image storage and distribution
- Runtime Security: Secure running containers and orchestration
Image Security
Use Trusted Base Images
# ❌ Bad - unofficial image
FROM some-random-image
# ❌ Bad - latest tag
FROM ubuntu:latest
# ✅ Good - official image with specific version
FROM ubuntu:22.04
# ✅ Better - minimal distroless image
FROM gcr.io/distroless/static-debian11
# ✅ Best - minimal scratch image for Go
FROM scratch
COPY --from=builder /app/binary /binary
ENTRYPOINT ["/binary"]
Minimize Image Layers
# ❌ Bad - multiple layers
RUN apt-get update
RUN apt-get install -y package1
RUN apt-get install -y package2
RUN apt-get clean
# ✅ Good - single layer, cleaned up
RUN apt-get update && \
apt-get install -y --no-install-recommends \
package1 \
package2 && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
Run as Non-Root User
# Create user
RUN groupadd -r appuser && useradd -r -g appuser appuser
# Set ownership
RUN chown -R appuser:appuser /app
# Switch to non-root user
USER appuser
# Verify
RUN whoami # Should output: appuser
For Alpine-based images:
RUN addgroup -g 1001 -S appuser && \
adduser -u 1001 -S appuser -G appuser
USER appuser
Multi-Stage Builds for Security
# Stage 1: Build
FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY . .
RUN go mod download
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .
# Stage 2: Production
FROM gcr.io/distroless/static-debian11
COPY --from=builder /app/main /main
EXPOSE 8080
USER nonroot:nonroot
ENTRYPOINT ["/main"]
Vulnerability Scanning
Trivy Scanner
# Install Trivy
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo "deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update && sudo apt-get install trivy
# Scan image
trivy image nginx:latest
# Scan with severity filter
trivy image --severity HIGH,CRITICAL nginx:latest
# Scan and fail on vulnerabilities
trivy image --exit-code 1 --severity CRITICAL myapp:latest
# Scan Dockerfile
trivy config Dockerfile
# Generate report
trivy image --format json --output report.json myapp:latest
# Scan with ignore file
cat > .trivyignore <<EOF
CVE-2021-12345
CVE-2022-67890
EOF
trivy image --ignorefile .trivyignore myapp:latest
Integrate Trivy in CI/CD
# .github/workflows/security-scan.yml
name: Container Security Scan
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build image
run: docker build -t myapp:$ .
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master
with:
image-ref: myapp:$
format: 'sarif'
output: 'trivy-results.sarif'
severity: 'CRITICAL,HIGH'
- name: Upload results to GitHub Security
uses: github/codeql-action/upload-sarif@v2
if: always()
with:
sarif_file: 'trivy-results.sarif'
Snyk Container Scanning
# Install Snyk CLI
npm install -g snyk
# Authenticate
snyk auth
# Scan image
snyk container test nginx:latest
# Monitor image
snyk container monitor nginx:latest --project-name=my-app
Grype Scanner
# Install Grype
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin
# Scan image
grype nginx:latest
# Output formats
grype nginx:latest -o json
grype nginx:latest -o cyclonedx
# Fail on severity
grype nginx:latest --fail-on critical
Docker Security Best Practices
Use Docker Bench Security
# Run Docker Bench Security
docker run --rm --net host --pid host --userns host --cap-add audit_control \
-e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
-v /var/lib:/var/lib \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /etc:/etc \
--label docker_bench_security \
docker/docker-bench-security
Enable Docker Content Trust
# Enable Content Trust
export DOCKER_CONTENT_TRUST=1
# Sign and push image
docker push myregistry.com/myapp:1.0
# Generate signing keys
docker trust key generate mykey
# Add signer
docker trust signer add --key mykey.pub myuser myregistry.com/myapp
Secure Docker Daemon
// /etc/docker/daemon.json
{
"icc": false,
"userns-remap": "default",
"no-new-privileges": true,
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3"
},
"live-restore": true,
"userland-proxy": false
}
Docker Security Options
# Run with security options
docker run \
--security-opt=no-new-privileges \
--cap-drop=ALL \
--cap-add=NET_BIND_SERVICE \
--read-only \
--tmpfs /tmp \
nginx:latest
# Use AppArmor profile
docker run --security-opt apparmor=docker-default nginx
# Use Seccomp profile
docker run --security-opt seccomp=/path/to/seccomp/profile.json nginx
Kubernetes Security
Pod Security Standards
# Restricted Pod Security
apiVersion: v1
kind: Pod
metadata:
name: secure-pod
labels:
app: myapp
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 3000
fsGroup: 2000
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: myapp:1.0
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
readOnlyRootFilesystem: true
volumeMounts:
- name: tmp
mountPath: /tmp
- name: cache
mountPath: /app/cache
resources:
limits:
cpu: "1"
memory: "512Mi"
requests:
cpu: "100m"
memory: "128Mi"
volumes:
- name: tmp
emptyDir: {}
- name: cache
emptyDir: {}
Pod Security Admission
# Namespace with Pod Security Standards
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
Network Policies
# Default deny all ingress and egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
# Allow specific traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-backend
namespace: production
spec:
podSelector:
matchLabels:
app: backend
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 8080
RBAC Configuration
# Service Account
apiVersion: v1
kind: ServiceAccount
metadata:
name: myapp-sa
namespace: production
---
# Role with minimal permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: myapp-role
namespace: production
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
resourceNames: ["myapp-secrets"]
---
# RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: myapp-rolebinding
namespace: production
subjects:
- kind: ServiceAccount
name: myapp-sa
namespace: production
roleRef:
kind: Role
name: myapp-role
apiGroup: rbac.authorization.k8s.io
Secrets Management
# Use External Secrets Operator
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: azure-keyvault
namespace: production
spec:
provider:
azurekv:
vaultUrl: "https://myvault.vault.azure.net"
authType: ManagedIdentity
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: app-secrets
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: azure-keyvault
kind: SecretStore
target:
name: app-secrets
creationPolicy: Owner
data:
- secretKey: db-password
remoteRef:
key: database-password
Runtime Security
Falco for Runtime Detection
# Install Falco
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco \
--namespace falco \
--create-namespace \
--set tty=true
# Custom Falco Rules
# /etc/falco/rules.d/custom_rules.yaml
- rule: Detect Shell in Container
desc: Detect execution of shell in container
condition: >
spawned_process and container and
proc.name in (bash, sh, zsh)
output: >
Shell spawned in container (user=%user.name container=%container.name
image=%container.image.repository command=%proc.cmdline)
priority: WARNING
tags: [shell, container]
- rule: Write to /etc
desc: Detect write to /etc directory
condition: >
open_write and container and
fd.name startswith /etc
output: >
Write to /etc (user=%user.name command=%proc.cmdline file=%fd.name
container=%container.name)
priority: ERROR
tags: [filesystem, container]
- rule: Outbound Connection to C2 Server
desc: Detect outbound connection to known C2 server
condition: >
outbound and container and
fd.sip in (suspicious_ips)
output: >
Outbound connection to C2 server (container=%container.name
dest=%fd.rip:%fd.rport command=%proc.cmdline)
priority: CRITICAL
tags: [network, container]
Sysdig Falco Sidekick
# Deploy Falco Sidekick
apiVersion: apps/v1
kind: Deployment
metadata:
name: falcosidekick
namespace: falco
spec:
replicas: 1
selector:
matchLabels:
app: falcosidekick
template:
metadata:
labels:
app: falcosidekick
spec:
containers:
- name: falcosidekick
image: falcosecurity/falcosidekick:latest
env:
- name: SLACK_WEBHOOKURL
valueFrom:
secretKeyRef:
name: falco-secrets
key: slack-webhook
- name: SLACK_MINIMUMPRIORITY
value: "error"
Image Signing and Verification
Sigstore Cosign
# Install Cosign
wget https://github.com/sigstore/cosign/releases/download/v2.2.0/cosign-linux-amd64
mv cosign-linux-amd64 /usr/local/bin/cosign
chmod +x /usr/local/bin/cosign
# Generate key pair
cosign generate-key-pair
# Sign image
cosign sign --key cosign.key myregistry.com/myapp:1.0
# Verify image
cosign verify --key cosign.pub myregistry.com/myapp:1.0
Admission Controller for Verification
# Install Sigstore Policy Controller
kubectl apply -f https://github.com/sigstore/policy-controller/releases/latest/download/release.yaml
# Create ClusterImagePolicy
apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
name: require-signed-images
spec:
images:
- glob: "myregistry.com/**"
authorities:
- key:
data: |
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
-----END PUBLIC KEY-----
Security Scanning Tools Comparison
| Tool | Type | Strengths | Best For |
|---|---|---|---|
| Trivy | Open Source | Fast, comprehensive | CI/CD integration |
| Grype | Open Source | Accurate, multiple sources | SBOM generation |
| Snyk | Commercial | Developer-friendly, fix advice | Developer workflow |
| Aqua Security | Commercial | Runtime protection | Enterprise |
| Sysdig | Commercial | Runtime + compliance | Production monitoring |
| Anchore | Open Source/Commercial | Policy enforcement | Compliance |
Compliance and Hardening
CIS Benchmark Compliance
# Install kube-bench
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
# View results
kubectl logs -f job/kube-bench
# Get JSON output
kubectl get cm kube-bench -o jsonpath='{.data.results}' | jq .
OPA Gatekeeper Policies
# Install Gatekeeper
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml
# Constraint Template
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8srequiredlabels
spec:
crd:
spec:
names:
kind: K8sRequiredLabels
validation:
openAPIV3Schema:
properties:
labels:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredlabels
violation[{"msg": msg, "details": {"missing_labels": missing}}] {
provided := {label | input.review.object.metadata.labels[label]}
required := {label | label := input.parameters.labels[_]}
missing := required - provided
count(missing) > 0
msg := sprintf("Required labels missing: %v", [missing])
}
---
# Constraint
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
name: require-app-labels
spec:
match:
kinds:
- apiGroups: ["apps"]
kinds: ["Deployment"]
namespaces:
- production
parameters:
labels: ["app", "owner", "environment"]
Security Monitoring
Prometheus Metrics
# ServiceMonitor for Falco
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: falco
namespace: falco
spec:
selector:
matchLabels:
app: falco
endpoints:
- port: metrics
interval: 30s
Grafana Dashboards
# ConfigMap with dashboard
apiVersion: v1
kind: ConfigMap
metadata:
name: security-dashboard
namespace: monitoring
data:
security.json: |
{
"dashboard": {
"title": "Container Security",
"panels": [
{
"title": "Vulnerability Count by Severity",
"targets": [{
"expr": "sum(trivy_image_vulnerabilities) by (severity)"
}]
},
{
"title": "Security Events",
"targets": [{
"expr": "rate(falco_events[5m])"
}]
}
]
}
}
Incident Response
Forensics with Sysdig Inspect
# Capture system state
kubectl exec -it pod-name -- sysdig -z -w capture.scap
# Analyze with Sysdig Inspect
sysdig-inspect capture.scap
Quarantine Compromised Pods
# Network Policy to isolate pod
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: quarantine-pod
namespace: production
spec:
podSelector:
matchLabels:
security: compromised
policyTypes:
- Ingress
- Egress
# No rules = deny all traffic
# Label compromised pod
kubectl label pod suspicious-pod security=compromised
# Drain node if needed
kubectl drain node-name --ignore-daemonsets --delete-emptydir-data
Security Checklist
Build Time
✅ Use minimal base images
✅ Run as non-root user
✅ Scan images for vulnerabilities
✅ Sign images
✅ Use multi-stage builds
✅ Remove unnecessary packages
✅ Set resource limits
✅ Use .dockerignore
Deployment Time
✅ Enable Pod Security Standards
✅ Implement Network Policies
✅ Use RBAC with least privilege
✅ Encrypt secrets at rest
✅ Enable audit logging
✅ Use admission controllers
✅ Implement resource quotas
✅ Use namespaces for isolation
Runtime
✅ Monitor with Falco/Sysdig
✅ Implement runtime policies
✅ Enable security contexts
✅ Use read-only filesystems
✅ Monitor network traffic
✅ Alert on suspicious behavior
✅ Regular security audits
✅ Incident response plan
Conclusion
Container security requires a multi-layered approach covering the entire lifecycle from build to runtime. By implementing these security practices—vulnerability scanning, runtime protection, network policies, and continuous monitoring—you’ll significantly reduce your attack surface and protect your containerized applications.
Resources
- CIS Docker Benchmark
- CIS Kubernetes Benchmark
- NIST Container Security Guide
- Kubernetes Security Best Practices
How do you secure your containers? Share your approach in the comments!