How to deploy MinIO on Kubernetes with high availability?

Question

How do I deploy MinIO on Kubernetes with high availability, proper storage configuration, and production-ready settings? What are the best practices for running MinIO workloads in Kubernetes?

Answer

Deploying MinIO on Kubernetes requires careful consideration of storage, networking, and high availability requirements. This guide covers multiple deployment approaches from Helm charts to the MinIO Operator.

1. Prerequisites

Kubernetes Cluster Requirements

Kubernetes Version: 1.20+
Storage: Persistent Volume support (CSI preferred)
Network: Pod-to-pod communication
Resources: Adequate CPU/Memory per node
RBAC: Enabled for operator functionality

Required Tools

# Install kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/

# Install Helm
curl https://get.helm.sh/helm-v3.14.0-linux-amd64.tar.gz | tar -xzO linux-amd64/helm > helm
chmod +x helm && sudo mv helm /usr/local/bin/

# Install kustomize
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
sudo mv kustomize /usr/local/bin/

2. Storage Planning

Storage Classes and Requirements

# storage-class.yaml - Example for local SSDs
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: minio-local-ssd
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Retain
parameters:
  type: "local-ssd"
---
# For cloud environments (AWS EBS example)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: minio-gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Retain

Persistent Volumes for Local Storage

# local-pvs.yaml - Local storage example
apiVersion: v1
kind: PersistentVolume
metadata:
  name: minio-pv-1
spec:
  capacity:
    storage: 1Ti
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: minio-local-ssd
  local:
    path: /mnt/disks/ssd1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - worker-node-1

3. Method 1: MinIO Operator Deployment

Install MinIO Operator

# Add MinIO Operator repository
helm repo add minio-operator https://operator.min.io
helm repo update

# Install the operator
helm install minio-operator minio-operator/operator \
  --namespace minio-operator \
  --create-namespace \
  --set console.enabled=true

# Verify installation
kubectl get pods -n minio-operator

Create MinIO Tenant

apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: minio-tenant-1
  namespace: minio-tenant
spec:
  # Image configuration
  image: quay.io/minio/minio:RELEASE.2024-01-16T16-07-38Z
  imagePullPolicy: IfNotPresent

  # Tenant configuration
  credsSecret:
    name: minio-creds-secret
  configuration:
    name: minio-env-configuration

  # Pool configuration for HA
  pools:
  - servers: 4                    # Number of MinIO servers
    name: pool-0
    volumesPerServer: 4           # Drives per server
    volumeClaimTemplate:
      metadata:
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Ti
        storageClassName: minio-local-ssd

    # Resource requests and limits
    resources:
      requests:
        cpu: "2"
        memory: "8Gi"
      limits:
        cpu: "4"
        memory: "16Gi"

    # Security context
    securityContext:
      runAsUser: 1000
      runAsGroup: 1000
      fsGroup: 1000

    # Affinity rules for HA
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: v1.min.io/tenant
              operator: In
              values:
              - minio-tenant-1
          topologyKey: kubernetes.io/hostname

  # Console configuration
  console:
    image: quay.io/minio/console:v0.22.5
    replicas: 2
    consoleSecret:
      name: console-secret

  # Features
  features:
    bucketDNS: false
    domains: {}

  # Certificate configuration (optional)
  # requestAutoCert: true
  # certConfig:
  #   commonName: "minio.example.com"
  #   organizationName: ["Example Corp"]
  #   dnsNames:
  #     - "minio.example.com"
  #     - "*.minio.example.com"

Create Required Secrets

# Create namespace
kubectl create namespace minio-tenant

# MinIO credentials secret
kubectl create secret generic minio-creds-secret \
  --from-literal=accesskey=minio-admin \
  --from-literal=secretkey=minio-admin-password \
  --namespace minio-tenant

# Console credentials secret
kubectl create secret generic console-secret \
  --from-literal=CONSOLE_ACCESS_KEY=console-admin \
  --from-literal=CONSOLE_SECRET_KEY=console-admin-password \
  --from-literal=CONSOLE_PBKDF_PASSPHRASE=console-secret \
  --from-literal=CONSOLE_PBKDF_SALT=console-salt \
  --namespace minio-tenant

# Configuration secret (optional)
kubectl create secret generic minio-env-configuration \
  --from-literal=config.env="
export MINIO_ROOT_USER=minio-admin
export MINIO_ROOT_PASSWORD=minio-admin-password
export MINIO_STORAGE_CLASS_STANDARD=EC:4
export MINIO_BROWSER=on
" \
  --namespace minio-tenant

Deploy the Tenant

# Apply the tenant configuration
kubectl apply -f minio-tenant.yaml

# Monitor deployment
kubectl get tenant -n minio-tenant -w

# Check pods
kubectl get pods -n minio-tenant

# Get service endpoints
kubectl get svc -n minio-tenant

4. Method 2: Helm Chart Deployment

Standard Helm Deployment

# Add MinIO Helm repository
helm repo add minio https://charts.min.io/
helm repo update

# Create values file
cat > minio-values.yaml << EOF
# MinIO Helm Chart Configuration

# Deployment mode - distributed for HA
mode: distributed

# Replica configuration
replicas: 8

# Persistence configuration
persistence:
  enabled: true
  storageClass: "minio-local-ssd"
  accessMode: ReadWriteOnce
  size: 1Ti

# Resource configuration
resources:
  requests:
    memory: "8Gi"
    cpu: "2"
  limits:
    memory: "16Gi"
    cpu: "4"

# Root credentials
auth:
  rootUser: "minio-admin"
  rootPassword: "minio-admin-password"

# Service configuration
service:
  type: ClusterIP
  port: "9000"
  nodePort: 32000

# Console configuration
consoleService:
  type: ClusterIP
  port: "9001"

# Ingress configuration
ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
  hosts:
    - host: minio.example.com
      paths:
        - path: /
          pathType: Prefix

consoleIngress:
  enabled: true
  ingressClassName: nginx
  hosts:
    - host: console.minio.example.com
      paths:
        - path: /
          pathType: Prefix

# Pod disruption budget
podDisruptionBudget:
  enabled: true
  maxUnavailable: 1

# Pod affinity for HA
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app.kubernetes.io/name
          operator: In
          values:
          - minio
      topologyKey: kubernetes.io/hostname

# Security context
securityContext:
  enabled: true
  runAsUser: 1000
  runAsGroup: 1000
  fsGroup: 1000

# Liveness and readiness probes
livenessProbe:
  httpGet:
    path: /minio/health/live
    port: 9000
    scheme: HTTP
  initialDelaySeconds: 120
  periodSeconds: 15
  timeoutSeconds: 10
  successThreshold: 1
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /minio/health/ready
    port: 9000
    scheme: HTTP
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 5
  successThreshold: 1
  failureThreshold: 3

# Metrics and monitoring
metrics:
  serviceMonitor:
    enabled: true
    namespace: monitoring
    interval: 30s
    scrapeTimeout: 10s
EOF

# Install MinIO
helm install minio minio/minio \
  --namespace minio \
  --create-namespace \
  --values minio-values.yaml

# Monitor deployment
kubectl get pods -n minio -w

5. Advanced Configuration

TLS/SSL Configuration

apiVersion: v1
kind: Secret
metadata:
  name: minio-tls
  namespace: minio-tenant
type: kubernetes.io/tls
data:
  tls.crt: |
    # Base64 encoded certificate
  tls.key: |
    # Base64 encoded private key
---
# Update tenant spec for TLS
spec:
  externalCertSecret:
    - name: minio-tls
      type: kubernetes.io/tls
  requestAutoCert: false

Network Policies

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: minio-network-policy
  namespace: minio-tenant
spec:
  podSelector:
    matchLabels:
      v1.min.io/tenant: minio-tenant-1
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: minio-tenant
    - podSelector:
        matchLabels:
          app: minio-client
    ports:
    - protocol: TCP
      port: 9000
    - protocol: TCP
      port: 9443
  egress:
  - to: []
    ports:
    - protocol: TCP
      port: 53
    - protocol: UDP
      port: 53
  - to:
    - namespaceSelector:
        matchLabels:
          name: minio-tenant
    ports:
    - protocol: TCP
      port: 9000

6. Monitoring and Observability

Prometheus Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-minio-config
  namespace: monitoring
data:
  minio.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
    - job_name: 'minio'
      metrics_path: /minio/v2/metrics/cluster
      static_configs:
      - targets: ['minio-tenant-1-hl.minio-tenant.svc.cluster.local:9000']
      basic_auth:
        username: 'minio-admin'
        password: 'minio-admin-password'

Grafana Dashboard

{
  "dashboard": {
    "title": "MinIO Dashboard",
    "panels": [
      {
        "title": "MinIO Uptime",
        "type": "stat",
        "targets": [
          {
            "expr": "minio_cluster_nodes_online_total"
          }
        ]
      },
      {
        "title": "Storage Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "minio_cluster_capacity_usable_total_bytes"
          }
        ]
      }
    ]
  }
}

7. Backup and Disaster Recovery

Cross-Site Replication Setup

# Configure replication between clusters
mc admin replicate add minio-primary minio-dr \
  --priority 1 \
  --healthcheck-seconds 60 \
  --bandwidth 1GB

# Set bucket replication
mc replicate add minio-primary/data-bucket \
  --remote-bucket minio-dr/data-bucket-replica \
  --priority 1

Backup Automation

apiVersion: batch/v1
kind: CronJob
metadata:
  name: minio-backup
  namespace: minio-tenant
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: minio/mc:latest
            command:
            - /bin/sh
            - -c
            - |
              mc alias set source $SOURCE_URL $SOURCE_ACCESS_KEY $SOURCE_SECRET_KEY
              mc alias set backup $BACKUP_URL $BACKUP_ACCESS_KEY $BACKUP_SECRET_KEY
              mc mirror source/ backup/$(date +%Y%m%d)/ --overwrite
            env:
            - name: SOURCE_URL
              value: "http://minio-tenant-1-hl.minio-tenant.svc.cluster.local:9000"
            - name: SOURCE_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-creds-secret
                  key: accesskey
            - name: SOURCE_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: minio-creds-secret
                  key: secretkey
          restartPolicy: OnFailure

8. Scaling and Maintenance

Horizontal Scaling

# Scale tenant pools
kubectl patch tenant minio-tenant-1 -n minio-tenant --type='merge' -p='
{
  "spec": {
    "pools": [
      {
        "servers": 8,
        "name": "pool-0",
        "volumesPerServer": 4
      }
    ]
  }
}'

# Add new pool for expansion
kubectl patch tenant minio-tenant-1 -n minio-tenant --type='merge' -p='
{
  "spec": {
    "pools": [
      {
        "servers": 4,
        "name": "pool-0",
        "volumesPerServer": 4
      },
      {
        "servers": 4,
        "name": "pool-1",
        "volumesPerServer": 4,
        "volumeClaimTemplate": {
          "spec": {
            "accessModes": ["ReadWriteOnce"],
            "resources": {"requests": {"storage": "1Ti"}},
            "storageClassName": "minio-local-ssd"
          }
        }
      }
    ]
  }
}'

Rolling Updates

# Update MinIO version
kubectl patch tenant minio-tenant-1 -n minio-tenant --type='merge' -p='
{
  "spec": {
    "image": "quay.io/minio/minio:RELEASE.2024-02-01T10-30-00Z"
  }
}'

# Monitor rolling update
kubectl rollout status statefulset/minio-tenant-1-pool-0 -n minio-tenant

9. Production Best Practices

Resource Planning

resources:
  requests:
    # CPU: 0.5 cores per TB of storage
    cpu: "2"
    # Memory: 2GB per instance + 1GB per TB
    memory: "8Gi"
  limits:
    # CPU: Allow bursting to 2x requests
    cpu: "4"
    # Memory: 2x requests for safety
    memory: "16Gi"

Pod Disruption Budgets

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: minio-pdb
  namespace: minio-tenant
spec:
  minAvailable: 50%
  selector:
    matchLabels:
      v1.min.io/tenant: minio-tenant-1

10. Troubleshooting Common Issues

Debug Pod Issues

# Check pod status
kubectl describe pods -n minio-tenant

# View logs
kubectl logs -f deployment/minio-tenant-1-pool-0-0 -n minio-tenant

# Check events
kubectl get events -n minio-tenant --sort-by='.lastTimestamp'

# Exec into pod for debugging
kubectl exec -it minio-tenant-1-pool-0-0 -n minio-tenant -- /bin/sh

Storage Issues

# Check PV/PVC status
kubectl get pv,pvc -n minio-tenant

# Check storage class
kubectl describe storageclass minio-local-ssd

# Check node storage
kubectl describe nodes | grep -A 5 "Allocated resources"

11. Performance Optimization

Node Affinity for Performance

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: node-type
          operator: In
          values:
          - storage-optimized
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: v1.min.io/tenant
            operator: In
            values:
            - minio-tenant-1
        topologyKey: kubernetes.io/hostname

This comprehensive Kubernetes deployment guide provides multiple approaches for running MinIO in production with high availability, proper monitoring, and enterprise-grade features.