How to configure MinIO in high availability mode?

Question

How do I configure MinIO to run in high availability (HA) mode to ensure my object storage is resilient to node failures and provides continuous service availability?

Answer

MinIO achieves high availability through its distributed architecture with erasure coding. Here’s a comprehensive guide to setting up MinIO in HA mode:

Prerequisites

Minimum 4 drives across multiple nodes (for basic HA)
Even number of drives for optimal erasure coding efficiency
Network connectivity between all nodes
Time synchronization (NTP) across all nodes
DNS resolution or proper /etc/hosts configuration

1. Planning Your HA Setup

Recommended Configurations

Small HA Setup (4 nodes):

4 nodes × 4 drives = 16 drives total
Erasure Coding: EC:2 (2 parity drives per erasure set)
Fault Tolerance: 2 drive failures

Medium HA Setup (8 nodes):

8 nodes × 8 drives = 64 drives total
Erasure Coding: EC:4 (4 parity drives per erasure set)
Fault Tolerance: 4 drive failures

Large HA Setup (16 nodes):

16 nodes × 16 drives = 256 drives total
Erasure Coding: EC:8 (8 parity drives per erasure set)
Fault Tolerance: 8 drive failures

2. Server Configuration

Hardware Requirements

CPU: 4+ cores per node
RAM: 8GB+ per node (more for larger deployments)
Storage: Fast SSDs or enterprise HDDs
Network: 10Gbps+ for production workloads

Operating System Setup

# On each node, configure hostname
sudo hostnamectl set-hostname minio-node-1  # Adjust for each node

# Update /etc/hosts on all nodes
cat >> /etc/hosts << EOF
10.0.1.11 minio-node-1
10.0.1.12 minio-node-2
10.0.1.13 minio-node-3
10.0.1.14 minio-node-4
EOF

# Install NTP for time synchronization
sudo apt update && sudo apt install -y ntp
sudo systemctl enable ntp && sudo systemctl start ntp

3. MinIO Binary Installation

# Download MinIO server binary
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin/

# Create MinIO user and directories
sudo useradd -r minio-user -s /sbin/nologin
sudo mkdir -p /opt/minio
sudo mkdir -p /etc/minio
sudo mkdir -p /var/log/minio
sudo chown minio-user:minio-user /opt/minio /var/log/minio

4. Distributed Configuration

Environment Configuration

Create /etc/minio/minio.conf on all nodes:

# MinIO Configuration
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minio123456789  # Change this!

# Distributed setup endpoints
MINIO_OPTS="--console-address :9001"

# For 4-node setup with 4 drives per node
MINIO_VOLUMES="http://minio-node-{1...4}/opt/minio/data{1...4}"

# Alternative: Direct IP addresses
# MINIO_VOLUMES="http://10.0.1.11/opt/minio/data{1...4} http://10.0.1.12/opt/minio/data{1...4} http://10.0.1.13/opt/minio/data{1...4} http://10.0.1.14/opt/minio/data{1...4}"

# TLS Configuration (recommended for production)
# MINIO_SERVER_URL="https://minio.example.com"
# MINIO_BROWSER_REDIRECT_URL="https://console.minio.example.com"

Systemd Service Configuration

Create /etc/systemd/system/minio.service:

[Unit]
Description=MinIO Object Storage Server
Documentation=https://min.io/docs/minio/linux/index.html
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local/

User=minio-user
Group=minio-user
ProtectProc=invisible

EnvironmentFile=-/etc/minio/minio.conf
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/minio/minio.conf\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

# Let systemd restart this service always
Restart=always

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=1048576

# Specifies the maximum number of threads this process can create
TasksMax=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

5. Storage Preparation

On each node, prepare the storage drives:

# Format drives (adjust device paths)
sudo mkfs.xfs /dev/sdb -f
sudo mkfs.xfs /dev/sdc -f
sudo mkfs.xfs /dev/sdd -f
sudo mkfs.xfs /dev/sde -f

# Create mount points
sudo mkdir -p /opt/minio/data{1..4}

# Add to /etc/fstab for persistent mounting
cat >> /etc/fstab << EOF
/dev/sdb /opt/minio/data1 xfs defaults,noatime 0 2
/dev/sdc /opt/minio/data2 xfs defaults,noatime 0 2
/dev/sdd /opt/minio/data3 xfs defaults,noatime 0 2
/dev/sde /opt/minio/data4 xfs defaults,noatime 0 2
EOF

# Mount all drives
sudo mount -a

# Set permissions
sudo chown -R minio-user:minio-user /opt/minio/

6. Starting the Cluster

Start MinIO on all nodes simultaneously:

# Enable and start on all nodes
sudo systemctl daemon-reload
sudo systemctl enable minio
sudo systemctl start minio

# Check status
sudo systemctl status minio
sudo journalctl -u minio -f

7. Load Balancer Configuration

Using HAProxy

global
    daemon
    maxconn 4096

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

# MinIO API Load Balancer
frontend minio_api
    bind *:9000
    default_backend minio_servers

backend minio_servers
    balance roundrobin
    option httpchk GET /minio/health/live
    server minio1 minio-node-1:9000 check
    server minio2 minio-node-2:9000 check
    server minio3 minio-node-3:9000 check
    server minio4 minio-node-4:9000 check

# MinIO Console Load Balancer
frontend minio_console
    bind *:9001
    default_backend minio_console_servers

backend minio_console_servers
    balance roundrobin
    server console1 minio-node-1:9001 check
    server console2 minio-node-2:9001 check
    server console3 minio-node-3:9001 check
    server console4 minio-node-4:9001 check

Using Nginx

upstream minio_servers {
    server minio-node-1:9000;
    server minio-node-2:9000;
    server minio-node-3:9000;
    server minio-node-4:9000;
}

upstream minio_console {
    server minio-node-1:9001;
    server minio-node-2:9001;
    server minio-node-3:9001;
    server minio-node-4:9001;
}

server {
    listen 80;
    server_name minio.example.com;

    # MinIO API
    location / {
        proxy_pass http://minio_servers;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 300;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;
    }
}

server {
    listen 80;
    server_name console.minio.example.com;

    # MinIO Console
    location / {
        proxy_pass http://minio_console;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

8. Verification and Testing

Cluster Health Check

# Using MinIO client
mc admin info myminio

# Check server status
curl http://minio.example.com:9000/minio/health/live

# Detailed cluster info
mc admin info myminio --json

Failover Testing

# Stop one node to test failover
sudo systemctl stop minio

# Verify cluster still responds
mc ls myminio/

# Test file upload/download
mc cp test-file.txt myminio/test-bucket/
mc cp myminio/test-bucket/test-file.txt downloaded-file.txt

# Restart the stopped node
sudo systemctl start minio

9. Production Best Practices

Security

Enable TLS for all communications
Use strong passwords and rotate regularly
Implement IAM policies for access control
Enable audit logging for compliance

Monitoring

Set up Prometheus metrics endpoint
Configure alerts for drive failures
Monitor disk space and performance
Track API response times

Backup Strategy

Configure replication to remote sites
Implement bucket versioning for data protection
Set up lifecycle policies for cost optimization
Regular backup verification and restore testing

Common Issues and Troubleshooting

Clock Synchronization: Ensure all nodes have synchronized time using NTP
Network Connectivity: Verify all nodes can communicate on required ports
Disk Space: Monitor available space and add drives before reaching capacity
Load Balancer Health Checks: Configure proper health check endpoints
Erasure Set Distribution: Ensure drives are evenly distributed across failure domains

This HA configuration provides resilience against node and drive failures while maintaining high performance for your object storage workloads.