How to setup v3 metrics? - MinIO Knowledge Base

Setting up MinIO v3 metrics with Prometheus requires proper authentication and configuration for comprehensive monitoring of your MinIO cluster across all metric categories.

This question covers:

V3 metrics endpoint configuration
Bearer token authentication setup
Prometheus scrape configuration
Kubernetes service discovery
Bucket-specific metrics collection

Answer

MinIO v3 metrics provide enhanced monitoring capabilities with bearer token authentication and organized metric categories for comprehensive cluster observability.

1. Gather MinIO Node Information

Identify Cluster Components:

Hostname or IP address for each MinIO node
Metrics port (commonly 9000 or 9001)
Network accessibility from Prometheus server

V3 Metrics Endpoints: MinIO v3 provides metrics organized into several major categories:

# Core metric categories
/minio/metrics/v3/api          # API request metrics
/minio/metrics/v3/audit        # Audit functionality metrics
/minio/metrics/v3/cluster      # Cluster-wide metrics (config, health, IAM, usage)
/minio/metrics/v3/debug        # Go runtime metrics
/minio/metrics/v3/ilm          # ILM (Lifecycle Management) metrics
/minio/metrics/v3/logger       # Logger webhook metrics
/minio/metrics/v3/notification # Notification functionality metrics
/minio/metrics/v3/replication  # Site and bucket replication metrics
/minio/metrics/v3/scanner      # Scanner metrics
/minio/metrics/v3/system       # System metrics (CPU, memory, drives, network)

2. Configure Authentication

Generate Bearer Token: MinIO v3 metrics require bearer token authentication:

# Configure MinIO server alias
mc alias set myminio http://your-minio-server:9000 admin your-password

# Generate bearer token for v3 metrics
mc admin prometheus generate myminio --api-version v3

Example Output:

scrape_configs:
- job_name: minio-job
  bearer_token: eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoiYWRtaW4iLCJleHAiOjQ5MDA2ODQ0MjF9._irfBI8ODRTyhf4tWYCUEl2aAihPkSD0VCbGR12cmUhLnywkziOIP29tFQ8Zan_HCTDdnoXfhryX4a-8GXPkxQ
  metrics_path: /minio/metrics/v3
  scheme: http
  static_configs:
  - targets: ['minio-1-1:9000']

Verify Token:

# Test bearer token authentication
curl -s -H "Authorization: Bearer your-token" \
  http://minio-node1.example.com:9000/minio/metrics/v3

# Successful response shows MinIO metrics output
# Invalid token returns authentication error

3. Prometheus Configuration

Static Configuration Example

Basic V3 Metrics Setup:

scrape_configs:
  # Main MinIO v3 metrics job
  - job_name: 'minio-v3'
    metrics_path: '/minio/metrics/v3'
    bearer_token: 'your-generated-bearer-token'
    static_configs:
      - targets:
          - 'minio-node1.example.com:9000'
          - 'minio-node2.example.com:9000'
          - 'minio-node3.example.com:9000'
    scrape_interval: 30s
    scrape_timeout: 10s

  # Optional: Separate job for cluster metrics only
  - job_name: 'minio-cluster'
    metrics_path: '/minio/metrics/v3/cluster'
    bearer_token: 'your-generated-bearer-token'
    static_configs:
      - targets:
          - 'minio-node1.example.com:9000'
    scrape_interval: 60s

Advanced Configuration with File Service Discovery

Per-Bucket API Metrics:

scrape_configs:
  # Bucket-specific API metrics
  - job_name: 'minio-bucket-api'
    bearer_token: 'your-generated-bearer-token'
    file_sd_configs:
      - files:
          - /opt/prometheus/minio_buckets.json
    relabel_configs:
      - source_labels: [__metrics_path__]
        target_label: __metrics_path__
    scrape_interval: 30s

Create /opt/prometheus/minio_buckets.json:

[
  {
    "targets": ["minio-node1.example.com:9000"],
    "labels": {
      "__metrics_path__": "/minio/metrics/v3"
    }
  },
  {
    "targets": ["minio-node1.example.com:9000"],
    "labels": {
      "__metrics_path__": "/minio/metrics/v3/bucket/api/production-bucket"
    }
  },
  {
    "targets": ["minio-node1.example.com:9000"],
    "labels": {
      "__metrics_path__": "/minio/metrics/v3/bucket/api/analytics-bucket"
    }
  }
]

4. Kubernetes Deployment Configuration

Service Discovery for K8s Deployments:

scrape_configs:
  - job_name: 'minio-k8s-v3'
    bearer_token: 'eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9...'
    metrics_path: /minio/metrics/v3
    scheme: https
    tls_config:
      insecure_skip_verify: true

    kubernetes_sd_configs:
    - role: pod
      namespaces:
        names:
        - minio-production

    relabel_configs:
    # Filter by MinIO pods
    - source_labels: [__meta_kubernetes_pod_label_app]
      action: keep
      regex: minio

    # Only scrape ready pods
    - source_labels: [__meta_kubernetes_pod_ready]
      action: keep
      regex: "true"

    # Set correct address and port
    - source_labels: [__meta_kubernetes_pod_ip]
      target_label: __address__
      replacement: ${1}:9000

    # Add useful labels
    - source_labels: [__meta_kubernetes_pod_name]
      target_label: pod
    - source_labels: [__meta_kubernetes_namespace]
      target_label: namespace
    - source_labels: [__meta_kubernetes_pod_label_v1_min_io_pool]
      target_label: minio_pool

5. Security and Best Practices

Token Security:

# Rotate bearer tokens periodically
mc admin prometheus generate myminio --api-version v3

# Store tokens securely
# - Use Kubernetes secrets for K8s deployments
# - Use environment variables for Docker deployments
# - Avoid committing tokens to version control

Network Security:

# Use HTTPS in production
scheme: https
tls_config:
  cert_file: /path/to/client.crt
  key_file: /path/to/client.key
  ca_file: /path/to/ca.crt

6. Verification and Troubleshooting

Check Prometheus Targets:

Visit Prometheus UI: http://prometheus-server:9090/targets
Verify all MinIO nodes show as “UP”
Check for authentication errors

Test Metrics Collection:

# Query v3 metrics in Prometheus
minio_cluster_capacity_usable_total_bytes

# Check for specific v3 metric categories
minio_api_requests_total
minio_system_memory_used
minio_cluster_health_status

Common Issues:

Authentication failures: Regenerate bearer token
Network connectivity: Verify firewall rules and port access
Missing metrics: Check endpoint paths and bucket specifications
High cardinality: Use specific metric paths to reduce data volume

7. Grafana Dashboard Integration

Pre-built V3 Dashboards:

Cluster Overview Dashboard:

URL: MinIO Dashboard v3
Features: Cluster health, capacity, performance metrics
Usage: Import JSON into Grafana for instant visualization

Bucket-Specific Dashboard:

URL: MinIO Bucket Dashboard v3
Features: Per-bucket API metrics, request rates, error tracking
Usage: Monitor individual bucket performance and usage

Dashboard Setup:

# Import dashboard via Grafana UI
1. Go to Grafana → Dashboards → Import
2. Upload JSON file or paste URL
3. Configure data source (Prometheus)
4. Customize variables and thresholds

8. Metric Categories and Use Cases

Essential Monitoring Setup:

# Comprehensive monitoring configuration
scrape_configs:
  # General cluster metrics
  - job_name: 'minio-cluster'
    metrics_path: '/minio/metrics/v3/cluster'
    bearer_token: 'your-token'
    static_configs:
      - targets: ['minio-cluster:9000']

  # System performance metrics
  - job_name: 'minio-system'
    metrics_path: '/minio/metrics/v3/system'
    bearer_token: 'your-token'
    static_configs:
      - targets: ['minio-node1:9000', 'minio-node2:9000']

  # API request metrics
  - job_name: 'minio-api'
    metrics_path: '/minio/metrics/v3/api'
    bearer_token: 'your-token'
    static_configs:
      - targets: ['minio-cluster:9000']

Key Advantages

MinIO v3 metrics provide:

Enhanced security - Bearer token authentication
Organized metrics - Category-specific endpoints
Scalable monitoring - Efficient data collection
Comprehensive coverage - All aspects of MinIO operations
Kubernetes native - Built-in service discovery support
Production ready - Battle-tested in enterprise environments

This v3 metrics setup ensures comprehensive monitoring with proper authentication, organized data collection, and enterprise-grade security for production MinIO deployments.