MinIO Knowledge Base

What network interfaces does MinIO support?

Asked by muratkars Answered by muratkars July 17, 2025
0 views
infrastructure intermediate verified networkingethernetinfinibandrdmarocev2performancegpuhardware

Understanding supported network interfaces is crucial for designing high-performance MinIO deployments, especially for AI/ML workloads, HPC environments, and large-scale object storage systems.

This addresses key infrastructure decisions:

  • Network hardware compatibility
  • Performance optimization options
  • Future-proofing network investments
  • GPU workload optimization

Answer

MinIO supports a comprehensive range of network interfaces, from standard Ethernet to cutting-edge high-speed networking technologies.

Ethernet Support

Full Range of Speeds:

  • 10 GbE - Entry-level datacenter
  • 25 GbE - Modern standard
  • 40 GbE - Previous generation high-speed
  • 100 GbE - Current high-performance standard
  • 200 GbE - Advanced deployments
  • 400 GbE - Cutting-edge infrastructure
  • 800 GbE - Future-ready support

Protocol Support:

  • TCP - Standard, universal compatibility
  • RoCEv2 (RDMA over Converged Ethernet v2) - Low latency, high throughput

InfiniBand Support

  • IPoIB (IP over InfiniBand) - Full compatibility
  • Leverages InfiniBand’s low latency
  • Common in HPC environments
  • Seamless integration with existing IB infrastructure

Revolutionary: S3-over-RDMA

Status: Private Preview

MinIO is pioneering S3-over-RDMA, delivering unprecedented performance:

Performance Achievements:

  • Saturates 400 GbE per storage node - Full bandwidth utilization
  • ~30% CPU load reduction on GPU servers - More compute for AI/ML
  • Direct memory access bypasses CPU
  • Ultra-low latency operations

Network Performance Characteristics

Interface TypeTypical LatencyCPU OverheadUse Case
10 GbE TCP50-100 μsModerateSmall deployments
25 GbE TCP30-50 μsModerateStandard production
100 GbE TCP10-20 μsHighLarge scale
100 GbE RoCEv22-5 μsLowPerformance critical
400 GbE TCP5-10 μsVery HighExtreme scale
400 GbE RDMA1-2 μsMinimalAI/ML, GPU workloads
InfiniBand1-3 μsLowHPC environments

S3-over-RDMA Benefits

For GPU Workloads:

  1. 30% CPU reduction - More cycles for compute
  2. Direct GPU memory access - Potential GPUDirect integration
  3. Reduced latency - Faster model training iterations
  4. Higher throughput - Saturates network capacity

For Storage Performance:

  1. Line-rate performance - Full 400 GbE utilization
  2. Minimal CPU usage - More efficient storage nodes
  3. Lower latency - Sub-microsecond possibilities
  4. Better scaling - Linear performance growth

Network Selection Guidelines

Small/Medium Deployments:

  • 10/25 GbE TCP
  • Cost-effective
  • Standard switches
  • Easy management

Large Production:

  • 100 GbE TCP/RoCEv2
  • Balance of performance and cost
  • Wide vendor support
  • Proven reliability

Performance Critical:

  • 200/400 GbE with RoCEv2
  • AI/ML workloads
  • Real-time analytics
  • Maximum throughput needs

HPC/Research:

  • InfiniBand (IPoIB)
  • Existing IB infrastructure
  • Lowest latency requirements
  • Specialized workloads

Implementation Considerations

RoCEv2 Requirements:

  • Lossless Ethernet fabric
  • Priority Flow Control (PFC)
  • Enhanced Transmission Selection (ETS)
  • Data Center Bridging (DCB) capable switches

RDMA Configuration:

Terminal window
# Check RDMA capabilities
ibv_devinfo


# Configure RoCEv2
echo 4096 > /sys/class/net/eth0/device/rdma/max_mtu


# Verify RDMA performance
ib_write_bw -d mlx5_0

Future-Proofing Strategies

  1. Choose RDMA-capable NICs even if using TCP initially
  2. Plan for 100 GbE minimum for new deployments
  3. Consider 400 GbE for AI/ML infrastructure
  4. Ensure switch compatibility for future protocols

Real-World Performance Examples

Traditional TCP (100 GbE):

  • 8-10 GB/s per node
  • 15-20% CPU utilization
  • 10-20 μs latency

RoCEv2 (100 GbE):

  • 11-12 GB/s per node
  • 5-10% CPU utilization
  • 2-5 μs latency

S3-over-RDMA (400 GbE):

  • 45-48 GB/s per node
  • 3-5% CPU utilization
  • 1-2 μs latency
  • 30% CPU savings on GPU nodes

Key Advantages

MinIO’s network flexibility enables:

  • Investment protection - Support from 10 GbE to 800 GbE
  • Performance optimization - RDMA for critical workloads
  • Cost efficiency - TCP for standard deployments
  • Future readiness - S3-over-RDMA for next-gen requirements

The S3-over-RDMA capability particularly positions MinIO as the ideal storage platform for AI/ML workloads, where every CPU cycle saved on storage operations translates directly to more compute available for model training and inference.

0