How are deletes processed by MinIO and what is their storage utilization impact?

Understanding MinIO’s delete processing mechanism is crucial for capacity planning, performance optimization, and ensuring predictable system behavior during high-deletion workloads.

This question addresses:

• Delete operation mechanics and timing
• Storage utilization during delete processing
• Impact on system performance
• Durability protection during deletes

Answer

MinIO implements a two-phase delete process that balances immediate user response with system stability and durability protection.

Two-Phase Delete Architecture

Phase 1: Instant Delete Marker (< 2ms)

• Creates an S3-compatible delete-marker immediately
• Object becomes inaccessible to clients instantly
• No actual data removal yet
• Minimal IO impact on the fast path

Phase 2: Asynchronous Physical Removal

• Objects are moved to .minio.sys/tmp/.trash
• Background worker performs bulk purging
• Throttled to prevent IO saturation
• Ensures smooth system operation

Delete Processing Flow

1. Client DELETE request
   ↓
2. Create delete marker (< 2ms)
   ↓ [Object now inaccessible]
3. Move to .minio.sys/tmp/.trash (async)
   ↓
4. Background worker bulk purge (throttled)
   ↓
5. Physical space reclaimed

Storage Utilization During Deletes

Temporary Double Storage:

• Deleted objects temporarily consume space in .trash
• Original space not immediately reclaimed
• Gradual space recovery as background worker progresses

Space Utilization Timeline:

Background Worker Throttling

The background purge worker is throttled at ~25 MiB/s per node to:

Benefits of Throttling:

• No ingest-path jitter - writes remain consistent
• Predictable read performance - no IO spikes
• System stability - prevents resource exhaustion
• Fair resource sharing - deletes don’t monopolize IO

Throttling Calculation Example:

16-node cluster:
- Per-node rate: 25 MiB/s
- Cluster rate: 400 MiB/s
- Daily capacity: ~33 TB/day deletion processing

Why Asynchronous Processing?

1. Durability Protection:

• Prevents accidental data loss
• Allows for delete recovery window
• Maintains consistency during failures

2. IO Smoothing:

• Avoids delete storms impacting performance
• Maintains consistent latency for reads/writes
• Prevents disk thrashing

3. System Stability:

• No blocking operations in fast path
• Graceful handling of mass deletes
• Predictable resource consumption

Operational Considerations

Capacity Planning:

# Monitor trash folder size
mc admin info myminio --json | jq '.info.buckets[].size'

# Estimate space needs:
# Peak deletion rate + throttle rate = temporary space required

High-Deletion Workloads:

• Plan for temporary storage overhead
• Monitor .trash folder growth
• Consider deletion patterns in capacity planning

Performance Impact:

• Delete markers: Negligible (< 2ms)
• Background purge: Controlled (25 MiB/s throttle)
• No impact on critical path operations

Best Practices

1. Monitor Trash Usage:

   # Check trash folder size
   mc du myminio/.minio.sys/tmp/.trash

2. Plan for Delete Patterns:

   • Batch deletes spread over time if possible
   • Account for trash space in capacity planning
   • Monitor background worker progress

3. Tune for Workload:

   • Default 25 MiB/s is conservative
   • Can be adjusted for specific needs
   • Balance between reclamation speed and IO impact

Comparison with Other Systems

Real-World Example

Log Rotation Scenario:

• Deleting 10TB of old logs
• Delete markers created instantly (< 2ms each)
• Background purge at 25 MiB/s per node
• 16-node cluster: ~7 hours for complete reclamation
• Zero impact on active log ingestion

This architecture ensures that delete operations never compromise system performance or durability, making MinIO suitable for workloads with unpredictable or high-volume deletion patterns.