Understanding MinIO AIStor’s lifecycle management and cleanup mechanisms is essential for managing storage costs and maintaining system health.
Answer
MinIO uses scanner-driven ILM (Information Lifecycle Management) with worker pools for expiry, transition, and garbage collection. The system automatically evaluates objects against lifecycle rules, executes appropriate actions, and performs background cleanup of stale data including incomplete multipart uploads.
ILM Processing
The lifecycle system supports multiple actions for managing object lifecycles.
Lifecycle Actions
| Action | Description | Use Case |
|---|---|---|
| Delete | Delete current object version | Expire objects after retention period |
| Delete Version | Delete specific version | Remove specific historical versions |
| Delete All Versions | Delete all versions of object | Complete object removal |
| Transition | Move to storage tier | Migrate to cheaper storage |
| Delete Restored | Delete restored copies | Clean up temporary restores from archive |
ILM Processing Flow
┌─────────────────────────────────────────────────────────┐│ ILM Processing │├─────────────────────────────────────────────────────────┤│ ││ Scanner Cycle ││ │ ││ ▼ ││ ┌─────────────────────────────────────────────────┐ ││ │ For each object: │ ││ │ └── Evaluate against bucket lifecycle rules │ ││ └─────────────────────────────────────────────────┘ ││ │ ││ ▼ ││ ┌─────────────────────────────────────────────────┐ ││ │ Rule Match? │ ││ │ ├── No → Skip │ ││ │ └── Yes → Determine action │ ││ └─────────────────────────────────────────────────┘ ││ │ ││ ▼ ││ ┌─────────────────────────────────────────────────┐ ││ │ Queue to appropriate worker pool │ ││ │ ├── Expiry Workers (delete actions) │ ││ │ ├── Transition Workers (tier moves) │ ││ │ └── Cleanup Workers (restored copies) │ ││ └─────────────────────────────────────────────────┘ ││ │└─────────────────────────────────────────────────────────┘Action Details
Delete Actions
Delete (Current Version)└── Removes current version└── Previous versions remain (if versioning enabled)
Delete Version (Specific)└── Removes specified version ID└── Other versions unaffected
Delete All Versions└── Removes all versions including delete markers└── Complete object erasureTransition Action
Transition to Tier │ ▼┌─────────────────────────────────────────────────────────┐│ 1. Read object from source tier ││ 2. Write object to destination tier ││ 3. Verify write success ││ 4. Update metadata with tier reference ││ 5. Remove data from source tier ││ 6. Keep metadata on source for access │└─────────────────────────────────────────────────────────┘Delete Restored
Restored Object (from archive tier) │ └── Temporary copy with expiry │ ▼ After restore period expires │ ▼ Delete Restored action removes temporary copy (Original in archive tier remains)Worker Pool Architecture
ILM operations are distributed across specialized worker pools.
┌─────────────────────────────────────────────────────────┐│ ILM Worker Pools │├─────────────────────────────────────────────────────────┤│ ││ ┌─────────────────┐ ┌─────────────────┐ ││ │ Expiry Workers │ │ Transition │ ││ │ │ │ Workers │ ││ │ • Delete │ │ │ ││ │ • DeleteVersion │ │ • Move to tier │ ││ │ • DeleteAll │ │ • Verify │ ││ └─────────────────┘ └─────────────────┘ ││ ││ ┌─────────────────┐ ┌─────────────────┐ ││ │ Cleanup Workers │ │ GC Workers │ ││ │ │ │ │ ││ │ • Restored │ │ • Multipart │ ││ │ • Temporary │ │ • Orphaned │ ││ └─────────────────┘ └─────────────────┘ ││ │└─────────────────────────────────────────────────────────┘Stale Multipart Cleanup
MinIO automatically cleans up incomplete multipart uploads.
Multipart Storage Location
.minio.sys/multipart/├── bucket1/│ ├── upload-id-1/│ │ ├── part.1│ │ ├── part.2│ │ └── ...│ └── upload-id-2/│ └── ...└── bucket2/ └── ...Cleanup Process
┌─────────────────────────────────────────────────────────┐│ Stale Multipart Cleanup │├─────────────────────────────────────────────────────────┤│ ││ Every 6 hours (default): ││ │ ││ ▼ ││ Scan .minio.sys/multipart/ ││ │ ││ ▼ ││ For each upload: ││ ├── Extract timestamp from upload ID ││ ├── Check against AbortIncompleteMultipartUpload rule ││ │ ││ └── If stale (exceeds DaysAfterInitiation): ││ │ ││ ▼ ││ Move to .trash/ ││ │ ││ ▼ ││ Recursive removal ││ │└─────────────────────────────────────────────────────────┘Cleanup Timing
| Parameter | Value | Description |
|---|---|---|
| Scan Interval | 6 hours[1] | How often cleanup runs (configurable) |
| Stale Threshold | Configurable | DaysAfterInitiation in lifecycle rule |
| Trash Location | .trash/ | Temporary holding before deletion |
Upload ID Timestamp
The upload ID contains an embedded timestamp used to determine age:
Upload ID: <random>-<timestamp>
Extract timestamp → Compare to current time→ If age > DaysAfterInitiation → Mark as staleScanner Integration
The lifecycle evaluator integrates with MinIO’s background scanner.
Batch Evaluation
┌─────────────────────────────────────────────────────────┐│ Scanner + Lifecycle Integration │├─────────────────────────────────────────────────────────┤│ ││ Scanner Process ││ │ ││ ▼ ││ Batch of objects ││ │ ││ ▼ ││ lifecycle.Evaluator ││ ├── Load bucket lifecycle rules ││ ├── For each object in batch: ││ │ ├── Match against filter (prefix, tags, size) ││ │ ├── Check expiration conditions ││ │ ├── Check transition conditions ││ │ └── Determine applicable action ││ └── Return list of actions to execute ││ │ ││ ▼ ││ Queue actions to worker pools ││ │└─────────────────────────────────────────────────────────┘Excessive Versions Alert
MinIO monitors for objects with excessive versions:
| Condition | Threshold | Alert |
|---|---|---|
| High version count | 100 (default)[2] | Warning logged |
| Total size | 1 TB[3] across all versions | Warning logged |
Object Version Check: │ ├── Version count > threshold │ └── Log: "Excessive versions for object X" │ └── Total version size > 1 TB └── Log: "Object X versions exceed 1TB"Why Monitor Excessive Versions?
- Performance impact: Many versions slow listing operations
- Storage costs: Uncontrolled version growth consumes space
- Lifecycle complexity: Large version sets take longer to process
Lifecycle Rule Examples
Expiration Rule
{ "Rules": [ { "ID": "expire-old-objects", "Status": "Enabled", "Filter": { "Prefix": "logs/" }, "Expiration": { "Days": 30 } } ]}Transition Rule
{ "Rules": [ { "ID": "archive-old-data", "Status": "Enabled", "Filter": { "Prefix": "archive/" }, "Transition": { "Days": 90, "StorageClass": "GLACIER" } } ]}Abort Incomplete Multipart
{ "Rules": [ { "ID": "cleanup-incomplete", "Status": "Enabled", "Filter": {}, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } } ]}Garbage Collection
Beyond lifecycle rules, MinIO performs automatic garbage collection.
GC Targets
| Target | Description | Frequency |
|---|---|---|
| Stale multipart | Incomplete uploads | Every 5 minutes |
| Orphaned parts | Parts without parent upload | Scanner cycle |
| Deleted versions | Versions marked for deletion | Background |
| Expired delete markers | Orphaned delete markers | Lifecycle evaluation |
GC Flow
Garbage Collection │ ├── Multipart GC │ └── Clean .minio.sys/multipart/ │ ├── Orphan GC │ └── Remove parts without upload reference │ └── Version GC └── Remove logically deleted versionsMonitoring Lifecycle Operations
Key Metrics
| Metric | Description | Alert Condition |
|---|---|---|
| Expiry queue depth | Pending deletions | Growing continuously |
| Transition queue depth | Pending tier moves | Backlog building |
| Failed transitions | Tier move failures | > 0 |
| Stale multipart count | Incomplete uploads | Unexpected growth |
Best Practices
- Set appropriate thresholds: Configure DaysAfterInitiation based on upload patterns
- Monitor queue depths: Alert on growing ILM backlogs
- Review excessive versions: Investigate objects with many versions
- Test transition rules: Verify tier moves work before applying broadly
- Audit lifecycle rules: Regularly review rules for relevance
Source Code References
cmd/globals.go:108-GlobalStaleUploadsCleanupInterval = time.Hour * 6cmd/data-scanner.go:97-scannerExcessObjectVersions = atomicNewInt64(100)cmd/data-scanner.go:98-scannerExcessObjectVersionsTotalSize = atomicNewInt64(1024 * 1024 * 1024 * 1024) // 1 TB