Can i delete volume snaps in /data/persistant-storage/replicas/pvc/volumesnap to reduce…
- User
Can i delete volume snaps in /data/persistant-storage/replicas/pvc/volumesnap to reduce my disk pressure
Assistant
You can potentially delete volume snapshots in that directory to free up disk space, but you should be very careful and consider the following:
Before deleting anything:
-
Identify what's using these snapshots - Check if this is part of a storage system like OpenEBS, Longhorn, or another CSI driver. Different systems have different approaches to snapshot management.
-
Check if they're actively in use - Some snapshots might be referenced by active volumes or restore operations. Deleting them could cause data loss or system instability.
-
Use the proper deletion method - Instead of manually deleting files, use the appropriate tools:
- For Kubernetes volume snapshots:
kubectl delete volumesnapshot <snapshot-name> - For the specific storage driver's CLI tools
- Through the storage system's management interface
- For Kubernetes volume snapshots:
Safe approaches:
-
List and inspect first:
ls -lah /data/persistant-storage/replicas/pvc/volumesnap # Check age, size, and any lock files -
Look for orphaned or old snapshots - These are usually safer to remove
-
Check your backup/snapshot retention policy - Ensure you're not deleting snapshots needed for recovery
-
Start with old, clearly unused snapshots - Don't delete recent ones
Alternative solutions:
- Configure automatic snapshot cleanup policies in your storage system
- Move snapshots to cheaper storage if you need to retain them
- Compress snapshots if your system supports it
If you're unsure about your specific storage system or which snapshots are safe to delete, it would be best to check your storage system's documentation or consult with your team before proceeding.
-
- User
is it safe to remove the volume-snap .img files?
Assistant
No, it's generally NOT safe to directly remove .img files from the volume-snap directory.
Here's why:
Risks of manual deletion:
- Metadata inconsistency - The storage system likely tracks these snapshots in a database or metadata files. Deleting the .img files directly will leave orphaned references
- Active references - Some volumes might still be using these snapshots as a base layer
- Broken snapshot chains - If snapshots are incremental, removing one could break the entire chain
- System corruption - The storage controller might crash or behave unpredictably when it can't find expected files
Instead, you should:
-
Use the storage system's commands to list and delete snapshots:
# First identify what storage system you're using # Then use its CLI, for example: # For OpenEBS: mayactl snapshot delete # For Longhorn: kubectl delete volumesnapshot -
Check for garbage collection settings - Many storage systems have automatic cleanup for old snapshots
-
If you must free space urgently:
- First check which snapshots are oldest:
ls -lt *.img | tail -20 - Verify they're not referenced: Check your storage system's metadata/database
- Make backups of critical data first
- Test on a non-production system if possible
- First check which snapshots are oldest:
-
Look for proper cleanup procedures in your storage system's documentation
What storage system are you using (OpenEBS, Longhorn, etc.)? I can provide more specific guidance based on that.
- User
I dont have kubectl in my worker node but my worker node is where the problem is, can i do it for my worker on my control node?
Assistant
Yes, you can absolutely manage volume snapshots on your worker node from your control node using kubectl! This is actually the standard way to do it.
From your control node:
-
List volume snapshots and their locations:
# See all volume snapshots kubectl get volumesnapshot -A # Get detailed info including which node they're on kubectl get pv -o wide kubectl get pvc -A -o wide -
Find snapshots on the specific worker node:
# Replace <worker-node-name> with your actual node name kubectl get pods -A -o wide --field-selector spec.nodeName=<worker-node-name> # Check PVs on that node kubectl get pv -o json | jq '.items[] | select(.spec.nodeAffinity.required.nodeSelectorTerms[].matchExpressions[].values[] | contains("<worker-node-name>")) | .metadata.name' -
Delete volume snapshots:
# Delete specific snapshot kubectl delete volumesnapshot <snapshot-name> -n <namespace> # Delete multiple old snapshots (be careful!) kubectl get volumesnapshot -n <namespace> --sort-by=.metadata.creationTimestamp | head -n 10 -
Monitor disk usage on the worker:
# Check disk usage on specific node kubectl get --raw /api/v1/nodes/<worker-node-name>/proxy/stats/summary | jq '.node.fs'
Important notes:
- Deleting via kubectl triggers the proper cleanup process through your CSI driver
- The actual .img files should be removed automatically after kubectl deletion
- If files remain after kubectl deletion, your storage driver might need a garbage collection run
Which storage driver are you using? This will help determine if there are any specific cleanup commands needed.
-
- User
Can I delete the snapshots directly in the volume from the Rancher UI?
Assistant
Yes, you can delete volume snapshots through the Rancher UI! This is actually a safer method than manually deleting files on the worker node.
Here's how to do it in Rancher:
-
Navigate to Storage section:
- Go to your cluster in Rancher
- Click on "Storage" in the left menu
- Select "Volume Snapshots" or "Snapshots"
-
Find and delete snapshots:
- You'll see a list of all volume snapshots
- Look for columns showing:
- Snapshot name
- Source PVC
- Creation time
- Size
- Select the snapshots you want to delete (usually older ones)
- Click the three dots menu (⋮) and choose "Delete"
-
Alternative path through Volumes:
- Go to "Storage" → "Persistent Volumes" or "PVCs"
- Find the volume consuming space
- Click on it to see associated snapshots
- Delete snapshots from there
Benefits of using Rancher UI:
- Properly triggers the CSI driver's cleanup process
- Shows you which snapshots are safe to delete
- Prevents accidental deletion of active snapshots
- Updates all metadata correctly
After deletion:
- The storage driver should automatically clean up the .img files
- Monitor your worker node's disk usage to confirm space is freed
- If space isn't freed immediately, the storage driver might run garbage collection on a schedule
Tip: Sort snapshots by creation date to easily identify old ones that are likely safe to delete.
Do you see the Volume Snapshots section in your Rancher UI?
-
- User
Can I delete the snapshot directly from the volume in the longhorn UI?
Assistant
Yes! The Longhorn UI is actually the best and safest way to delete snapshots for Longhorn volumes. This is much safer than deleting .img files directly.
Here's how to delete snapshots in Longhorn UI:
-
Access Longhorn UI:
- In Rancher: Apps → Installed Apps → Find Longhorn → Click on the Longhorn URL
- Or directly access:
https://<your-cluster>/k8s/clusters/<cluster-id>/api/v1/namespaces/longhorn-system/services/http:longhorn-frontend:80/proxy/
-
Navigate to the volume:
- Click on "Volume" in the top menu
- Find the volume taking up space on your worker node
- Click on the volume name to see details
-
View and delete snapshots:
- In the volume detail page, you'll see a "Snapshots" tab or section
- You'll see a snapshot chain showing:
- Snapshot names
- Creation time
- Size
- Whether it's the "Volume Head" (active data)
-
Delete snapshots:
- Click the trash icon next to old snapshots
- Don't delete:
- The "Volume Head" (current active data)
- Recent snapshots you might need
- Safe to delete:
- Old snapshots
- Snapshots marked as "Removed" but still consuming space
Longhorn specific benefits:
- Shows actual space usage per snapshot
- Handles snapshot chain dependencies automatically
- Triggers proper cleanup on the worker node
- Shows which node the replica is on
After deletion:
- Longhorn will automatically clean up the .img files
- Space should be freed within minutes
- You can monitor the volume size decrease in the UI
The Longhorn UI gives you the clearest view of what's safe to delete and will properly clean up your worker node's disk space.
-
- User
I am using Longhorn version v1.3.2
Assistant
Perfect! Longhorn v1.3.2 has a straightforward UI for managing snapshots. Here's how to delete snapshots in that version:
In Longhorn v1.3.2 UI:
-
Access the Longhorn UI
- From Rancher or directly via the Longhorn service URL
-
Go to Volumes:
- Click "Volume" in the top navigation
- Find the volume using space on your worker node
- Click on the volume name
-
In the Volume Detail page:
- You'll see the Snapshot List section
- Each snapshot shows:
- Name
- Created time
- Size
- "Remove" button (trash icon)
- Labels indicating if it's the "Volume Head"
-
Safe deletion process:
- DO NOT delete the snapshot marked as "volume-head" - this is your active data
- Start with the oldest snapshots
- Click the trash icon (Remove) next to snapshots you want to delete
- Confirm the deletion
-
Space reclamation:
- In v1.3.2, space is reclaimed after deletion, but it might take a few minutes
- Check the "Size" column in the volume list to see the space being freed
- The volume's "Actual Size" should decrease
Important for v1.3.2:
- This version handles snapshot deletion well, but make sure your recurring snapshot jobs aren't creating new ones faster than you're deleting
- You can also check Settings → General → "Snapshot Data Integrity" settings
To verify space is freed on your worker:
# From control node, check the specific node's disk usage kubectl top nodesIs the Longhorn UI showing the snapshots for your problematic volume?
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.