Persistent storage on TKGi worker node is almost 80%. noticed…
- User
Persistent storage on TKGi worker node is almost 80%. noticed /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/ is 24G. How can we do cleanup without impacting any running containers
- Assistant
Here’s a safe, non-disruptive recipe to reclaim that 24 GB of overlayfs snapshots on your TKGi worker without touching any live containers:
-
Inspect which snapshots and images are actually in use
# List all snapshots in the k8s.io namespace, show their usage sudo ctr --namespace k8s.io snapshot ls -q | xargs -n1 sudo ctr --namespace k8s.io snapshot info | grep -E 'Key|Ref'You’ll see which snapshots have a non-zero Ref count (i.e. still backing a running container).
-
Remove unused images (thus orphaning old snapshots)
Kubernetes won’t generally do a full image‐prune for you, butcontainerdhas a pruning command:# This will only delete images not referenced by any container sudo ctr --namespace k8s.io images prune --all→ As images go, each deleted image automatically releases its overlayfs layers/snapshots that aren’t in use.
-
Trigger a manual GC pass
After you prune images, you need to garbage-collect the orphaned content. You can either:
a. Send a SIGUSR1 to containerd (built-in GC trigger)
bash PID=$(pidof containerd) sudo kill -USR1 $PID
b. Or run the standalone GC tool against your live config
bash sudo ctr --namespace k8s.io content garbage-collect /etc/containerd/config.toml
Either way, containerd will sweep away those now-unreferenced snapshots. -
Verify reclaimed space
df -h /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots -
(Optional) Automate future cleanups
Edit/etc/containerd/config.tomland under[plugins."io.containerd.snapshotter.v1.overlayfs"]you can tune:[plugins."io.containerd.snapshotter.v1.overlayfs"] # how often to GC (default 10m) gc_period = "5m" # threshold of disk-usage before aggressive GC kicks in gc_unconditional = falseThen
systemctl restart containerd.
Or simply add a cronjob:# every hour: prune + GC 0 * * * * /usr/local/bin/ctr --namespace k8s.io images prune --all \ && /usr/local/bin/ctr --namespace k8s.io content garbage-collect /etc/containerd/config.toml
— This approach only removes images & snapshots that have no active refs, so your running pods remain untouched. Let me know if you hit any snags! 🚀
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.