Skip to content
The Chat Archive
o4-mini-2025-04-16· OpenAI LMArena Human Preference 140K

Persistent storage on TKGi worker node is almost 80%. noticed…

  1. User

    Persistent storage on TKGi worker node is almost 80%. noticed /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/ is 24G. How can we do cleanup without impacting any running containers

  2. Assistant

    Here’s a safe, non-disruptive recipe to reclaim that 24 GB of overlayfs snapshots on your TKGi worker without touching any live containers:

    1. Inspect which snapshots and images are actually in use

      # List all snapshots in the k8s.io namespace, show their usage
      sudo ctr --namespace k8s.io snapshot ls -q | xargs -n1 sudo ctr --namespace k8s.io snapshot info | grep -E 'Key|Ref'
      

      You’ll see which snapshots have a non-zero Ref count (i.e. still backing a running container).

    2. Remove unused images (thus orphaning old snapshots)
      Kubernetes won’t generally do a full image‐prune for you, but containerd has a pruning command:

      # This will only delete images not referenced by any container
      sudo ctr --namespace k8s.io images prune --all
      

      → As images go, each deleted image automatically releases its overlayfs layers/snapshots that aren’t in use.

    3. Trigger a manual GC pass
      After you prune images, you need to garbage-collect the orphaned content. You can either:
      a. Send a SIGUSR1 to containerd (built-in GC trigger)
      bash PID=$(pidof containerd) sudo kill -USR1 $PID
      b. Or run the standalone GC tool against your live config
      bash sudo ctr --namespace k8s.io content garbage-collect /etc/containerd/config.toml
      Either way, containerd will sweep away those now-unreferenced snapshots.

    4. Verify reclaimed space

      df -h /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots
      
    5. (Optional) Automate future cleanups
      Edit /etc/containerd/config.toml and under [plugins."io.containerd.snapshotter.v1.overlayfs"] you can tune:

      [plugins."io.containerd.snapshotter.v1.overlayfs"]
        # how often to GC (default 10m)
        gc_period = "5m"
        # threshold of disk-usage before aggressive GC kicks in
        gc_unconditional = false
      

      Then systemctl restart containerd.
      Or simply add a cronjob:

      # every hour: prune + GC
      0 * * * * /usr/local/bin/ctr --namespace k8s.io images prune --all \
                && /usr/local/bin/ctr --namespace k8s.io content garbage-collect /etc/containerd/config.toml
      

    — This approach only removes images & snapshots that have no active refs, so your running pods remain untouched. Let me know if you hit any snags! 🚀

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.