Skip to content
TheChatArchiveThe Chat Archive
Qwenqwen-max-2025-01-25· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

Persistent storage on TKGi worker node is almost 80%. noticed…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with o4-mini-2025-04-16’s answer.
  1. User

    Persistent storage on TKGi worker node is almost 80%. noticed /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/ is 24G. How can we do cleanup without impacting any running containers

  2. Qwen Assistant

    To address the issue of persistent storage on a TKGi (Tanzu Kubernetes Grid Integrated Edition) worker node being almost 80% full, particularly due to the large size of the /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/ directory, it is crucial to clean up unused or orphaned resources without impacting running containers. Below is a structured approach to safely perform this cleanup:


    1. Understand the Problem

    The directory /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/ contains snapshots used by containerd for managing container filesystem layers. Over time, unused or orphaned snapshots can accumulate, consuming significant disk space.

    2. Precautions Before Cleanup

    • Do not manually delete files: Directly removing files from the snapshots directory can corrupt the container runtime and disrupt running containers.
    • Backup critical data: Ensure that any important data on the node is backed up before proceeding.
    • Test in a non-production environment: If possible, test the cleanup process in a staging environment to ensure safety.

    3. Steps to Safely Clean Up Unused Resources

    Step 1: Identify Unused Snapshots

    Use the ctr command-line tool (part of containerd) to list and inspect snapshots:

    ctr -n k8s.io snapshots list
    

    This command lists all snapshots managed by containerd. Look for snapshots that are not associated with any running containers.

    Step 2: Remove Unused Container Images

    Unused container images can also contribute to disk usage. Use the following commands to identify and remove unused images:

    1. List all images:
      ctr -n k8s.io images list
      
    2. Remove unused images:
      ctr -n k8s.io images prune
      
      This command removes images that are not referenced by any container.

    Step 3: Remove Stopped Containers

    Stopped containers may still hold references to snapshots. Use the following commands to identify and remove them:

    1. List all containers:
      ctr -n k8s.io containers list
      
    2. Remove stopped containers:
      ctr -n k8s.io containers rm <container-id>
      

    Step 4: Prune Unused Snapshots

    After cleaning up unused images and containers, you can prune unused snapshots:

    ctr -n k8s.io snapshots prune
    

    This command removes snapshots that are no longer referenced by any container or image.

    Step 5: Verify Disk Usage

    After performing the cleanup, verify the disk usage to ensure the space has been reclaimed:

    du -sh /var/vcap/store/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/
    df -h
    

    4. Automate Cleanup (Optional)

    To prevent future disk space issues, consider automating the cleanup process using a cron job or a Kubernetes job. For example:

    #!/bin/bash
    ctr -n k8s.io images prune -q
    ctr -n k8s.io containers rm $(ctr -n k8s.io containers list -q | grep -v "RUNNING")
    ctr -n k8s.io snapshots prune
    

    Schedule this script to run periodically using cron.


    5. Monitor Disk Usage

    Set up monitoring and alerts for disk usage on your worker nodes to proactively address similar issues in the future. Tools like Prometheus and Grafana can be integrated with TKGi for this purpose.


    6. Restart the Node (If Necessary)

    If the cleanup does not fully resolve the issue, consider restarting the worker node as a last resort. However, ensure that:

    • The node is cordoned and drained (kubectl cordon and kubectl drain) to prevent disruption to running workloads.
    • The node is uncordoned and rescheduled after the restart.

    7. Summary

    By following the steps above, you can safely clean up unused resources in the snapshots directory without impacting running containers. The key is to use the ctr tool to manage containerd resources and avoid manual deletion of files.

    If you encounter any specific errors or issues during the cleanup process, please provide additional details so I can assist further.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.