Kubernetes 节点磁盘空间空了怎么办？解决 containerd overlay 100%问题

Kubernetes 节点磁盘空间空了怎么办解决 containerd overlay 100%问题当 Kubernetes 集群中的节点出现如下错误MountVolume.SetUp failedforvolume xxx: no space left on device或者执行df -h显示:/dev/mapper/centos-root 50G 50G 0G 100% / overlay 50G 50G 0G 100% /run/containerd/io.containerd.runtime...这意味着根分区/空间已满containerd 依赖 overlayfs 已无法写入导致 Pod 创建失败。一、问题原因containerd 将所有容器 rootfs overlay 挂载到/分区/var/lib/containerd,/var/lib/kubelet空间大量占用/var/log 系统日志、Pod 输出日志积历二、快速解决方案清理 containerd 無用容器和镜像# 清理容器ctr-nk8s.io containerslsctr-nk8s.io containersrm$(ctr-nk8s.io containersls-q)# 清理镜像ctr-nk8s.io imageslsctr-nk8s.io imagesrm$(ctr-nk8s.io imagesls-q)清除 kubelet 临时 pod 目录systemctl stop kubeletrm-rf/var/lib/kubelet/pods/* systemctl start kubelet清除系统日志journalctl --vacuum-time1dfind/var/log-typef-exectruncate-s0{}\;三、持久解决方案将 containerd kubelet 数据目录移动到大盘/home步骤 1 停止服务systemctl stop kubelet systemctl stop containerd步骤 2 移动数据 设置软连接mv/var/lib/containerd /home/containerd_datamv/var/lib/kubelet /home/kubelet_data# 建立软连接ln-s/home/containerd_data /var/lib/containerdln-s/home/kubelet_data /var/lib/kubelet步骤 3 重启服务systemctl start containerd systemctl start kubelet步骤 4 确认解决df-h/ kubectl get nodes四、建议配套建议配合 cron 时间件定期清理cat/usr/local/bin/clean-logs.shEOF #!/bin/bash journalctl --rotate journalctl --vacuum-time1d find /var/log -type f -exec truncate -s 0 {} \; EOFchmodx /usr/local/bin/clean-logs.shecho0 3 * * * root /usr/local/bin/clean-logs.sh/etc/crontab如果需要containerd root目录指定把root /var/lib/containerd state /run/containerd替换为# /etc/containerd/config.toml root /home/containerd_data state /home/containerd_statesystemctl daemon-reexec systemctl restart containerd如果需要修改 kubelet 启动参数如使用 systemd编辑/etc/systemd/system/kubelet.service.d/10-kubeadm.conf中加入--root-dir/home/kubelet_data[Service]EnvironmentKUBELET_KUBECONFIG_ARGS--bootstrap-kubeconfig/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig/etc/kubernetes/kubelet.confEnvironmentKUBELET_CONFIG_ARGS--config/var/lib/kubelet/config.yamlEnvironmentFile-/var/lib/kubelet/kubeadm-flags.envEnvironmentFile-/etc/default/kubeletEnvironmentKUBELET_EXTRA_ARGS--node-ip192.168.1.102 --hostname-overrideworker3 --root-dir/home/kubelet_dataExecStartExecStart/usr/local/bin/kubelet$KUBELET_KUBECONFIG_ARGS$KUBELET_CONFIG_ARGS$KUBELET_KUBEADM_ARGS$KUBELET_EXTRA_ARGS结论问题原因解决方案节点/分区 100%containerd kubelet logs 占满移动到/homePod 无法创建overlay rootfs 写入失败重启服务释放空间kubelet 出现 FailedMountpod 目录无法 mkdir移目录 空间利用