Kubernetes（1.31 到 1.35）集群、CRI containerd（1.7 到 2.3）升级流程与问题记录

Kubernetes 升级1.版本升级规划K8s 官方文档明确说明Skipping MINOR versions when upgrading is unsupported也就是说从v1.31.14升级到v1.35.6不能一步到位必须逐个版本升级共经历四跳v1.31.14 - v1.32.x - v1.33.x - v1.34.x - v1.35.62.版本升级流程由于不能跳版本v1.31 - v1.35本质上是把单次升级流程重复执行四遍以下步骤针对每个控制平面节点每完成一个版本升级后所有节点同步升级再进入下一个版本升级流程。2.1.腾空节点只对当前升级的节点进行kubectl drain xxx \ --ignore-daemonsets \ --delete-emptydir-data \ --timeout300s2.2.备份 ETCD在所有控制平面节点上执行。ETCDCTL_API3 etcdctl --endpointshttps://127.0.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ snapshot save /root/backup/etcd-$(date %F-%H%M).db执行成功后输出一堆info日志最终看到Snapshot saved at ...就备份成功了{level:info,ts:xxxx-xx-xx,caller:snapshot/v3_snapshot.go:65,msg:created temporary db file,path:/backup/etcd-xxxx-xx-xx-xxxx.db.part} {level:info,ts:xxxx-xx-xx,logger:client,caller:v3v3.5.21/maintenance.go:212,msg:opened snapshot stream; downloading} {level:info,ts:xxxx-xx-xx,caller:snapshot/v3_snapshot.go:88,msg:fetched snapshot,endpoint:https://127.0.0.1:2379,size:xx MB,took:xx second ago} {level:info,ts:xxxx-xx-xx,caller:snapshot/v3_snapshot.go:97,msg:saved,path:/backup/etcd-xxxx-xx-xx-xxxx.db} Snapshot saved at /backup/etcd-x.x.x-xxxx.db2.3.更新软件源当前环境使用清华源。每次版本升级都需要把旧版本源注释或清理换成新版本源并下载对应的仓库公钥。在所有控制平面节点更新# 下载前先将旧版本公钥改名备份一下,等升级结束后再清理 curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key \ | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg # 配置新版本源 # $K8S_VERSION: 具体要升级的版本 cat /etc/apt/sources.list.d/kubernetes.list deb [signed-by/etc/apt/keyrings/kubernetes-apt-keyring.gpg] http://mirrors4.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/$K8S_VERSION/deb/ / # 更新源 apt-get update # 列出当前可用版本 apt-cache madison kubeadm | head -n 1 # 示例输出 kubeadm | 1.32.13-1.1 | http://mirrors4.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.32/deb Packages apt-cache madison kubelet | head -n 1 apt-cache madison kubectl | head -n 1 # 确认组件未被 apt 锁定(hold) apt-mark showhold | grep -Ei kubeadm|kubelet|kubectl2.4.验证升级计划检查可升级到哪些版本并验证当前集群是否可升级。该命令只能在存在 kubeconfig 文件admin.conf的控制平面节点上运行。# 安装目标版本的 kubeadm apt-get install -y kubeadm1.32.13-1.1 # 确认版本 kubeadm version # 验证升级计划: kubeadm 会列出当前版本/目标版本/各组件变更 kubeadm upgrade plan # 输出内容(忽略输出的版本信息) [preflight] Running pre-flight checks. [upgrade/config] Reading configuration from the kubeadm-config ConfigMap in namespace kube-system... [upgrade/config] Use kubeadm init phase upload-config --config your-config-file to re-upload it. [upgrade] Running cluster health checks [upgrade] Fetching available versions to upgrade to [upgrade/versions] Cluster version: 1.32.13 [upgrade/versions] kubeadm version: v1.33.13 I0622 18:27:10.934338 3715038 version.go:261] remote version is much newer: v1.36.2; falling back to: stable-1.33 [upgrade/versions] Target version: v1.33.13 [upgrade/versions] Latest version in the v1.32 series: v1.32.13 Components that must be upgraded manually after you have upgraded the control plane with kubeadm upgrade apply: COMPONENT NODE CURRENT TARGET kubelet x.x.x v1.32.13 v1.33.13 kubelet x.x.x v1.32.13 v1.33.13 kubelet x.x.x v1.32.13 v1.33.13 Upgrade to the latest stable version: COMPONENT NODE CURRENT TARGET kube-apiserver x.x.x v1.32.13 v1.33.13 kube-apiserver x.x.x v1.32.13 v1.33.13 kube-apiserver x.x.x v1.32.13 v1.33.13 kube-controller-manager x.x.x v1.32.13 v1.33.13 kube-controller-manager x.x.x v1.32.13 v1.33.13 kube-controller-manager x.x.x v1.32.13 v1.33.13 kube-scheduler x.x.x v1.32.13 v1.33.13 kube-scheduler x.x.x v1.32.13 v1.33.13 kube-scheduler x.x.x v1.32.13 v1.33.13 kube-proxy 1.32.13 v1.33.13 CoreDNS v1.11.3 v1.12.0 etcd x.x.x 3.5.24-0 3.5.24-0 etcd x.x.x 3.5.24-0 3.5.24-0 etcd x.x.x 3.5.24-0 3.5.24-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.33.13 _____________________________________________________________________ The table below shows the current state of component configs as understood by this version of kubeadm. Configs that have a yes mark in the MANUAL UPGRADE REQUIRED column require manual config upgrade or resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually upgrade to is denoted in the PREFERRED VERSION column. API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED kubeproxy.config.k8s.io v1alpha1 v1alpha1 no kubelet.config.k8s.io v1beta1 v1beta1 no2.5.关闭 kube-apiserver 连接因为kube-apiserver/etcd静态 Pod 始终在运行所以执行 kubeadm 升级时对服务器正在进行的请求将停滞。可以在运行kubeadm upgrade apply命令前主动停止kube-apiserver进程几秒钟。这样可以能让正在进行的请求完成处理并关闭现有连接最大限度地减少 etcd 停机的后果killall -s SIGTERM kube-apiserver2.6.集群升级2.6.1.控制节点升级该命令只需执行一次且仅可在具有 kubeconfig 文件admin.conf的控制平面节点上运行。它的作用是preflight升级前预检查control-plane升级本节点上的控制平面实例upload-config将 kubeadm 和 kubelet 配置上传到 ConfigMap集群级唯一3.1. /kubeadm将 ClusterConfiguration 写入 kubeadm-config ConfigMap3.2. /kubelet将 kubelet 配置写入 kubelet-config ConfigMapkubelet-config升级本节点的 kubelet 配置bootstrap-token配置 bootstrap token 和 cluster-info RBAC 规则集群级唯一addon升级默认插件CoreDNS、kube-proxypost-upgrade升级后任务。# 第一个控制平面节点用 apply kubeadm upgrade apply 1.32.13 # 输出内容 [upgrade] Reading configuration from the kubeadm-config ConfigMap in namespace kube-system... [upgrade/preflight] Running preflight checks [upgrade] Running cluster health checks [upgrade/preflight] You have chosen to upgrade the cluster version to v1.32.13 [upgrade/versions] Cluster version: v1.31.14 [upgrade/versions] kubeadm version: v1.32.13 [upgrade] Are you sure you want to proceed? [y/N]: y [upgrade/preflight] Pulling images required for setting up a Kubernetes cluster [upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection # 这里告警提示: 基础镜像版本不一致 # 参考 问题处理 1.1 基础镜像版本不一致 解决 W0622 16:00:12 checks.go:843] detected that the sandbox image registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8 of the container runtime is inconsistent with that used by kubeadm. It is recommended to use registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.10 as the CRI sandbox image. [upgrade/control-plane] Upgrading your Static Pod-hosted control plane to version v1.32.13 (timeout: 5m0s)... [upgrade/staticpods] Preparing for etcd upgrade [upgrade/staticpods] Renewing etcd-server / etcd-peer / etcd-healthcheck-client certificate [upgrade/staticpods] Moving new manifest to .../etcd.yaml and backing up old manifest [upgrade/staticpods] Component etcd upgraded successfully! [upgrade/etcd] Waiting for etcd to become available [upgrade/staticpods] Preparing for kube-apiserver upgrade [upgrade/staticpods] Renewing apiserver / apiserver-kubelet-client / front-proxy-client / apiserver-etcd-client certificate [upgrade/staticpods] Component kube-apiserver upgraded successfully! [upgrade/staticpods] Component kube-controller-manager upgraded successfully! [upgrade/staticpods] Component kube-scheduler upgraded successfully! [upgrade/control-plane] The control plane instance for this node was successfully upgraded! [upload-config] Storing the configuration used in ConfigMap kubeadm-config in the kube-system Namespace [kubelet] Creating a ConfigMap kubelet-config in namespace kube-system [upgrade/kubeconfig] The kubeconfig files for this node were successfully upgraded! [upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded! [upgrade/addon] Skipping upgrade of addons because control plane instances [x.x.x x.x.x] have not been upgraded [upgrade] SUCCESS! A control plane node of your cluster was upgraded to v1.32.13. [upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.2.6.2.其余节点升级在上面某台 master 节点执行 apply 后执行该命令。与apply相比node没有upload-config和bootstrap-token这两个集群级阶段这正是它可以在多个节点上重复执行的原因。在control-plane阶段当前节点是控制平面就升级控制平面实例是工作节点则跳过因此该命令可同时用于两种节点Upgrade the control plane instance deployed on this node,if any# 其余控制节点/Node 节点 kubeadm upgrade node # 输出内容 [upgrade] Reading configuration from the kubeadm-config ConfigMap in namespace kube-system... [upgrade] Use kubeadm init phase upload-config --config your-config-file to re-upload it. [upgrade/preflight] Running pre-flight checks [upgrade/preflight] Pulling images required for setting up a Kubernetes cluster [upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection [upgrade/preflight] You can also perform this action beforehand using kubeadm config images pull W0622 18:36:03.656485 1715747 checks.go:843] detected that the sandbox image registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8 of the container runtime is inconsistent with that used by kubeadm.It is recommended to use registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.10 as the CRI sandbox image. [upgrade/control-plane] Upgrading your Static Pod-hosted control plane instance to version v1.33.13... [upgrade/staticpods] Writing new Static Pod manifests to /etc/kubernetes/tmp/kubeadm-upgraded-manifests2277431111 [upgrade/staticpods] Preparing for etcd upgrade [upgrade/staticpods] Renewing etcd-server certificate [upgrade/staticpods] Renewing etcd-peer certificate [upgrade/staticpods] Renewing etcd-healthcheck-client certificate [upgrade/staticpods] Restarting the etcd static pod and backing up its manifest to /etc/kubernetes/tmp/kubeadm-backup-manifests-2026-06-22-18-36-49/etcd.yaml [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This can take up to 5m0s [apiclient] Found 3 Pods for label selector componentetcd [upgrade/staticpods] Component etcd upgraded successfully! [upgrade/etcd] Waiting for etcd to become available [upgrade/staticpods] Preparing for kube-apiserver upgrade [upgrade/staticpods] Renewing apiserver certificate [upgrade/staticpods] Renewing apiserver-kubelet-client certificate [upgrade/staticpods] Renewing front-proxy-client certificate [upgrade/staticpods] Renewing apiserver-etcd-client certificate [upgrade/staticpods] Moving new manifest to /etc/kubernetes/manifests/kube-apiserver.yaml and backing up old manifest to /etc/kubernetes/tmp/kubeadm-backup-manifests-2026-06-22-18-36-49/kube-apiserver.yaml [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This can take up to 5m0s [apiclient] Found 3 Pods for label selector componentkube-apiserver [upgrade/staticpods] Component kube-apiserver upgraded successfully! [upgrade/staticpods] Preparing for kube-controller-manager upgrade [upgrade/staticpods] Renewing controller-manager.conf certificate [upgrade/staticpods] Moving new manifest to /etc/kubernetes/manifests/kube-controller-manager.yaml and backing up old manifest to /etc/kubernetes/tmp/kubeadm-backup-manifests-2026-06-22-18-36-49/kube-controller-manager.yaml [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This can take up to 5m0s [apiclient] Found 3 Pods for label selector componentkube-controller-manager [upgrade/staticpods] Component kube-controller-manager upgraded successfully! [upgrade/staticpods] Preparing for kube-scheduler upgrade [upgrade/staticpods] Renewing scheduler.conf certificate [upgrade/staticpods] Moving new manifest to /etc/kubernetes/manifests/kube-scheduler.yaml and backing up old manifest to /etc/kubernetes/tmp/kubeadm-backup-manifests-2026-06-22-18-36-49/kube-scheduler.yaml [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This can take up to 5m0s [apiclient] Found 3 Pods for label selector componentkube-scheduler [upgrade/staticpods] Component kube-scheduler upgraded successfully! [upgrade/control-plane] The control plane instance for this node was successfully upgraded! ## 此告警是指配置中 controlPlaneEndpoint 与实际 bindPort 不一致 ## 参考 问题处理 1.3 处在本文中是符合预期的 W0622 18:38:43.473778 1715747 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [upgrade/kubeconfig] The kubeconfig files for this node were successfully upgraded! W0622 18:38:50.167686 1715747 postupgrade.go:117] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config2351329468 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR [upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config2351329468/config.yaml [kubelet-start] Writing kubelet configuration to file /var/lib/kubelet/config.yaml [upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded! [addons] Applied essential addon: CoreDNS W0622 18:38:50.338251 1715747 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [addons] Applied essential addon: kube-proxy2.7.升级 kubelet 与 kubectlapt-get install -y kubelet1.32.13-1.1 kubectl1.32.13-1.1 # 升级后会自动重启 kubelet containerd(可通过日志观察) # systemctl daemon-reload # systemctl restart kubelet/containerd2.8.恢复节点调度并验证kubectl uncordon xxx kubectl get node -o wide确认节点VERSION更新后登录下一个控制平面节点重复上述步骤。注上述 2.1 2.8 步要在所有控制节点上都走一遍且四要分别四次执行版本升级即整体循环四次...Containerd 升级K8s 集群升级后参考 containerd 官网版本对应关系计划将 containerd 升到 2.3.2。虽然 Kubernetes test grid 中 1.35 未覆盖 containerd 2.3.x但二者 CRI Version 一致且官方升级路径支持 LTS 跨版本升级因此本次直接从 1.7.27 直接升级到 2.3.2Upgrades are supported for sequential minor releases. For example, an upgrade from 2.0 to 2.1 is supported, but an upgrade from 2.0 to 2.2 is not. Patch releases are always backward compatible with their minor version.In addition to sequential minor release upgrades, direct upgrades between sequential LTS (Long Term Stable) releases are also supported. For example, a direct upgrade from 1.7 (LTS) to 2.3 (LTS) will be tested and supported, but 1.7 (LTS) to 2.6 (LTS, tentatively) will not. This allows users who prefer to stay on LTS releases to have a clear and safe upgrade path.升级前的状态# containerd --version containerd github.com/containerd/containerd v1.7.27 05044ec0a9a75232cad458027ca83437aae3f4da # runc --version runc version 1.4.3 commit: v1.4.3-0-gbb14dabeb spec: 1.3.0 go: go1.25.11 libseccomp: 2.6.0 # kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME x.x.x Ready control-plane 346d v1.35.6 x.x.x.x Ubuntu 24.04.4 LTS 6.8.0-124-generic containerd://2.3.2 x.x.x Ready control-plane 346d v1.35.6 x.x.x.x Ubuntu 24.04.4 LTS 6.8.0-124-generic containerd://2.3.2 x.x.x Ready control-plane 346d v1.35.6 x.x.x.x Ubuntu 24.04.4 LTS 6.8.0-124-generic containerd://2.3.21.升级前检查先查询当前版本是否有废弃配置 / API 提示# ctr deprecations list # ctr deprecations list --format json从 containerd 2.0 开始官方废弃了cri-containerd-(cni-)VERSION-OS-ARCH.tar.gz全量安装包。他最初是面向 k8s 提供的把 containerd、runc 和 CNI 打包在一起无需再分别下载各个组件便于快速安装。不过自 k8s 1.24 移除 Dockershim 起containerd 社区也在讨论中决定不再维护全量包。在 2.0 版本后改为各组件由用户按需安装依赖。因此从 2.0 开始containerd 作为 K8s CRI 时需要分别安装以下三个组件组件下载地址containerdhttps://github.com/containerd/containerd/releasesrunchttps://github.com/opencontainers/runc/releasesCNI pluginshttps://github.com/containernetworking/plugins/releases2.升级流程所有 k8s 节点依次完成 2.1 2.7 中所有操作2.1.腾空节点kubectl cordon xxx kubectl drain xxx --ignore-daemonsets --delete-emptydir-data --timeout300s2.2.下载新版本组件wget https://github.com/containerd/containerd/releases/download/v2.3.2/containerd-2.3.2-linux-amd64.tar.gz wget https://github.com/opencontainers/runc/releases/download/v1.4.3/runc.amd64 wget https://github.com/containernetworking/plugins/releases/download/v1.9.1/cni-plugins-linux-amd64-v1.9.1.tgz2.3.停止 containerdsystemctl stop containerd2.4.升级 runc# 查询旧版本 # runc --version runc version 1.2.5 commit: v1.2.5-0-g59923ef1 spec: 1.2.0 go: go1.23.7 # 备份旧版本 mv /usr/local/sbin/runc /usr/local/sbin/runc_1.2.5.old # 替换为新版本 mv ./runc.amd64 /usr/local/sbin/runc chmod 755 /usr/local/sbin/runc # 确认版本 runc --version runc version 1.4.3 commit: v1.4.3-0-gbb14dabeb spec: 1.3.0 go: go1.25.11 libseccomp: 2.6.02.5.升级 CNI plugins看下更新前/opt/cni/bin/下的内容# ll /opt/cni/bin/ total 215M -rwxr-xr-x 1 root root 4.0M Jul 15 2025 tuning -rwxr-xr-x 1 root root 4.3M Jul 15 2025 portmap -rwxr-xr-x 1 root root 3.9M Jul 15 2025 loopback -rwxr-xr-x 1 root root 3.8M Jul 15 2025 host-local -rwxr-xr-x 1 root root 2.7M Jul 15 2025 flannel -rwxr-xr-x 1 root root 66M Jul 15 2025 calico-ipam -rwxr-xr-x 1 root root 66M Jul 15 2025 calico -rwxr-xr-x 1 root root 4.3M Jul 15 2025 bandwidth -rwxr-xr-x 1 root root 4.3M Aug 30 2024 macvlan -rwxr-xr-x 1 root root 3.8M Aug 30 2024 sbr -rwxr-xr-x 1 root root 4.6M Aug 30 2024 bridge -rwxr-xr-x 1 root root 11M Aug 30 2024 dhcp -rwxr-xr-x 1 root root 4.3M Aug 30 2024 dummy -rwxr-xr-x 1 root root 4.8M Aug 30 2024 firewall -rwxr-xr-x 1 root root 4.2M Aug 30 2024 host-device -rwxr-xr-x 1 root root 4.3M Aug 30 2024 ipvlan -rw-r--r-- 1 root root 12K Aug 30 2024 LICENSE -rwxr-xr-x 1 root root 4.4M Aug 30 2024 ptp -rw-r--r-- 1 root root 2.3K Aug 30 2024 README.md -rwxr-xr-x 1 root root 3.1M Aug 30 2024 static -rwxr-xr-x 1 root root 4.3M Aug 30 2024 tap -rwxr-xr-x 1 root root 4.3M Aug 30 2024 vlan -rwxr-xr-x 1 root root 4.0M Aug 30 2024 vrf解压新版本到 CNI 目录会覆盖同名插件tar zxvf ./cni-plugins-linux-amd64-v1.9.1.tgz -C /opt/cni/bin/查看更新后内容# ll /opt/cni/bin/ total 228M -rwxr-xr-x 1 root root 4.2M Mar 16 22:18 host-local -rw-r--r-- 1 root root 12K Mar 16 22:18 LICENSE -rw-r--r-- 1 root root 2.3K Mar 16 22:18 README.md -rwxr-xr-x 1 root root 3.7M Mar 16 22:18 static -rwxr-xr-x 1 root root 14M Mar 16 22:18 dhcp -rwxr-xr-x 1 root root 5.3M Mar 16 22:18 tap -rwxr-xr-x 1 root root 5.1M Mar 16 22:18 vlan -rwxr-xr-x 1 root root 5.1M Mar 16 22:18 ipvlan -rwxr-xr-x 1 root root 4.2M Mar 16 22:18 loopback -rwxr-xr-x 1 root root 5.1M Mar 16 22:18 macvlan -rwxr-xr-x 1 root root 5.3M Mar 16 22:18 ptp -rwxr-xr-x 1 root root 5.5M Mar 16 22:18 bridge -rwxr-xr-x 1 root root 5.1M Mar 16 22:18 dummy -rwxr-xr-x 1 root root 5.0M Mar 16 22:18 host-device -rwxr-xr-x 1 root root 4.9M Mar 16 22:18 portmap -rwxr-xr-x 1 root root 4.4M Mar 16 22:18 sbr -rwxr-xr-x 1 root root 4.2M Mar 16 22:18 tuning -rwxr-xr-x 1 root root 4.5M Mar 16 22:18 vrf -rwxr-xr-x 1 root root 5.5M Mar 16 22:18 firewall -rwxr-xr-x 1 root root 4.9M Mar 16 22:18 bandwidth -rwxr-xr-x 1 root root 2.7M Jul 15 2025 flannel -rwxr-xr-x 1 root root 66M Jul 15 2025 calico-ipam -rwxr-xr-x 1 root root 66M Jul 15 2025 calico