K8s Pod 调度全解nodeName、nodeSelector、亲和性、污点容忍到 cordon/drain学习参考将 Pod 指派给节点默认情况下K8s 调度器会自动把 Pod 分散到资源充足的节点上。但生产环境经常有特殊需求把有大量磁盘 I/O 的 Pod 放到 SSD 节点、把 GPU 任务放到带 GPU 的节点、把同一服务的副本打散到不同节点做高可用。本文把 K8s 控制 Pod 运行位置的5 种手段全 部过一遍从最简单的 nodeName 到最灵活的亲和性/反亲和性再到污点容忍和节点维护操作。一、五种调度约束手段总览手段方向灵活度适用场景nodeNamePod → 指定节点最死板直接写死节点名临时调试生产不推荐nodeSelectorPod → 选带标签的节点简单等值匹配最常用的基础选择nodeAffinityPod → 选节点标签支持 In/NotIn/Exists/Gt/Lt、硬/软策略需要复杂规则的场景podAffinity/podAntiAffinityPod → 跟其他 Pod 同/不同拓扑域基于已运行 Pod 标签打散副本、就近通信Taint Toleration节点 → 选 Pod节点主动拒绝Pod 申请容忍专用节点、GPU 节点、主节点隔离二、nodeName最直接也最死板nodeName是 PodSpec 的一个字段值就是节点名称调度器直接把 Pod 塞到那个节点上。优先级最高但通常不推荐使用。限制指定节点不存在 → Pod 不会运行甚至被自动删除指定节点资源不够 → 调度失败报 OutOfmemory / OutOfcpu云环境节点名称不可预测写死了迁移困难apiVersion:apps/v1kind:Deploymentmetadata:labels:app:webappname:webappspec:replicas:2selector:matchLabels:app:webapptemplate:metadata:labels:app:webappspec:nodeName:worker32.laoma.cloud# 直接写死节点containers:-image:hub.laoma.cloud/library/nginxname:nginxrootmaster30:~# kubectl apply -f deploy-with-nodeName.yamlrootmaster30:~# kubectl get pods -o wide --no-headers | awk {print $1,$3,$7}webapp-55c6b896bc-ns9m5 Running worker32.laoma.cloud webapp-55c6b896bc-qr892 Running worker32.laoma.cloud三、nodeSelector最常用的基础选择给节点打标签Pod 选标签。这是节点选择约束的推荐形式。# 查看节点标签rootmaster30:~# kubectl get node --show-labels# 给 worker31 打 disktypessd 标签rootmaster30:~# kubectl label node worker31.laoma.cloud disktypessdrootmaster30:~# kubectl get node -L disktypeNAME STATUS ROLES AGE VERSION DISKTYPE master30.laoma.cloud Ready control-plane 11d v1.30.2 worker31.laoma.cloud Readynone11d v1.30.2 ssd worker32.laoma.cloud Readynone11d v1.30.2apiVersion:apps/v1kind:Deploymentmetadata:labels:app:webappname:webappspec:replicas:2selector:matchLabels:app:webapptemplate:metadata:labels:app:webappspec:nodeSelector:disktype:ssd# 只调度到带 disktypessd 的节点containers:-image:hub.laoma.cloud/library/nginxname:nginxrootmaster30:~# kubectl apply -f deploy-with-nodeSelector.yamlrootmaster30:~# kubectl get pods -o wide --no-headers | awk {print $1,$3,$7}webapp-7d675f9d9c-rz9hx Running worker31.laoma.cloud webapp-7d675f9d9c-zrj97 Running worker31.laoma.cloud注意删除节点标签不会触发 Pod 重新调度亲和性只在调度那一刻生效。只有修改 Deployment 的 nodeSelector 才会触发滚动重建。四、nodeAffinity比 nodeSelector 强大得多节点亲和性让你根据节点标签约束 Pod 调度位置有两种模式requiredDuringSchedulingIgnoredDuringExecution硬规则必须满足不满足就 PendingpreferredDuringSchedulingIgnoredDuringExecution软规则优先满足不满足也能调度apiVersion:apps/v1kind:Deploymentmetadata:labels:app:webname:webspec:replicas:2selector:matchLabels:app:webtemplate:metadata:labels:app:webspec:affinity:nodeAffinity:# 硬规则必须调度到 CPUL1 或 CPUL2 的节点requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:-matchExpressions:-key:CPUoperator:Invalues:-L1-L2# 软规则优先选 MEML1 或 MEML2 的节点preferredDuringSchedulingIgnoredDuringExecution:-weight:1preference:matchExpressions:-key:MEMoperator:Invalues:-L1-L2containers:-image:hub.laoma.cloud/library/nginxname:nginxoperator 支持In、NotIn、Exists、DoesNotExist、Gt、Lt权重规则weight范围 1-100调度器给每个满足硬规则的节点打分把所有匹配的软规则 weight 累加总分最高的节点优先如果同时指定了nodeSelector和nodeAffinity两者都必须满足实验验证# 节点未打 CPU 标签 → Pod Pendingrootmaster30:~# kubectl apply -f deploy-with-nodeAffinity.yamlrootmaster30:~# kubectl get podsNAME READY STATUS RESTARTS AGE web-854cd4986-598lb0/1 Pending02s web-854cd4986-mp2660/1 Pending02s# 给两个 worker 都打 CPU 标签 → 均衡调度rootmaster30:~# kubectl label nodes worker31.laoma.cloud CPUL1rootmaster30:~# kubectl label nodes worker32.laoma.cloud CPUL2rootmaster30:~# kubectl rollout restart deployment webrootmaster30:~# kubectl get pods -o wide --no-headers | awk {print $1,$3,$7}web-85c8ffff8-q2j7r Running worker31.laoma.cloud web-85c8ffff8-rkq7s Running worker32.laoma.cloud# 给 worker32 加 MEML2 软偏好标签 → 都调度到 worker32rootmaster30:~# kubectl label nodes worker32.laoma.cloud MEML2rootmaster30:~# kubectl rollout restart deployment webrootmaster30:~# kubectl get pods -o wide --no-headers | awk {print $1,$3,$7}web-6c45779498-glxbg Running worker32.laoma.cloud web-6c45779498-sn996 Running worker32.laoma.cloud注意如果你修改或删除了 Pod 所调度到的节点标签Pod 不会被删除。亲和性选择只在 Pod 调度期间有效。五、Pod 间亲和性与反亲和性前面都是Pod 选节点这一节是Pod 选其他 Pod 的邻居——基于已经在节点上运行的Pod 标签来约束调度位置。亲和性podAffinity希望 Pod A 和 Pod B 在同一个拓扑域反亲和性podAntiAffinity不希望 Pod A 和 Pod B 在同一个拓扑域topologyKey表示拓扑域可以是节点、机架、可用区等用节点标签的 key 表示。⚠️ Pod 间亲和性会消耗大量计算资源不建议在超过数百个节点的集群中使用。5.1 示例亲和性调度同拓扑域把两个 worker 节点都打上topology.kubernetes.io/zonev表示它们属于同一个可用区。然后 web2 的 Pod 亲和性要求跟 web1 在同一个 zone---apiVersion:apps/v1kind:Deploymentmetadata:name:web1spec:selector:matchLabels:app:web1replicas:1template:metadata:labels:app:web1spec:nodeSelector:topology.kubernetes.io/zone:vcontainers:-name:web-serverimage:hub.laoma.cloud/library/nginx---apiVersion:apps/v1kind:Deploymentmetadata:name:web2spec:selector:matchLabels:app:web2replicas:1template:metadata:labels:app:web2spec:affinity:podAffinity:requiredDuringSchedulingIgnoredDuringExecution:-labelSelector:matchExpressions:-key:appoperator:Invalues:-web1topologyKey:topology.kubernetes.io/zonecontainers:-name:web-serverimage:hub.laoma.cloud/library/nginxrootmaster30:~# kubectl apply -f deploy-with-podaffinity.yamlrootmaster30:~# kubectl get pods -o wide --no-headers | awk {print $1,$3,$7}web1-849b694c75-t2spf Running worker31.laoma.cloud web2-84fdf8f999-vn5l9 Running worker32.laoma.cloud# 两个节点都属于 zonev所以 web2 可以调度到 worker32# 删除 worker32 的 zone 标签 → web2 只能调度到 worker31跟 web1 同 zonerootmaster30:~# kubectl label nodes worker32.laoma.cloud topology.kubernetes.io/zone-rootmaster30:~# kubectl rollout restart deployment web2rootmaster30:~# kubectl get pods -o wide --no-headers | awk {print $1,$3,$7}web1-849b694c75-t2spf Running worker31.laoma.cloud web2-7947994f89-pwnkv Running worker31.laoma.cloud结论Pod 亲和性必须满足两个条件节点必须有相应的 topologyKey 标签在有相应标签的节点上至少运行一个匹配 labelSelector 的 Pod5.2 示例反亲和性调度打散副本把同一服务的多个副本分散到不同节点上提高高可用性。topologyKey: kubernetes.io/hostname表示以单个节点为拓扑域——每个节点最多跑一个副本apiVersion:apps/v1kind:Deploymentmetadata:name:storespec:selector:matchLabels:app:storereplicas:3template:metadata:labels:app:storespec:affinity:podAntiAffinity:requiredDuringSchedulingIgnoredDuringExecution:-labelSelector:matchExpressions:-key:appoperator:Invalues:-storetopologyKey:kubernetes.io/hostnamecontainers:-name:web-serverimage:hub.laoma.cloud/library/nginxrootmaster30:~# kubectl get pods -o wide --no-headers | awk {print $1,$3,$7}store-5596cf4c84-9z5ch Pendingnonestore-5596cf4c84-dswkq Running worker32.laoma.cloud store-5596cf4c84-ww2l4 Running worker31.laoma.cloud# 3 个副本只有 2 个 worker 节点第 3 个 Pending5.3 多亲和反亲和组合spec:affinity:podAntiAffinity:# 不要跟自己的副本同节点requiredDuringSchedulingIgnoredDuringExecution:-labelSelector:matchExpressions:-key:appoperator:Invalues:-web-storetopologyKey:kubernetes.io/hostnamepodAffinity:# 必须跟 appstore 的 Pod 同节点requiredDuringSchedulingIgnoredDuringExecution:-labelSelector:matchExpressions:-key:appoperator:Invalues:-storetopologyKey:kubernetes.io/hostname六、Taint 和 Toleration节点选 Pod前面都是Pod 挑节点污点是反过来——节点主动拒绝 Pod。Taint污点Node 上打标记不匹配的 Pod 不能上来Toleration容忍度Pod 上声明能容忍哪些污点nodeAffinity vs Taint 区别nodeAffinity 是 Pod 选 Node我想去哪Taint 是 Node 选 Pod我不要谁。就像相亲——女选男是亲和性男选女是污点。6.1 设置节点污点# 给 worker31 加污点CPUL1效果 NoSchedulerootmaster30:~# kubectl taint nodes worker31.laoma.cloud CPUL1:NoSchedule# 移除污点rootmaster30:~# kubectl taint nodes worker31.laoma.cloud CPU:NoSchedule-6.2 effect 三种效果effect作用对已运行 PodNoSchedule新 Pod 不调度上来不影响已运行的PreferNoSchedule尽量不调度上来不强制不影响已运行的NoExecute新 Pod 不调度已运行的也会被驱逐容忍的可留不容忍的马上驱逐6.3 Pod 容忍度tolerations:-key:CPUoperator:Equal# Equal默认 键值都匹配Exists 只要 key 存在就匹配value:L1effect:NoSchedule# 给两个 worker 打不同污点rootmaster30:~# kubectl taint nodes worker31.laoma.cloud CPUL1:NoSchedulerootmaster30:~# kubectl taint nodes worker32.laoma.cloud CPUL2:NoSchedule# Pod 容忍 CPUL1 → 只能调度到 worker31spec:tolerations:-key:CPUoperator:Equalvalue:L1effect:NoSchedulerootmaster30:~# kubectl apply -f deploy-with-tolerations.yamlrootmaster30:~# kubectl get pod -o wideNAME READY STATUS RESTARTS AGE IP NODE web-7c99d767-w9rzt1/1 Running05s10.224.162.142 worker31.laoma.cloud web-7c99d767-wwjhk1/1 Running05s10.224.19.13 worker31.laoma.cloud6.4 多污点多容忍度匹配节点可以有多个污点Pod 可以有多个容忍度。匹配逻辑像过滤器从节点所有污点开始过滤掉 Pod 能容忍的剩下不能容忍的 effect 决定结果有未过滤的NoSchedule→ 不调度只有未过滤的PreferNoSchedule→ 尽量不调度有未过滤的NoExecute→ 不调度或驱逐6.5 内置污点节点控制器自动给异常节点加污点污点含义node.kubernetes.io/not-ready节点未就绪node.kubernetes.io/unreachable节点不可达node.kubernetes.io/memory-pressure内存压力node.kubernetes.io/disk-pressure磁盘压力node.kubernetes.io/network-unavailable网络不可用node.kubernetes.io/unschedulable不可调度K8s 默认给所有 Pod 自动加了not-ready和unreachable的容忍度tolerationSeconds3005 分钟。也就是说节点挂了Pod 还会在原地待 5 分钟才被驱逐。6.6 典型用例专用节点dedicatedgroupName:NoSchedule只有带对应容忍度的 Pod 才能上来GPU 节点specialtrue:NoSchedule不需要 GPU 的 Pod 不要来占用资源七、cordon / uncordon / drain节点维护7.1 cordon标记不可调度rootmaster30:~# kubectl cordon worker31.laoma.cloudrootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE master30.laoma.cloud Ready control-plane 9d worker31.laoma.cloud Ready,SchedulingDisablednone9d worker32.laoma.cloud Readynone9d# 新 Pod 不会再调度到 worker31但已有 Pod 继续运行rootmaster30:~# kubectl create deployment web --imagenginx --replicas2rootmaster30:~# kubectl get pod -o wideNAME READY STATUS NODE web-59b9bb7664-flhvx1/1 Running worker32.laoma.cloud web-59b9bb7664-ss5861/1 Running worker32.laoma.cloud# 恢复可调度rootmaster30:~# kubectl uncordon worker31.laoma.cloud7.2 drain安全驱逐节点上所有 Pod节点维护内核升级、硬件更换之前先用 drain 把 Pod 安全迁走# 默认不驱逐 DaemonSet 管理的 Podcalico-node、kube-proxy 等rootmaster30:~# kubectl drain worker31.laoma.clouderror: cannot delete DaemonSet-managed Pods: kube-system/calico-node-jwdzx, kube-system/kube-proxy-98qkl# 加 --ignore-daemonsets 忽略 DaemonSetrootmaster30:~# kubectl drain worker31.laoma.cloud --ignore-daemonsetsnode/worker31.laoma.cloud already cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-jwdzx, kube-system/kube-proxy-98qkl evicting pod laoma/web-59b9bb7664-jkb5q evicting pod laoma/web-59b9bb7664-w6mbs pod/web-59b9bb7664-w6mbs evicted node/worker31.laoma.cloud evicted# 维护完后恢复rootmaster30:~# kubectl uncordon worker31.laoma.clouddrain cordon 驱逐 Pod一步到位。并行 drain 多个节点时会自动遵守 PodDisruptionBudget不会一次性把所有副本都赶走。八、总结手段一句话记忆nodeName写死节点名最简单最死板nodeSelector节点打标签Pod 选标签最常用nodeAffinity标签匹配 硬/软规则 权重比 nodeSelector 灵活podAffinity我要跟某个 Pod 待在同一个地方podAntiAffinity我不要跟自己的副本待在同一个节点Taint节点说我不要谁Pod 说我能忍cordon停止接新活老活继续干drain停止接新活 老活全部迁走环境清理kubectl delete ns scheduler
