1. 项目概述与核心目标在云原生技术快速发展的当下Kubernetesk8s已成为容器编排的事实标准。本次部署方案基于Ubuntu 22.04 LTS操作系统搭建k8s 1.28版本的高可用集群采用多Master节点架构配合Keepalived和Nginx实现负载均衡。这种架构设计能够确保控制平面在单个节点故障时仍保持可用性满足生产环境对稳定性的严苛要求。关键组件版本说明操作系统Ubuntu 22.04.3 LTSJammy JellyfishKubernetesv1.28.2当前稳定版Container Runtimecontainerd 1.7.3Keepalivedv2.2.7Nginx1.25.2高可用集群的核心价值在于消除单点故障。传统单Master架构一旦控制节点宕机整个集群的管理功能将瘫痪。而通过三节点Master配置即使一个节点失效剩余节点仍能维持集群的正常运作。Keepalived负责VIP虚拟IP的漂移管理Nginx则作为反向代理实现API Server的负载均衡两者配合形成完整的容错体系。2. 环境准备与系统配置2.1 硬件资源规划建议生产环境采用以下配置最低要求Master节点3台4核CPU/8GB内存/50GB磁盘Worker节点按需扩展建议2核CPU/4GB内存/40GB磁盘起网络要求所有节点间网络延迟1ms专用千兆网络接口用于集群通信唯一主机名和静态IP分配实际操作中我在物理服务器上配置了# 各节点hosts文件示例/etc/hosts 192.168.1.101 k8s-master1 192.168.1.102 k8s-master2 192.168.1.103 k8s-master3 192.168.1.201 k8s-worker12.2 系统基础配置在所有节点执行以下初始化操作禁用swapk8s强制要求sudo swapoff -a sudo sed -i / swap / s/^\(.*\)$/#\1/g /etc/fstab加载内核模块并修改sysctl参数sudo modprobe overlay sudo modprobe br_netfilter cat EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter EOF cat EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 EOF sudo sysctl --system配置时间同步关键sudo timedatectl set-timezone Asia/Shanghai sudo apt install chrony -y sudo systemctl enable --now chrony常见问题如果节点间时间偏差超过100ms会导致证书验证失败。建议部署前用chronyc sources命令检查同步状态。3. 容器运行时与k8s组件安装3.1 containerd安装与配置Ubuntu 22.04默认使用containerd作为容器运行时sudo apt update sudo apt install -y containerd sudo mkdir -p /etc/containerd containerd config default | sudo tee /etc/containerd/config.toml修改配置文件启用systemd cgroup驱动[plugins.io.containerd.grpc.v1.cri.containerd.runtimes.runc] ... [plugins.io.containerd.grpc.v1.cri.containerd.runtimes.runc.options] SystemdCgroup true重启服务sudo systemctl restart containerd sudo systemctl enable containerd3.2 kubeadm/kubelet/kubectl安装添加官方GPG密钥和仓库sudo apt install -y apt-transport-https ca-certificates curl curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg echo deb [signed-by/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ / | sudo tee /etc/apt/sources.list.d/kubernetes.list安装指定版本组件sudo apt update sudo apt install -y kubelet1.28.2-1.1 kubeadm1.28.2-1.1 kubectl1.28.2-1.1 sudo apt-mark hold kubelet kubeadm kubectl4. 高可用控制平面部署4.1 首个Master节点初始化使用kubeadm生成默认配置模板kubeadm config print init-defaults kubeadm-config.yaml修改关键参数apiVersion: kubeadm.k8s.io/v1beta3 kind: InitConfiguration nodeRegistration: criSocket: unix:///var/run/containerd/containerd.sock imagePullPolicy: IfNotPresent --- apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: 1.28.2 controlPlaneEndpoint: k8s-vip:6443 # Keepalived虚拟IP apiServer: certSANs: - k8s-vip - 192.168.1.100 # VIP地址 networking: podSubnet: 10.244.0.0/16 # Flannel网络方案 serviceSubnet: 10.96.0.0/12执行初始化sudo kubeadm init --configkubeadm-config.yaml --upload-certs成功后会输出join命令保存供其他节点使用mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config4.2 其他Master节点加入使用首个节点输出的control-plane join命令sudo kubeadm join k8s-vip:6443 --token xxxx \ --discovery-token-ca-cert-hash sha256:xxxx \ --control-plane --certificate-key xxxx验证集群状态kubectl get nodes -o wide kubectl get pods -n kube-system排错技巧如果节点长时间NotReady检查kubelet日志journalctl -u kubelet -f常见问题包括网络插件未安装或证书过期。5. Keepalived与Nginx负载均衡5.1 Keepalived配置所有Master节点安装Keepalivedsudo apt install -y keepalived主节点配置/etc/keepalived/keepalived.confvrrp_script chk_nginx { script killall -0 nginx interval 2 weight -20 } vrrp_instance VI_1 { state MASTER interface ens160 # 修改为实际网卡名 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 42 } virtual_ipaddress { 192.168.1.100/24 } track_script { chk_nginx } }备节点配置差异点state BACKUP priority 90 # 第二个备节点设为80启动服务sudo systemctl enable --now keepalived5.2 Nginx四层代理配置所有Master节点安装Nginxsudo apt install -y nginx配置TCP负载均衡/etc/nginx/nginx.confstream { upstream k8s-apiserver { server 192.168.1.101:6443 weight5 max_fails3 fail_timeout30s; server 192.168.1.102:6443 weight5 max_fails3 fail_timeout30s; server 192.168.1.103:6443 weight5 max_fails3 fail_timeout30s; } server { listen 6443; proxy_pass k8s-apiserver; proxy_connect_timeout 1s; proxy_timeout 3s; } }启动Nginxsudo systemctl enable --now nginx验证VIP漂移# 在主节点停止keepalived观察VIP是否迁移 sudo systemctl stop keepalived ip addr show ens160 # 检查备节点是否获得VIP6. 网络插件与验证测试6.1 安装Flannel网络插件kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml检查Pod网络状态kubectl get pods -n kube-system -l appflannel kubectl get nodes -o jsonpath{range .items[*]}{.status.addresses[?(.typeInternalIP)].address} {.spec.podCIDR}{\n}{end}6.2 高可用性验证测试模拟Master节点故障# 随机关闭一个Master节点的kube-apiserver sudo systemctl stop kube-apiserver # 观察集群操作是否正常应无感知 kubectl get nodes负载均衡测试# 连续执行多次观察请求分发 for i in {1..20}; do kubectl get nodes; done网络连通性测试kubectl create deployment nginx --imagenginx kubectl expose deployment nginx --port80 kubectl run testpod --rm -it --imagebusybox -- sh # 在testpod内执行 wget -qO- nginx7. 运维优化与监控7.1 关键参数调优调整kubelet资源预留/var/lib/kubelet/config.yamlapiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration evictionHard: memory.available: 500Mi nodefs.available: 10% kubeReserved: cpu: 500m memory: 500Mi systemReserved: cpu: 500m memory: 500Mi7.2 基础监控部署安装Metrics Serverkubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml验证资源监控kubectl top nodes kubectl top pods -A8. 常见问题解决方案8.1 证书相关错误现象x509: certificate is valid for 10.96.0.1, not 192.168.1.100解决重新生成证书包含VIP地址sudo kubeadm init phase certs all --configkubeadm-config.yaml sudo systemctl restart kubelet8.2 节点NotReady状态排查步骤检查网络插件Pod是否运行正常验证节点间网络连通性查看kubelet日志中的错误信息确认防火墙规则未阻断流量8.3 Keepalived VIP不漂移检查要点节点间组播是否畅通tcpdump抓包防火墙是否放行VRRP协议IP协议号112优先级配置是否正确网卡名称是否与实际一致经过完整部署和验证后这套高可用k8s集群已经可以承载生产负载。在实际运维中建议定期检查证书有效期、备份etcd数据并考虑增加集群监控告警系统。对于需要更高性能的场景可以尝试替换Nginx为HAProxy或者使用BGP模式下的MetalLB实现负载均衡。
