该笔记将记录：Kubernetes Cluster 升级的方法，以及升级相关问题的解决办法。
注意事项：
1）这里仅简单记录升级过程，建议阅读相关官方文档，以获取升级过程的详细细节；
2）生产环境的升级还需要考虑业务可用性，建议详细阅读官方文档，以完成升级；
v1.18 to v1.20
Upgrading kubeadm clusters | Kubernetes
该部分将记录：将 Kubernetes Cluster 1.18 升级到 1.20 版本的过程，以及相关问题的处理办法。
鉴于是跨次版本号升级，根据官方文档，升级必须依次进行：1.18 ⇒ 1.19 ⇒ 1.12
1）Upgrading kubeadm clusters | Kubernetes/v1.19
2）Upgrading kubeadm clusters | Kubernetes/v1.20
1）环境检查、重要数据备份；
2）执行升级命令进行升级：
// ============================================================================> Determine which version to upgrade to apt update apt-cache madison kubeadm | grep 1.20.15-00 // ============================================================================> For the first control plane node apt-get install -y kubeadm=1.20.15-00 kubectl=1.20.15-00 --allow-change-held-packages kubeadm upgrade plan kubeadm upgrade apply 1.20.15 apt-get install -y kubelet=1.20.15-00 --allow-change-held-packages systemctl reload kubelet.service systemctl restart kubelet.service // ============================================================================> For the other control plane nodes kubeadm upgrade node apt-get install -y kubeadm=1.20.15-00 kubelet=1.20.15-00 kubectl=1.20.15-00 --allow-change-held-packages systemctl daemon-reload systemctl restart kubelet // ============================================================================> Upgrade worker nodes. apt-get install -y kubeadm=1.20.15-00 kubelet=1.20.15-00 kubectl=1.20.15-00 --allow-change-held-packages kubeadm upgrade node systemctl daemon-reload && systemctl restart kubelet
… CoreDNS cannot migrate the following plugins …
Can’t upgrade to 1.16.0: [ERROR CoreDNSUnsupportedPlugins]: there are unsupported plugins in the CoreDNS · Issue #82889 · kubernetes/kubernetes
whoami
问题描述：
# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[preflight] The corefile contains plugins that kubeadm/CoreDNS does not know how to migrate. Each plugin listed should be manually verified for compatibility with the newer version of CoreDNS. Once ready, the upgrade can be initiated by skipping the preflight check. During the upgrade, kubeadm will migrate the configuration while leaving the listed plugin configs untouched, but cannot guarantee that they will work with the newer version of CoreDNS.
[preflight] Some fatal errors occurred:
[ERROR CoreDNSUnsupportedPlugins]: CoreDNS cannot migrate the following plugins:
[Plugin "whoami" is unsupported by this migration tool in 1.6.7.]
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
原因分析：
通常不会遇到该问题，我们因为调试原因才使用 whoami 插件，因此可以删除。
解决方案：
编辑 ConfigMap 文件，移除 whoami 插件。
v1.20 to v1.21
Upgrading kubeadm clusters | Kubernetes
第一步、升级控制节点
# apt update # apt-cache madison kubeadm | grep 1.21 kubeadm | 1.22.15-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages ... // ---------------------------------------------------------------------------- // 第一个控制节点 apt-get update && \ apt-get install -y --allow-change-held-packages kubeadm=1.22.15-00 kubeadm version kubeadm upgrade plan kubeadm upgrade apply v1.22.15 # kubectl drain <node-to-drain> --ignore-daemonsets apt-get update && \ apt-get install -y --allow-change-held-packages kubelet=1.22.15-00 kubectl=1.22.15-00 systemctl daemon-reload systemctl restart kubelet # kubectl uncordon <node-to-drain> // ---------------------------------------------------------------------------- // 其他的控制节点 apt-get update && \ apt-get install -y --allow-change-held-packages kubeadm=1.22.15-00 kubeadm version kubeadm upgrade node # kubectl drain <node-to-drain> --ignore-daemonsets apt-get update && \ apt-get install -y --allow-change-held-packages kubelet=1.22.15-00 kubectl=1.22.15-00 systemctl daemon-reload && systemctl restart kubelet # kubectl uncordon <node-to-drain>
第二步、升级工作节点
apt-get update && \ apt-get install -y --allow-change-held-packages kubeadm=1.22.15-00 kubeadm upgrade node # kubectl drain <node-to-drain> --ignore-daemonsets apt-get update && \ apt-get install -y --allow-change-held-packages kubelet=1.22.15-00 kubectl=1.22.15-00 systemctl daemon-reload && systemctl restart kubelet # kubectl uncordon <node-to-drain>
第三步、验证集群状态
kubectl get nodes
1.22.13 to 1.23.17
Master
// -------------------------------------------------------- // Andy Master apt-mark unhold kubeadm && \ apt-get update && apt-get install -y kubeadm='1.23.17-00' && \ apt-mark hold kubeadm kubeadm upgrade plan kubeadm upgrade apply v1.23.17 // -------------------------------------------------------- // Other Master apt-mark unhold kubeadm && \ apt-get update && apt-get install -y kubeadm='1.23.17-00' && \ apt-mark hold kubeadm kubeadm upgrade node // -------------------------------------------------------- // All Master apt-mark unhold kubelet kubectl && \ apt-get update && apt-get install -y kubelet='1.23.17-00' kubectl='1.23.17-00' && \ apt-mark hold kubelet kubectl systemctl daemon-reload systemctl restart kubelet
Worker
apt-mark unhold kubeadm && \ apt-get update && apt-get install -y kubeadm=1.23.17-00 && \ apt-mark hold kubeadm kubeadm upgrade node apt-mark unhold kubelet kubectl && \ apt-get update && apt-get install -y kubelet=1.23.17-00 kubectl=1.23.17-00 && \ apt-mark hold kubelet kubectl systemctl daemon-reload systemctl restart kubelet
1.23.17 to 1.24.17
Master
// -------------------------------------------------------- // Andy Master apt-mark unhold kubeadm && \ apt-get update && apt-get install -y kubeadm='1.24.17-00' && \ apt-mark hold kubeadm kubeadm upgrade plan kubeadm upgrade apply v1.24.17 // -------------------------------------------------------- // Other Master apt-mark unhold kubeadm && \ apt-get update && apt-get install -y kubeadm='1.24.17-00' && \ apt-mark hold kubeadm kubeadm upgrade node // -------------------------------------------------------- // All Master apt-mark unhold kubelet kubectl && \ apt-get update && apt-get install -y kubelet='1.24.17-00' kubectl='1.24.17-00' && \ apt-mark hold kubelet kubectl systemctl daemon-reload systemctl restart kubelet
Feb 21 09:05:13 k8s-infra-cp122 kubelet[15305]: --volume-stats-agg-period duration Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes. To disable volume calculations, set to a negative number. (default 1m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) Feb 21 09:05:13 k8s-infra-cp122 kubelet[15305]: Error: failed to parse kubelet flag: unknown flag: --network-plugin Feb 21 09:05:23 k8s-infra-cp122 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 17. Feb 21 09:05:23 k8s-infra-cp122 systemd[1]: Stopped kubelet: The Kubernetes Node Agent. Feb 21 09:05:23 k8s-infra-cp122 systemd[1]: Started kubelet: The Kubernetes Node Agent. // # vim /var/lib/kubelet/kubeadm-flags.env // 删除 --network-plugin 选项
Worker
apt-mark unhold kubeadm && \ apt-get update && apt-get install -y kubeadm=1.23.17-00 && \ apt-mark hold kubeadm kubeadm upgrade node apt-mark unhold kubelet kubectl && \ apt-get update && apt-get install -y kubelet=1.23.17-00 kubectl=1.23.17-00 && \ apt-mark hold kubelet kubectl systemctl daemon-reload systemctl restart kubelet
1.24.13 to 1.25.14
# 02/21/2024 参考 Master
// -------------------------------------------------------- // Andy Master apt-mark unhold kubeadm && \ apt-get update && apt-get install -y kubeadm='1.25.14-00' && \ apt-mark hold kubeadm kubeadm upgrade plan kubeadm upgrade apply v1.25.14 // -------------------------------------------------------- // Other Master apt-mark unhold kubeadm && \ apt-get update && apt-get install -y kubeadm='1.25.14-00' && \ apt-mark hold kubeadm kubeadm upgrade node // -------------------------------------------------------- // All Master apt-mark unhold kubelet kubectl && \ apt-get update && apt-get install -y kubelet='1.25.14-00' kubectl='1.25.14-00' && \ apt-mark hold kubelet kubectl systemctl daemon-reload systemctl restart kubelet
相关问题：
# kubeadm upgrade apply v1.25.14
...
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: kube-proxy
[upgrade/postupgrade] FATAL post-upgrade error: unable to create/update the DNS service: Service "kube-dns" is invalid: spec.clusterIPs[0]: Invalid value: []string{"10.130.0.10"}: may not change once set
To see the stack trace of this error execute with --v=5 or higher
https://github.com/kubernetes/kubeadm/issues/2358
修改 kube-dns 服务，使用 10.130.0.10 地址
Worker
apt-mark unhold kubeadm && \ apt-get update && apt-get install -y kubeadm=1.25.14-00 && \ apt-mark hold kubeadm kubeadm upgrade node apt-mark unhold kubelet kubectl && \ apt-get update && apt-get install -y kubelet=1.25.14-00 kubectl=1.25.14-00 && \ apt-mark hold kubelet kubectl sudo systemctl daemon-reload sudo systemctl restart kubelet
1.24.13 to 1.27.6
#1 集群组件升级
若集群现有插件（例如 Ingress Controller，Storage Provisioner 等等）不支持 1.27 版本集群，那当升级集群后，插件将无法运行。所以，我们将先进行集群插件的升级，使插件能够同时支持 1.24 与 1.27 版本的集群。
CRI
现在 containerd 1.6.12 版本，根据 containerd/Kubernetes Support 文档，我们计划升级到 containerd 1.6.26 版本（其为 Docker APT 仓库里最新的版本）。
CNI
现在 Calico 3.24.5 版本，并且通过对比 Kubernetes Mannifest 文件，其为「Install Calico with Kubernetes API datastore, more than 50 nodes」方式部署。根据 Calico/System requirements/Kubernetes requirements 文档，我们将升级到 Calico 3.27 版本，我们将采用 Operator Helm 部署。
CSI
Others