二进制文件安装高可用k8s集群（一）etcd集群安装 安装高可用kubernetes集群所需要的etcd数据库，使用二进制方式进行安装。

k8s集群（containerd）安装——node篇 环境配置禁用selinuxsed -i '/^SELINUX=/s//SELINUX=disabled/' /etc/selinux/config禁用swap swapoff -a && sed -i '/swap/d' /etc/fstab禁用防火墙 yum -y remove firewalld修改hosts文件 echo "192.168.2.100 node1" >> /etc/hosts设置网桥（端口转发）for i in overlay br_netfilter do modprobe ${i} echo "${i}" >>/etc/modules-load.d/containerd.conf done cat >/etc/sysctl.d/99-kubernetes-cri.conf<<EOF net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 EOF sysctl --system安装软件包yum -y install kubeadm kubelet ipvsadm ipset nfs-utils containerd修改containerd配置文件参考文章 containerd安装与配置设置开机自启动systemctl enable --now containerd systemctl enable --now kubelet获取master的token#在master主机主机上曹组操作 kubeadm token list TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS abcdef.0123456789abcdef 22h 2025-03-28T02:16:52Z authentication,signing <none> system:bootstrappers:kubeadm:default-node-token #发现默认的token只剩下22个小时的有效期，删除这个token，重新生成一个长期有效的token kubeadm token delete abcdef.0123456789abcdef #创建一个新token kubeadm token create --ttl=0 --print-join-command kubeadm join 192.168.99.100:6443 --token truj3i.83g7lpw6cxj8medc --discovery-token-ca-cert-hash sha256:c5ce5be2f926c19b2760468618ef4746fa5cc63ca863ed01daba1021b06d7a32 #创建token命令返回的信息就是node加入master的命令#在node上面操作 kubeadm join 192.168.99.100:6443 --token truj3i.83g7lpw6cxj8medc --discovery-token-ca-cert-hash sha256:c5ce5be2f926c19b2760468618ef4746fa5cc63ca863ed01daba1021b06d7a32 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.在master上查看集群信息node节点加入集群后需要等待一段时间（一般是几分钟）才会显示ready状态。 kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready control-plane 132m v1.28.15 node1 Ready <none> 13m v1.28.15 kubectl -n kube-system get pods NAME READY STATUS RESTARTS AGE calico-kube-controllers-658d97c59c-9z9km 1/1 Running 0 119m calico-node-7ftlf 1/1 Running 0 119m calico-node-fg6zv 1/1 Running 0 11m coredns-66f779496c-n7h9t 1/1 Running 0 131m coredns-66f779496c-x7tvt 1/1 Running 0 131m etcd-master 1/1 Running 0 131m kube-apiserver-master 1/1 Running 0 131m kube-controller-manager-master 1/1 Running 0 131m kube-proxy-tdp72 1/1 Running 0 131m kube-proxy-vrbvv 1/1 Running 0 11m kube-scheduler-master 1/1 Running 0 131m #刚加入集群时的状态 kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready control-plane 124m v1.28.15 node1 NotReady <none> 5m13s v1.28.15 kubectl -n kube-system get pods NAME READY STATUS RESTARTS AGE calico-kube-controllers-658d97c59c-9z9km 1/1 Running 0 114m calico-node-7ftlf 1/1 Running 0 114m calico-node-fg6zv 0/1 Init:1/3 0 5m57s coredns-66f779496c-n7h9t 1/1 Running 0 125m coredns-66f779496c-x7tvt 1/1 Running 0 125m etcd-master 1/1 Running 0 125m kube-apiserver-master 1/1 Running 0 125m kube-controller-manager-master 1/1 Running 0 125m kube-proxy-tdp72 1/1 Running 0 125m kube-proxy-vrbvv 1/1 Running 0 5m57s kube-scheduler-master 1/1 Running 0 125m

k8s集群（containerd）安装——master篇 master安装环境配置禁用selinux sed -i '/^SELINUX=/s//SELINUX=disabled/' /etc/selinux/config禁用swap swapoff -a && sed -i '/swap/d' /etc/fstab禁用防火墙 yum -y remove firewalld修改hosts文件 echo "192.168.2.51 master" >> /etc/hosts不添加会有如下报错[root@master1 k8s]# kubeadm init --config=kubeadm.yml --dry-run [init] Using Kubernetes version: v1.28.0 [preflight] Running pre-flight checks [WARNING Hostname]: hostname "master" could not be reached [WARNING Hostname]: hostname "master": lookup master on 114.114.114.114:53: no such host [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' 设置网桥for i in overlay br_netfilter;do modprobe ${i} echo "${i}" >>/etc/modules-load.d/containerd.conf done cat >/etc/sysctl.d/99-kubernetes-cri.conf<<EOF net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 EOF sysctl --system安装软件包安装kubeadm、kubectl、kubelet、containerdyum install -y kubeadm kubelet kubectl containerd #安装代理软件包 yum install -y ipvsadm ipset设置开机自启动systemctl enable --now containerd systemctl enable --now kubelet#配置快捷键 source <(kubeadm completion bash|tee /etc/bash_completion.d/kubeadm) source <(kubectl completion bash|tee /etc/bash_completion.d/kubectl)修改containerd配置文件，安装crictl，详见文章 containerd安装与配置下载镜像#查看安装集群所需要的镜像信息 kubeadm config images list I0326 20:53:28.890691 6257 version.go:256] remote version is much newer: v1.32.3; falling back to: stable-1.28 registry.k8s.io/kube-apiserver:v1.28.15 registry.k8s.io/kube-controller-manager:v1.28.15 registry.k8s.io/kube-scheduler:v1.28.15 registry.k8s.io/kube-proxy:v1.28.15 registry.k8s.io/pause:3.9 registry.k8s.io/etcd:3.5.15-0 registry.k8s.io/coredns/coredns:v1.10.1 #将镜像信息保存到临时文件中，方便下一步操作 kubeadm config images list > a.txt # 下载所需要的镜像 for i in ` awk -F'/' '{print "registry.aliyuncs.com/google_containers/"$2}' a.txt` do crictl pull $i doneImage is up to date for sha256:9dc6939e7c573673801790fcfad6f994282c216e005578f5836b5fafc6685fc2 Image is up to date for sha256:10541d8af03f40fae257735edd69b6c5dd0084bb9796649409ac7b5660705148 Image is up to date for sha256:9d3465f8477c6b383762d90ec387c9d77da8a402a849265805f86feaa57aeeea Image is up to date for sha256:ba6d7f8bc25be40b51dfeb5ddfda697527ba55073620c1c5fa04a5f0ae9e3816 Image is up to date for sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c Image is up to date for sha256:2e96e5913fc06e3d26915af3d0f2ca5048cc4b6327e661e80da792cbf8d8d9d4 Image is up to date for sha256:1cf5f116067c67da67f97bff78c4bbc76913f59057c18627b96facaced73ea0b [root@master1 ~]# crictl images ls IMAGE TAG IMAGE ID SIZE registry.aliyuncs.com/google_containers/coredns latest 1cf5f116067c6 20.9MB registry.aliyuncs.com/google_containers/etcd 3.5.15-0 2e96e5913fc06 56.9MB registry.aliyuncs.com/google_containers/kube-apiserver v1.28.15 9dc6939e7c573 34.4MB registry.aliyuncs.com/google_containers/kube-controller-manager v1.28.15 10541d8af03f4 33.3MB registry.aliyuncs.com/google_containers/kube-proxy v1.28.15 ba6d7f8bc25be 28.3MB registry.aliyuncs.com/google_containers/kube-scheduler v1.28.15 9d3465f8477c6 18.5MB registry.aliyuncs.com/google_containers/pause 3.9 e6f1816883972 322kB 初始化master集群#生成配置模版 kubeadm config print init-defaults > init.yaml修改配置文件nodeRegistration下name字段改为master（name: master）imageRepository字段改为阿里云的地址（imageRepository: registry.aliyuncs.com/google_containers）localAPIEndpoint字段下advertiseAddress修改为master的IP地址（advertiseAddress: 192.168.2.51）设置cgroupDriver为systemd---kind: KubeletConfigurationapiVersion: kubelet.config.k8s.io/v1beta1cgroupDriver: systemd使用--dry-run模拟安装过程，查看是否有报错kubeadm init --config=init.yaml --dry-run #输出信息若干，没有 Error 和 Warning 就是正常正式部署masterrm -rf /etc/kubernetes/tmp kubeadm init --config=init.yaml |tee init/init.log #根据安装提示执行命令 mkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config安装calico网络插件kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml查看集群状态#使用下面两条命令中的任意一条即可 kubectl -n kube-system get pods kubectl get pods -A NAME READY STATUS RESTARTS AGE calico-kube-controllers-658d97c59c-vkk2n 0/1 Pending 0 144m calico-node-2gjfd 0/1 Init:ImagePullBackOff 0 144m coredns-66f779496c-562gq 0/1 Pending 0 152m coredns-66f779496c-v4msh 0/1 Pending 0 152m etcd-master 1/1 Running 0 152m kube-apiserver-master 1/1 Running 0 152m kube-controller-manager-master 1/1 Running 0 152m kube-proxy-kwp62 1/1 Running 0 152m kube-scheduler-master 1/1 Running 0 152m等待几分钟后，calcio插件会变成Running状态kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-658d97c59c-9z9km 1/1 Running 0 9m5s kube-system calico-node-7ftlf 1/1 Running 0 9m6s kube-system coredns-66f779496c-n7h9t 1/1 Running 0 20m kube-system coredns-66f779496c-x7tvt 1/1 Running 0 20m kube-system etcd-master 1/1 Running 0 20m kube-system kube-apiserver-master 1/1 Running 0 20m kube-system kube-controller-manager-master 1/1 Running 0 20m kube-system kube-proxy-tdp72 1/1 Running 0 20m kube-system kube-scheduler-master 1/1 Running 0 20m

Containerd 常见命令 nerdctlnerdctl是containerd客户端工具，使用语法与docker一致，推荐使用。地址：https://github.com/containerd/nerdctl/releases，其中完整版（nerdctl-full-2.0.4-linux-amd64.tar.gz）包含containerd以及runc、CNI等依赖。nerctl实现了许多docker不具备的功能，如延迟拉取镜像（lazy-pulling）、镜像加密（imgcrypt）等功能。containerd常见命令crictl：遵循 CRI 接口规范的一个命令行工具，通常用它来检查和管理容器运行时和镜像。ctr 是 containerd 的一个客户端工具命令dockerctrcrictl查看运行的容器docker psctr task ls/ctr container lscrictl ps查看镜像docker imagesctr image lscrictl images查看容器日志docker logs-crictl logs查看容器数据信息docker inspectctr container infocrictl inspect查看容器资源docker stats-crictl stats启动/关闭已有的容器docker start/stopctr task start/killcrictl start/stop运行一个新的容器docker runctr run-(最小单元为pod)打标签docker tagctr inage tag-创建一个新的容器docker createctr container createcrictl create导入镜像docker loadcri image import-导出镜像docker savecri image export-删除容器docker rmctr container rmcrictl rm删除镜像docker rmictr image rmcrictl rmi拉取镜像docker pullctr image pullcrictl pull推送镜像docker pushctr image push-登录或在容器内部执行命令docker exec-crictl exec清空不用的容器docker image prune-crictl rmi --prune

containerd安装与配置 新版本的kubernetes不在使用docker作为容器运行时，而是直接使用containerd。因此在安装kubernetes时需要安装containerd。安装安装containerd非常简单，配置好yum仓库，一条yum命令即可安装成功。#containerd网络仓库配置 dnf install -y yum-utils yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo 添加仓库自：https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yum -y install containerd安装完成之后需要修改配置文件。containerd安装完成后会默认在/etc/containerd/目录下生成一个配置文件config.toml，但是该文件缺少很多信息，因此我们需要重新生成一个配置文件，然后对新生成的配置文件进行修改。#生成新配置文件 containerd config default > /etc/containerd/config.toml修改配置信息修改Cgroup为systemd在[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] 下找到SystemdCgroup，将false改为true修改sandbox镜像地址把配置文件中的 sandbox_image = "registry.k8s.io/pause:3.6" 改为阿里云的地址 sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.8"sandbox_image 是容器运行时（如 containerd、CRI-O 等）或容器编排系统（如 Kubernetes）中一个关键配置参数，用于指定 沙箱容器（Sandbox Container） 的基础镜像。沙箱容器是一个特殊的容器，用于为其他容器提供共享的底层环境（如网络、IPC 命名空间等）。例如：在 Kubernetes 中，每个 Pod 的第一个容器就是沙箱容器（称为 pause 容器），它负责维持 Pod 的网络命名空间。配置镜像加速# 修改配置文件，添加配置镜像加速相关信息的配置文件的路径 vim /etc/containerd/config.toml [plugins."io.containerd.grpc.v1.cri".registry] config_path = "/etc/containerd/certs.d" # 创建文件夹 mkdir -p /etc/containerd/certs.d/docker.io #创建配置文件 cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF server = "https://docker.io" [host."https://docker.1ms.run"] capabilities = ["pull", "resolve"] EOF说明：该配置方式对crictl命令有效，对ctr命令无效，使用ctr下载镜像时，需要指定镜像的地址和版本，如： ctr --debug=true image pull docker.1ms.run/busybox:latest{alert type="info"}特别提醒：在/etc/containerd/certs.d目录下，每个镜像加速地址都要放在单独的文件夹中，且这个文件夹必须以.io结尾，镜像加速地址的文件名称必须为hosts.toml{/alert}tree containerd/ containerd/ ├── certs.d │   └── docker.io │   └── hosts.toml └── config.toml 2 directories, 2 files 安装crictl工具#下载源码包 wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.29.0/crictl-v1.29.0-linux-amd64.tar.gz #解压缩 tar zxvf crictl-v1.29.0-linux-amd64.tar.gz -C /usr/local/bin # -C：指定解压缩后文件存放的位置修改配置文件cat > /etc/crictl.yaml <<EOF runtime-endpoint: unix:///run/containerd/containerd.sock image-endpoint: unix:///run/containerd/containerd.sock timeout: 10 debug: false EOF # 重启containerd systemctl restart containerd或者通过命令行工具修改crictl config runtime-endpoint unix:///run/containerd/containerd.sock crictl config image-endpoint unix:///run/containerd/containerd.sock安装cni插件在没有安装cni插件时，containerd会报错缺少cni插件。#下载安装包 wget https://github.com/containernetworking/plugins/releases/download/v1.4.0/cni-plugins-linux-amd64-v1.4.0.tgz #创建目录 mkdir -p /opt/cni/bin/ #解压缩 tar xzvf cni-plugins-linux-amd64-v1.4.0.tgz -C /opt/cni/bin/修改配置文件mkdir -p /etc/cni/net.d/ cat > /etc/cni/net.d/10-containerd-net.conflist <<EOF { "cniVersion": "1.0.0", "name": "containerd-net", "plugins": [ { "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "promiscMode": true, "ipam": { "type": "host-local", "ranges": [ [{ "subnet": "10.88.0.0/16" }], [{ "subnet": "2001:4860:4860::/64" }] ], "routes": [ { "dst": "0.0.0.0/0" }, { "dst": "::/0" } ] } }, { "type": "portmap", "capabilities": {"portMappings": true} } ] } EOF [root@rocky9 ~]# systemctl restart containerd [root@rocky9 ~]# crictl info …… #重启containerd即可下载镜像示例[root@node-001 docker.io]# crictl pull httpd Image is up to date for sha256:83d938198316505b3aebd52bed1e54e5f2a49591cc41e09a30f480e5c00ea0cf [root@node-001 docker.io]# crictl pull nginx Image is up to date for sha256:53a18edff8091d5faff1e42b4d885bc5f0f897873b0b8f0ace236cd5930819b0 [root@node-001 docker.io]# crictl pull bitnami/php-fpm Image is up to date for sha256:0fdfdfd1cd46d5ef32ed47af20e8ae3ad58b0f4d62516519994bf74b8f6611a5 [root@node-001 docker.io]# crictl images ls IMAGE TAG IMAGE ID SIZE docker.1ms.run/calico/cni v3.25.0 d70a5947d57e5 88MB docker.1ms.run/calico/node v3.25.0 08616d26b8e74 87.2MB docker.io/bitnami/php-fpm latest 0fdfdfd1cd46d 115MB docker.io/library/httpd latest 83d9381983165 58.5MB docker.io/library/nginx latest 53a18edff8091 72.2MB registry.aliyuncs.com/google_containers/kube-proxy v1.28.0 ea1030da44aa1 24.6MB registry.aliyuncs.com/google_containers/pause 3.9 e6f1816883972 322kB