前言
Kubernetes（简称K8s）作为当今最流行的容器编排平台，已经成为企业部署和管理容器化应用的事实标准。而高可用（High Availability, HA）的Kubernetes集群则是生产环境中的核心要求。本文将详细介绍如何使用kubeadm工具从零开始搭建一个高可用的Kubernetes集群环境。
什么是高可用Kubernetes集群？
高可用Kubernetes集群是指在集群中的控制平面（Control Plane）和工作节点（Worker Nodes）出现故障时，集群仍能继续提供服务的能力。具体来说，高可用集群需要满足以下条件：
控制平面高可用
：多个Master节点，避免单点故障
数据存储高可用
：使用外部的高可用存储（如etcd集群）
网络高可用
：网络插件支持高可用部署
监控与恢复
：完善的监控和自动恢复机制
为什么选择kubeadm？
kubeadm是Kubernetes官方提供的集群部署工具，具有以下优势：
简单易用
：通过简单的命令即可完成集群初始化
最佳实践
：遵循Kubernetes社区的最佳实践
可扩展性强
：支持自定义配置和插件集成
社区支持
：拥有活跃的社区和丰富的文档
环境准备
在开始搭建之前，我们需要准备以下环境和资源。
硬件要求
对于生产环境的高可用集群，建议满足以下硬件配置：
节点类型
CPU
内存
磁盘
网络
Master节点
4核+
8GB+
100GB+ SSD
千兆网卡
Worker节点
2核+
4GB+
50GB+
千兆网卡
etcd节点
2核+
4GB+
50GB+ SSD
千兆网卡
软件要求
操作系统
：Ubuntu 20.04 LTS / CentOS 7.9 / RHEL 8.4+
Docker版本
：19.03+ 或 containerd 1.4+
kubeadm版本
：1.20+（建议与Kubernetes版本保持一致）
网络插件
：Calico / Flannel / Cilium
网络规划
在部署之前，需要规划好网络：
Pod网络
：10.244.0.0/16（示例）
Service网络
：10.96.0.0/12（示例）
API Server地址
：VIP（虚拟IP）+ 端口（6443）
etcd集群端口
：2379, 2380
节点规划
本次教程规划以下节点：
节点角色
主机名
IP地址
说明
Master节点
k8s-master-01
192.168.1.101
主Master节点
Master节点
k8s-master-02
192.168.1.102
备用Master节点
Master节点
k8s-master-03
192.168.1.103
备用Master节点
Worker节点
k8s-worker-01
192.168.1.201
工作节点
Worker节点
k8s-worker-02
192.168.1.202
工作节点
虚拟IP
VIP
192.168.1.100
API Server高可用地址
基础环境配置
1. 系统初始化（所有节点）
1.1 配置主机名和hosts
在所有节点上执行：
# 在k8s-master-01上执行
hostnamectl set-hostname k8s-master-01
cat >> /etc/hosts << EOF
192.168.1.101 k8s-master-01
192.168.1.102 k8s-master-02
192.168.1.103 k8s-master-03
192.168.1.201 k8s-worker-01
192.168.1.202 k8s-worker-02
192.168.1.100 k8s-vip
EOF
1.2 关闭Swap分区
# 临时关闭
swapoff -a
# 永久关闭
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
1.3 配置内核参数
# 加载br_netfilter模块
modprobe br_netfilter
# 配置内核参数
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
# 应用配置
sysctl -p /etc/sysctl.d/k8s.conf
1.4 配置时间同步
# 安装chrony
apt-get install chrony -y # Ubuntu/Debian
# 或
yum install chrony -y # CentOS/RHEL
# 配置时间同步
systemctl enable chrony
systemctl start chrony
# 验证时间同步
chronyc sources -v
2. 安装容器运行时
2.1 安装Docker（所有节点）
# 卸载旧版本
apt-get remove docker docker-engine docker.io containerd runc
# 安装依赖
apt-get update
apt-get install -y apt-transport-https ca-certificates curl software-properties-common gnupg2
# 添加Docker官方GPG密钥
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
# 添加Docker仓库
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
# 安装Docker
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io
# 配置Docker daemon
cat > /etc/docker/daemon.json << EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
# 启动Docker
systemctl enable docker
systemctl start docker
2.2 安装containerd（可选，推荐生产环境使用）
# 配置containerd
cat > /etc/containerd/config.toml << EOF
version = 2
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
EOF
# 启动containerd
systemctl enable containerd
systemctl start containerd
3. 安装kubeadm、kubelet和kubectl
3.1 配置Kubernetes仓库
# 添加Kubernetes GPG密钥
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
# 添加Kubernetes仓库
cat > /etc/apt/sources.list.d/kubernetes.list << EOF
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
# 更新包索引
apt-get update
3.2 安装Kubernetes组件
# 安装指定版本（推荐）
apt-get install -y kubelet=1.27.3-00 kubeadm=1.27.3-00 kubectl=1.27.3-00
# 或安装最新版本
apt-get install -y kubelet kubeadm kubectl
# 锁定版本（防止意外升级）
apt-mark hold kubelet kubeadm kubectl
# 启动kubelet
systemctl enable kubelet
systemctl start kubelet
4. 配置内核模块加载
# 配置modules-load.d
cat > /etc/modules-load.d/k8s.conf << EOF
overlay
br_netfilter
EOF
# 加载模块
modprobe overlay
modprobe br_netfilter
高可用etcd集群搭建
1. etcd集群规划
etcd是Kubernetes的后端存储，高可用的etcd集群是高可用Kubernetes的基础。我们使用3个节点搭建etcd集群：
节点
主机名
IP地址
etcd端口
etcd-01
k8s-master-01
192.168.1.101
2379, 2380
etcd-02
k8s-master-02
192.168.1.102
2379, 2380
etcd-03
k8s-master-03
192.168.1.103
2379, 2380
2. 生成etcd证书
使用cfssl工具生成etcd集群的TLS证书。
2.1 安装cfssl
# 下载cfssl工具
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64
# 安装
chmod +x cfssl_1.6.4_linux_amd64
mv cfssl_1.6.4_linux_amd64 /usr/local/bin/cfssl
chmod +x cfssljson_1.6.4_linux_amd64
mv cfssljson_1.6.4_linux_amd64 /usr/local/bin/cfssljson
2.2 创建CA配置文件
# 创建证书目录
mkdir -p /opt/etcd/ssl
cd /opt/etcd/ssl
# 创建CA配置文件
cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"etcd": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
# 创建CA证书请求文件
cat > ca-csr.json << EOF
{
"CN": "etcd-ca",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "etcd",
"OU": "etcd Security"
}
]
}
EOF
# 生成CA证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2.3 创建etcd证书请求文件
# 创建etcd证书请求文件
cat > etcd-csr.json << EOF
{
"CN": "etcd",
"hosts": [
"192.168.1.101",
"192.168.1.102",
"192.168.1.103",
"k8s-master-01",
"k8s-master-02",
"k8s-master-03",
"localhost",
"127.0.0.1"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "etcd",
"OU": "etcd Security"
}
]
}
EOF
# 生成etcd证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd
2.4 分发证书到所有etcd节点
# 在k8s-master-01上执行
for node in k8s-master-01 k8s-master-02 k8s-master-03; do
ssh $node "mkdir -p /opt/etcd/ssl"
scp /opt/etcd/ssl/* $node:/opt/etcd/ssl/
done
3. 部署etcd集群
3.1 创建etcd systemd服务文件
在每个etcd节点上创建etcd服务文件：
# 在k8s-master-01上创建
cat > /etc/systemd/system/etcd.service << EOF
[Unit]
Description=etcd
After=network.target
[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \\
--name etcd-01 \\
--data-dir /var/lib/etcd \\
--listen-client-urls https://192.168.1.101:2379 \\
--advertise-client-urls https://192.168.1.101:2379 \\
--listen-peer-urls https://192.168.1.101:2380 \\
--initial-advertise-peer-urls https://192.168.1.101:2380 \\
--initial-cluster etcd-01=https://192.168.1.101:2380,etcd-02=https://192.168.1.102:2380,etcd-03=https://192.168.1.103:2380 \\
--initial-cluster-token etcd-cluster \\
--initial-cluster-state new \\
--client-cert-auth \\
--trusted-ca-file /opt/etcd/ssl/ca.pem \\
--cert-file /opt/etcd/ssl/etcd.pem \\
--key-file /opt/etcd/ssl/etcd-key.pem \\
--peer-client-cert-auth \\
--peer-trusted-ca-file /opt/etcd/ssl/ca.pem \\
--peer-cert-file /opt/etcd/ssl/etcd.pem \\
--peer-key-file /opt/etcd/ssl/etcd-key.pem \\
--wal-dir /var/lib/etcd/wal \\
--max-request-bytes 10485760 \\
--quota-backend-bytes 8589934592 \\
--listen-metrics-urls http://192.168.1.101:2381
Restart=always
RestartSec=10
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
3.2 修改其他节点配置
在k8s-master-02和k8s-master-03上创建类似的服务文件，只需修改
--name
和
--listen-client-urls
、
--advertise-client-urls
、
--listen-peer-urls
、
--initial-advertise-peer-urls
中的IP地址。
3.3 启动etcd集群
# 在所有etcd节点上执行
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
# 检查etcd状态
systemctl status etcd
# 查看etcd日志
journalctl -u etcd -f
3.4 验证etcd集群
# 在任意etcd节点上执行
ETCDCTL_API=3 etcdctl \\
--endpoints=https://192.168.1.101:2379,https://192.168.1.102:2379,https://192.168.1.103:2379 \\
--cacert=/opt/etcd/ssl/ca.pem \\
--cert=/opt/etcd/ssl/etcd.pem \\
--key=/opt/etcd/ssl/etcd-key.pem \\
endpoint health
# 查看集群成员
ETCDCTL_API=3 etcdctl \\
--endpoints=https://192.168.1.101:2379 \\
--cacert=/opt/etcd/ssl/ca.pem \\
--cert=/opt/etcd/ssl/etcd.pem \\
--key=/opt/etcd/ssl/etcd-key.pem \\
member list
高可用Kubernetes控制平面搭建
1. 配置Keepalived + HAProxy
为了实现API Server的高可用，我们需要配置Keepalived和HAProxy。
1.1 安装Keepalived和HAProxy
在所有Master节点上安装：
# Ubuntu/Debian
apt-get install keepalived haproxy -y
# CentOS/RHEL
yum install keepalived haproxy -y
1.2 配置HAProxy
在所有Master节点上创建相同的HAProxy配置：
cat > /etc/haproxy/haproxy.cfg << EOF
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
log global
mode tcp
option tcplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
frontend kube_apiserver
bind *:6443
mode tcp
option tcplog
default_backend kube_apiserver
backend kube_apiserver
mode tcp
balance roundrobin
option tcp-check
server k8s-master-01 192.168.1.101:6443 check fall 3 rise 2
server k8s-master-02 192.168.1.102:6443 check fall 3 rise 2
server k8s-master-03 192.168.1.103:6443 check fall 3 rise 2
listen stats
bind *:8080
mode http
stats enable
stats uri /
stats refresh 15s
stats show-legends
stats show-node
EOF
1.3 配置Keepalived
在k8s-master-01上配置：
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
router_id k8s-master-01
script_user root
enable_script_security
}
vrrp_script check_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 3
weight -2
fall 10
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 4be37dc3b4c0
}
virtual_ipaddress {
192.168.1.100
}
track_script {
check_apiserver
}
}
EOF
在k8s-master-02上配置：
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
router_id k8s-master-02
script_user root
enable_script_security
}
vrrp_script check_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 3
weight -2
fall 10
rise 2
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 4be37dc3b4c0
}
virtual_ipaddress {
192.168.1.100
}
track_script {
check_apiserver
}
}
EOF
在k8s-master-03上配置：
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
router_id k8s-master-03
script_user root
enable_script_security
}
vrrp_script check_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 3
weight -2
fall 10
rise 2
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 4be37dc3b4c0
}
virtual_ipaddress {
192.168.1.100
}
track_script {
check_apiserver
}
}
EOF
1.4 创建健康检查脚本
在所有Master节点上创建健康检查脚本：
cat > /etc/keepalived/check_apiserver.sh << EOF
#!/bin/sh
errorExit() {
echo "*** $*" 1>&2
exit 1
}
curl –silent –max-time 2 –insecure https://localhost:6443/ -o /dev/null || errorExit “Error GET https://localhost:6443/”
if ip addr | grep -q 192.168.1.100; then
curl –silent –max-time 2 –insecure https://192.168.1.100:6443/ -o /dev/null || errorExit “Error GET https://192.168.1.100:6443/”
fi
EOF
chmod +x /etc/keepalived/check_apiserver.sh
1.5 启动Keepalived和HAProxy
# 启动HAProxy
systemctl enable haproxy
systemctl start haproxy
# 启动Keepalived
systemctl enable keepalived
systemctl start keepalived
# 检查状态
systemctl status haproxy
systemctl status keepalived
# 查看VIP是否漂移
ip addr show
2. 初始化Kubernetes集群
2.1 创建kubeadm配置文件
在k8s-master-01上创建kubeadm配置文件：
cat > /opt/kubeadm-config.yaml << EOF
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
bootstrapTokens:
- token: "9a08jv.c0izixklc2m8d3ex"
description: "kubeadm bootstrap token"
ttl: "24h"
- token: "9a08jv.c0izixklc2m8d3ex"
description: "kubeadm bootstrap token"
ttl: "24h"
nodeRegistration:
criSocket: "unix:///var/run/containerd/containerd.sock"
kubeletExtraArgs:
node-ip: 192.168.1.101
taints: null
localAPIEndpoint:
advertiseAddress: 192.168.1.101
bindPort: 6443
certificateKey: "0123456789abcdef0123456789abcdef"
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.27.3
controlPlaneEndpoint: "192.168.1.100:6443"
networking:
podSubnet: "10.244.0.0/16"
serviceSubnet: "10.96.0.0/12"
dnsDomain: "cluster.local"
apiServer:
extraArgs:
authorization-mode: "Node,RBAC"
advertise-address: 192.168.1.101
bind-address: 0.0.0.0
certSANs:
- "192.168.1.100"
- "192.168.1.101"
- "192.168.1.102"
- "192.168.1.103"
- "k8s-vip"
- "k8s-master-01"
- "k8s-master-02"
- "k8s-master-03"
- "localhost"
- "127.0.0.1"
controllerManager:
extraArgs:
bind-address: 0.0.0.0
scheduler:
extraArgs:
bind-address: 0.0.0.0
etcd:
external:
endpoints:
- https://192.168.1.101:2379
- https://192.168.1.102:2379
- https://192.168.1.103:2379
caFile: /opt/etcd/ssl/ca.pem
certFile: /opt/etcd/ssl/etcd.pem
keyFile: /opt/etcd/ssl/etcd-key.pem
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
EOF
2.2 初始化集群
# 拉取所需镜像
kubeadm config images pull --config /opt/kubeadm-config.yaml
# 初始化集群
kubeadm init --config /opt/kubeadm-config.yaml --upload-certs
# 注意：初始化成功后，会输出类似以下内容：
# Your Kubernetes control-plane has initialized successfully!
#
# To start using your cluster, you need to run the following as a regular user:
#
# mkdir -p $HOME/.kube
# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config
#
# Alternatively, if you are the root user, you can run:
#
# export KUBECONFIG=/etc/kubernetes/admin.conf
#
# You should now deploy a pod network to the cluster.
# Read "docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md" for details.
#
# You can now join any number of the control-plane node running the following command on each as root:
#
# kubeadm join 192.168.1.100:6443 --token 9a08jv.c0izixklc2m8d3ex \
# --discovery-token-ca-cert-hash sha256:6170c3255d03e2c5e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8 \
# --control-plane --certificate-key 0123456789abcdef0123456789abcdef
#
# Then you can join any number of worker nodes by running the following on each as root:
#
# kubeadm join 192.168.1.100:6443 --token 9a08jv.c0izixklc2m8d3ex \
# --discovery-token-ca-cert-hash sha256:6170c3255d03e2c5e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8
2.3 配置kubectl
# 配置kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 验证集群状态
kubectl get nodes
kubectl get pods -n kube-system
3. 添加其他Master节点
3.1 手动分发证书
在k8s-master-01上执行：
# 分发证书到其他Master节点
for node in k8s-master-02 k8s-master-03; do
ssh $node "mkdir -p /etc/kubernetes/pki/etcd"
scp /etc/kubernetes/pki/ca.crt $node:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/ca.key $node:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.key $node:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.pub $node:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.crt $node:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.key $node:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/etcd/ca.crt $node:/etc/kubernetes/pki/etcd/
scp /etc/kubernetes/pki/etcd/ca.key $node:/etc/kubernetes/pki/etcd/
done
3.2 添加其他Master节点
在k8s-master-02和k8s-master-03上执行：
# 使用初始化时输出的join命令
kubeadm join 192.168.1.100:6443 --token 9a08jv.c0izixklc2m8d3ex \
--discovery-token-ca-cert-hash sha256:6170c3255d03e2c5e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8 \
--control-plane --certificate-key 0123456789abcdef0123456789abcdef
# 配置kubectl
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
4. 添加Worker节点
在k8s-worker-01和k8s-worker-02上执行：
# 使用初始化时输出的join命令（不带--control-plane参数）
kubeadm join 192.168.1.100:6443 --token 9a08jv.c0izixklc2m8d3ex \
--discovery-token-ca-cert-hash sha256:6170c3255d03e2c5e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8e8
网络插件部署
1. 安装Calico网络插件
Calico是Kubernetes推荐的网络插件，支持网络策略和高可用部署。
1.1 下载Calico manifest文件
# 下载Calico manifest
curl https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml -o calico.yaml
1.2 修改Calico配置
# 编辑calico.yaml，修改IP池配置
# 找到以下内容并修改：
# - name: CALICO_IPV4POOL_CIDR
# value: "10.244.0.0/16"
# 如果需要修改节点间通信接口，可以添加：
# - name: IP_AUTODETECTION_METHOD
# value: "interface=eth0"
1.3 应用Calico配置
# 应用Calico
kubectl apply -f calico.yaml
# 检查Calico Pod状态
kubectl get pods -n kube-system -w
2. 验证网络插件
# 检查节点状态
kubectl get nodes
# 检查Pod状态
kubectl get pods -n kube-system
# 检查网络插件日志
kubectl logs -n kube-system -l k8s-app=calico-node
验证集群高可用性
1. 检查集群状态
# 检查节点状态
kubectl get nodes -o wide
# 检查控制平面Pod状态
kubectl get pods -n kube-system -l component=kube-apiserver
kubectl get pods -n kube-system -l component=kube-controller-manager
kubectl get pods -n kube-system -l component=kube-scheduler
# 检查etcd集群状态
kubectl get pods -n kube-system -l k8s-app=etcd
2. 测试故障转移
2.1 测试API Server故障转移
# 查看当前VIP在哪台节点
ip addr show | grep 192.168.1.100
# 停止k8s-master-01上的HAProxy
systemctl stop haproxy
# 检查VIP是否漂移到其他节点
ip addr show | grep 192.168.1.100
# 验证集群是否仍然可用
kubectl get nodes
# 恢复k8s-master-01的HAProxy
systemctl start haproxy
2.2 测试etcd故障转移
# 停止k8s-master-01上的etcd
systemctl stop etcd
# 检查etcd集群状态
ETCDCTL_API=3 etcdctl \\
--endpoints=https://192.168.1.102:2379,https://192.168.1.103:2379 \\
--cacert=/opt/etcd/ssl/ca.pem \\
--cert=/opt/etcd/ssl/etcd.pem \\
--key=/opt/etcd/ssl/etcd-key.pem \\
endpoint health
# 验证集群是否仍然可用
kubectl get nodes
# 恢复etcd
systemctl start etcd
3. 部署测试应用
# 创建测试Deployment
cat > test-deployment.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-test
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.23
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-test
spec:
selector:
app: nginx
ports:
- port: 80
targetPort: 80
type: ClusterIP
EOF
kubectl apply -f test-deployment.yaml
# 检查Pod状态
kubectl get pods -l app=nginx
# 测试Service
kubectl run curl-test --image=curlimages/curl --rm -it -- curl nginx-test
生产环境优化建议
1. 安全加固
1.1 配置RBAC权限
# 创建最小权限ServiceAccount
cat > serviceaccount.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: limited-sa
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: limited-role
namespace: default
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: limited-binding
namespace: default
subjects:
- kind: ServiceAccount
name: limited-sa
namespace: default
roleRef:
kind: Role
name: limited-role
apiGroup: rbac.authorization.k8s.io
EOF
kubectl apply -f serviceaccount.yaml
1.2 配置Pod安全策略（PSP）
# 创建PodSecurityPolicy
cat > psp.yaml << EOF
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted
spec:
privileged: false
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
- 'persistentVolumeClaim'
runAsUser:
rule: 'MustRunAsNonRoot'
seLinux:
rule: 'RunAsAny'
fsGroup:
rule: 'MustRunAs'
ranges:
- min: 1
max: 65535
readOnlyRootFilesystem: true
EOF
kubectl apply -f psp.yaml
2. 监控与日志
2.1 部署Prometheus + Grafana
# 使用kube-prometheus-stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace \
--set grafana.adminPassword=admin123
2.2 部署EFK日志收集
# 部署Elasticsearch + Fluentd + Kibana
kubectl apply -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch-rbac.yaml
3. 备份与恢复
3.1 备份etcd数据
# 创建备份脚本
cat > /opt/etcd-backup.sh << EOF
#!/bin/bash
BACKUP_DIR="/opt/etcd-backup"
DATE=\$(date +%Y%m%d%H%M%S)
mkdir -p \$BACKUP_DIR
ETCDCTL_API=3 etcdctl \\
--endpoints=https://192.168.1.101:2379,https://192.168.1.102:2379,https://192.168.1.103:2379 \\
--cacert=/opt/etcd/ssl/ca.pem \\
--cert=/opt/etcd/ssl/etcd.pem \\
--key=/opt/etcd/ssl/etcd-key.pem \\
snapshot save \$BACKUP_DIR/etcd-snapshot-\$DATE.db
# 保留最近7天的备份
find \$BACKUP_DIR -name "etcd-snapshot-*.db" -mtime +7 -delete
echo "Backup completed: \$BACKUP_DIR/etcd-snapshot-\$DATE.db"
EOF
chmod +x /opt/etcd-backup.sh
# 添加到crontab
echo "0 2 * * * /opt/etcd-backup.sh" >> /etc/crontab
3.2 恢复etcd数据
# 停止etcd服务
systemctl stop etcd
# 恢复快照
ETCDCTL_API=3 etcdctl snapshot restore /opt/etcd-backup/etcd-snapshot-20230701120000.db \\
–data-dir /var/lib/etcd-new \\
–name etcd-01 \\
–initial-cluster etcd-01=https://192.168.1.101:2380,etcd-02=https://192.168.1.102:2380,etcd-03=https://192.168.1.103:2380 \\
–initial-cluster-token etcd-cluster \\
–initial-advertise-peer-urls https://192.168.1.101:2380
# 备份旧数据
mv /var/lib/etcd /var/lib/etcd-old
mv /var/lib/etcd-new /var/lib/etcd
# 启动etcd
systemctl start etcd
常见问题排查
1. 节点无法加入集群
问题描述
：执行
kubeadm join
时失败，提示证书错误或网络不通。
排查步骤
：
# 1. 检查网络连通性
ping 192.168.1.100
telnet 192.168.1.100 6443
# 2. 检查token是否有效
kubeadm token list
# 3. 重新生成token
kubeadm token create –print-join-command
# 4. 检查时间同步
date
chronyc sources -v
# 5. 检查防火墙
iptables -L -n | grep 6443
2. Pod无法分配IP
问题描述
：Pod处于ContainerCreating状态，无法获取IP地址。
排查步骤
：
# 1. 检查CNI插件状态
kubectl get pods -n kube-system -l k8s-app=calico-node
# 2. 检查CNI插件日志
kubectl logs -n kube-system -l k8s-app=calico-node
# 3. 检查节点网络配置
ip addr show
ip route show
# 4. 检查iptables规则
iptables -L -n -t nat | grep KUBE-SERVICES
# 5. 重启CNI插件
kubectl delete pods -n kube-system -l k8s-app=calico-node
3. API Server无法访问
问题描述
：kubectl命令无法连接API Server。
排查步骤
：
# 1. 检查kubeconfig
echo $KUBECONFIG
cat ~/.kube/config
# 2. 检查API Server状态
kubectl get pods -n kube-system -l component=kube-apiserver
# 3. 检查HAProxy状态
systemctl status haproxy
netstat -tlnp | grep 6443
# 4. 检查Keepalived状态
systemctl status keepalived
ip addr show | grep 192.168.1.100
# 5. 检查API Server日志
kubectl logs -n kube-system -l component=kube-apiserver
4. etcd集群不健康
问题描述
：etcd集群无法正常工作，数据同步异常。
排查步骤
：
# 1. 检查etcd集群状态
ETCDCTL_API=3 etcdctl \\
–endpoints=https://192.168.1.101:2379,https://192.168.1.102:2379,https://192.168.1.103:2379 \\
–cacert=/opt/etcd/ssl/ca.pem \\
–cert=/opt/etcd/ssl/etcd.pem \\
–key=/opt/etcd/ssl/etcd-key.pem \\
endpoint health
# 2. 检查etcd日志
journalctl -u etcd -f
# 3. 检查etcd成员
ETCDCTL_API=3 etcdctl \\
–endpoints=https://192.168.1.101:2379 \\
–cacert=/opt/etcd/ssl/ca.pem \\
–cert=/opt/etcd/ssl/etcd.pem \\
–key=/opt/etcd/ssl/etcd-key.pem \\
member list
# 4. 检查磁盘空间
df -h /var/lib/etcd
# 5. 检查网络连通性
telnet 192.168.1.102 2379
telnet 192.168.1.103 2379
总结
通过本文的详细指导，您应该已经成功搭建了一个高可用的Kubernetes集群环境。整个过程包括：
环境准备
：硬件、软件和网络规划
基础配置
：系统初始化、容器运行时安装
etcd集群
：高可用的分布式存储
控制平面
：Keepalived + HAProxy实现API Server高可用
集群初始化
：使用kubeadm初始化集群
网络插件
：部署Calico实现容器网络
验证测试
：验证集群高可用性
生产优化
：安全、监控、备份等最佳实践
关键要点回顾
高可用etcd
：使用3个节点，通过TLS证书保证安全通信
API Server高可用
：使用Keepalived实现VIP漂移，HAProxy实现负载均衡
证书管理
：使用cfssl生成所有必要的TLS证书
网络插件
：Calico提供稳定的容器网络和网络策略
故障恢复
：完善的备份和恢复机制
后续维护建议
定期更新
：关注Kubernetes安全公告，及时更新版本
监控告警
：建立完善的监控体系，及时发现和解决问题
定期演练
：定期进行故障转移演练，确保高可用机制有效
文档更新
：记录所有配置变更和故障处理过程
权限管理
：遵循最小权限原则，定期审计权限配置
通过遵循这些最佳实践，您可以确保Kubernetes集群在生产环境中稳定、安全、高效地运行。