引言：自动化部署Kubernetes的重要性
在现代DevOps实践中，Kubernetes已经成为容器编排的事实标准。然而，手动部署Kubernetes集群是一项复杂且容易出错的任务，涉及多个组件的配置、网络设置、证书管理等。使用Ansible Playbook自动化部署Kubernetes不仅能大幅提高效率，还能确保部署的一致性和可重复性。
为什么选择Ansible？
Ansible是一个简单但功能强大的自动化工具，具有以下优势：
无代理架构
：只需SSH即可工作，无需在目标节点安装额外软件
声明式语法
：使用YAML格式，易于理解和维护
模块化设计
：丰富的内置模块覆盖几乎所有运维场景
幂等性
：多次执行相同Playbook不会产生副作用
Kubernetes部署的常见痛点
在手动部署Kubernetes时，我们通常会遇到以下挑战：
组件版本兼容性
：不同版本的kubelet、kube-apiserver等组件需要严格匹配
证书管理
：PKI基础设施复杂，证书过期会导致集群不可用
网络配置
：CNI插件选择和配置容易出错
系统依赖
：内核参数、防火墙规则、swap状态等需要预先配置
高可用部署
：多主节点配置复杂，需要处理负载均衡和etcd集群
前置准备
环境要求
硬件要求
：
控制节点：2核CPU，4GB内存，20GB存储
工作节点：4核CPU，8GB内存，40GB存储
网络：所有节点间网络互通，特别是6443、2379-2380、10250等端口
软件要求
：
Ansible 2.9+
Python 3.6+
目标节点：Ubuntu 20.04 LTS / CentOS
7
⁄
8
/ RHEL
7
⁄
8
网络规划
在开始之前，需要规划以下网络参数：
Pod CIDR
：10.244.0.0/16
Service CIDR
：10.96.0.0/12
Cluster DNS
：10.96.0.10
API Server VIP
：192.168.1.100（用于高可用）
Ansible Playbook架构设计
项目结构
k8s-ansible/
├── inventory/
│ ├── hosts.ini
│ └── group_vars/
│ ├── all.yml
│ ├── masters.yml
│ └── workers.yml
├── roles/
│ ├── common/
│ │ ├── tasks/
│ │ │ └── main.yml
│ │ └── handlers/
│ │ └── main.yml
│ ├── etcd/
│ ├── master/
│ ├── worker/
│ └── network/
├── playbooks/
│ ├── deploy.yml
│ ├── upgrade.yml
│ └── destroy.yml
└── templates/
├── kubelet.service.j2
├── kube-apiserver.service.j2
└── kubeadm-config.yaml.j2
变量管理
在
inventory/group_vars/all.yml
中定义全局变量：
# 全局Kubernetes配置
k8s_version: “1.28.2”
container_runtime: “containerd”
cni_plugin: “calico”
# 网络配置
pod_cidr: “10.244.0.0/16”
service_cidr: “10.96.0.0/12”
cluster_dns: “10.96.0.10”
# 证书配置
cert有效期: 3650
cert_location: “/etc/kubernetes/pki”
# 高可用配置
loadbalancer:
enabled: false
vip: “192.168.1.100”
port: 6443
核心角色实现
1. Common角色：系统基础配置
Common角色负责所有节点的基础配置，包括禁用swap、配置内核参数、安装依赖包等。
tasks/main.yml
:
—
– name: 禁用swap分区
command: swapoff -a
when: ansible_swaptotal_mb > 0
– name: 永久禁用swap
replace:
path: /etc/fstab
regexp: ‘^([^#].*?\sswap\s+.*)$’
replace: ‘# \1’
– name: 加载必要的内核模块
modprobe:
name: “{{ item }}”
state: present
loop:
– overlay
– br_netfilter
– name: 配置sysctl参数
sysctl:
name: “{{ item.key }}”
value: “{{ item.value }}”
state: present
reload: yes
loop:
– { key: ‘net.bridge.bridge-nf-call-iptables’, value: ‘1’ }
– { key: ‘net.bridge.bridge-nf-call-ip6tables’, value: ‘1’ }
– { key: ‘net.ipv4.ip_forward’, value: ‘1’ }
– name: 配置iptables规则
iptables:
chain: “{{ item.chain }}”
policy: “{{ item.policy }}”
loop:
– { chain: ‘FORWARD’, policy: ‘ACCEPT’ }
– { chain: ‘INPUT’, policy: ‘ACCEPT’ }
– name: 安装基础依赖包
package:
name: “{{ item }}”
state: present
loop:
– curl
– wget
– git
– vim
– apt-transport-https
– ca-certificates
– software-properties-common
– gnupg2
– name: 配置ulimit
pam_limits:
domain: ‘*’
limit_type: “{{ item.type }}”
limit_item: “{{ item.item }}”
value: “{{ item.value }}”
loop:
– { type: ‘soft’, item: ‘nofile’, value: ‘65536’ }
– { type: ‘hard’, item: ‘nofile’, value: ‘65536’ }
– { type: ‘soft’, item: ‘nproc’, value: ‘65536’ }
– { type: ‘hard’, item: ‘nproc’, value: ‘65536’ }
– name: 配置hosts文件
lineinfile:
path: /etc/hosts
line: “{{ hostvars[item].ansible_host }} {{ item }}”
state: present
loop: “{{ groups[‘all’] }}”
when: hostvars[item].ansible_host is defined
2. Container Runtime角色：安装容器运行时
以containerd为例，展示如何安装和配置容器运行时。
tasks/main.yml
:
—
– name: 安装containerd
block:
– name: 下载containerd二进制文件
get_url:
url: “https://github.com/containerd/containerd/releases/download/v{{ containerd_version }}/containerd-{{ containerd_version }}-linux-{{ ansible_architecture }}.tar.gz”
dest: /tmp/containerd.tar.gz
– name: 解压并安装containerd
unarchive:
src: /tmp/containerd.tar.gz
dest: /usr/local
remote_src: yes
– name: 创建containerd配置目录
file:
path: /etc/containerd
state: directory
mode: ‘0755’
– name: 生成默认配置文件
command: containerd config default > /etc/containerd/config.toml
args:
creates: /etc/containerd/config.toml
– name: 配置containerd使用systemd cgroup
ini_file:
path: /etc/containerd/config.toml
section: plugins.”io.containerd.grpc.v1.cri”.containerd.runtimes.runc.options
option: SystemdCgroup
value: true
– name: 创建containerd服务文件
copy:
dest: /etc/systemd/system/containerd.service
content: |
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target
[Service]
ExecStart=/usr/local/bin/containerd
Restart=always
RestartSec=5
Delegate=yes
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
[Install]
WantedBy=multi-user.target
– name: 启动并启用containerd服务
systemd:
name: containerd
state: started
enabled: yes
daemon_reload: yes
– name: 安装crictl工具
block:
– name: 下载crictl
get_url:
url: “https://github.com/kubernetes-sigs/cri-tools/releases/download/v{{ crictl_version }}/crictl-v{{ crictl_version }}-linux-{{ ansible_architecture }}.tar.gz”
dest: /tmp/crictl.tar.gz
– name: 解压crictl
unarchive:
src: /tmp/crictl.tar.gz
dest: /usr/local/bin
remote_src: yes
– name: 配置crictl
copy:
dest: /etc/crictl.yaml
content: |
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
3. ETCD角色：部署高可用etcd集群
对于高可用Kubernetes集群，etcd的部署至关重要。
tasks/main.yml
:
—
– name: 下载etcd二进制文件
get_url:
url: “https://github.com/etcd-io/etcd/releases/download/v{{ etcd_version }}/etcd-v{{ etcd_version }}-linux-{{ ansible_architecture }}.tar.gz”
dest: /tmp/etcd.tar.gz
– name: 解压etcd
unarchive:
src: /tmp/etcd.tar.gz
dest: /usr/local/bin
remote_src: yes
extra_opts: [–strip=1]
– name: 创建etcd数据目录
file:
path: “{{ item }}”
state: directory
mode: ‘0700’
owner: root
group: root
loop:
– /var/lib/etcd
– /etc/kubernetes/pki/etcd
– name: 生成etcd证书
block:
– name: 创建etcd证书配置
template:
src: etcd-csr.json.j2
dest: /tmp/etcd-csr.json
– name: 生成etcd证书
command: |
cfssl gencert -initca /tmp/etcd-csr.json | cfssljson -bare /etc/kubernetes/pki/etcd/etcd
args:
creates: /etc/kubernetes/pki/etcd/etcd.pem
– name: 创建etcd服务文件
template:
src: etcd.service.j2
dest: /etc/systemd/system/etcd.service
– name: 启动etcd服务
systemd:
name: etcd
state: started
enabled: yes
daemon_reload: yes
– name: 验证etcd集群状态
command: etcdctl endpoint health –endpoints=https://{{ ansible_host }}:2379 –cacert=/etc/kubernetes/pki/etcd/ca.pem –cert=/etc/kubernetes/pki/etcd/etcd.pem –key=/etc/kubernetes/pki/etcd/etcd-key.pem
register: etcd_health
failed_when: “‘is healthy’ not in etcd_health.stdout”
changed_when: false
templates/etcd.service.j2
:
[Unit]
Description=etcd key-value store
Documentation=https://github.com/etcd-io/etcd
After=network.target
[Service]
User=root
Type=notify
ExecStart=/usr/local/bin/etcd \
–name={{ etcd_name }} \
–data-dir=/var/lib/etcd \
–listen-client-urls=https://{{ ansible_host }}:2379 \
–advertise-client-urls=https://{{ ansible_host }}:2379 \
–listen-peer-urls=https://{{ ansible_host }}:2380 \
–initial-advertise-peer-urls=https://{{ ansible_host }}:2380 \
–initial-cluster={{ etcd_initial_cluster }} \
–initial-cluster-token=etcd-cluster-1 \
–initial-cluster-state=new \
–client-cert-auth=true \
–trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem \
–cert-file=/etc/kubernetes/pki/etcd/etcd.pem \
–key-file=/etc/kubernetes/pki/etcd/etcd-key.pem \
–peer-client-cert-auth=true \
–peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem \
–peer-cert-file=/etc/kubernetes/pki/etcd/etcd.pem \
–peer-key-file=/etc/kubernetes/pki/etcd/etcd-key.pem
Restart=always
RestartSec=10
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
4. Master角色：初始化控制平面
tasks/main.yml
:
—
– name: 安装kubeadm、kubelet、kubectl
package:
name:
– “kubeadm-{{ k8s_version }}-0”
– “kubelet-{{ k8s_version }}-0”
– “kubectl-{{ k8s_version }}-0”
state: present
– name: 配置kubelet服务
template:
src: kubelet.service.j2
dest: /etc/systemd/system/kubelet.service
– name: 创建kubeadm配置文件
template:
src: kubeadm-config.yaml.j2
dest: /tmp/kubeadm-config.yaml
– name: 初始化Kubernetes控制平面
command: kubeadm init –config=/tmp/kubeadm-config.yaml –upload-certs
register: init_result
failed_when: “‘already initialized’ not in init_result.stdout and ‘Your Kubernetes control-plane has initialized successfully’ not in init_result.stdout”
changed_when: “‘already initialized’ not in init_result.stdout”
– name: 配置kubectl
copy:
src: /etc/kubernetes/admin.conf
dest: “{{ ansible_user_dir }}/.kube/config”
owner: “{{ ansible_user }}”
group: “{{ ansible_user }}”
remote_src: yes
– name: 安装网络插件
command: kubectl apply -f “https://raw.githubusercontent.com/projectcalico/calico/v{{ calico_version }}/manifests/calico.yaml”
register: network_install
changed_when: “‘created’ in network_install.stdout”
– name: 等待控制平面就绪
command: kubectl get pods -n kube-system -l component=kube-apiserver -o jsonpath='{.items[0].status.phase}’
register: api_server_status
until: api_server_status.stdout == “Running”
retries: 30
delay: 10
templates/kubeadm-config.yaml.j2
:
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
criSocket: “unix:///run/containerd/containerd.sock”
kubeletExtraArgs:
cgroup-driver: systemd
node-ip: “{{ ansible_host }}”
{% if loadbalancer.enabled %}
controlPlaneEndpoint: “{{ loadbalancer.vip }}:{{ loadbalancer.port }}”
{% else %}
controlPlaneEndpoint: “{{ ansible_host }}:6443”
{% endif %}
—
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: “v{{ k8s_version }}”
clusterName: “kubernetes”
networking:
podSubnet: “{{ pod_cidr }}”
serviceSubnet: “{{ service_cidr }}”
dnsDomain: “cluster.local”
etcd:
external:
endpoints:
{% for host in groups[‘etcd’] %}
– https://{{ hostvars[host].ansible_host }}:2379{% endfor %}
caFile: /etc/kubernetes/pki/etcd/ca.pem
certFile: /etc/kubernetes/pki/etcd/etcd.pem
keyFile: /etc/kubernetes/pki/etcd/etcd-key.pem
apiServer:
extraArgs:
advertise-address: “{{ ansible_host }}”
audit-log-path: “/var/log/kubernetes/audit.log”
audit-policy-file: “/etc/kubernetes/audit-policy.yaml”
extraVolumes:
– name: audit
hostPath: /var/log/kubernetes
mountPath: /var/log/kubernetes
pathType: DirectoryOrCreate
controllerManager:
extraArgs:
cluster-cidr: “{{ pod_cidr }}”
service-cluster-ip-range: “{{ service_cidr }}”
scheduler:
extraArgs:
bind-address: “0.0.0.0”
5. Worker角色：加入工作节点
tasks/main.yml
:
—
– name: 安装kubeadm和kubelet
package:
name:
– “kubeadm-{{ k8s_version }}-0”
– “kubelet-{{ k8s_version }}-0”
state: present
– name: 配置kubelet服务
template:
src: kubelet.service.j2
dest: /etc/systemd/system/kubelet.service
– name: 获取join命令
command: kubeadm token create –print-join-command
register: join_command
delegate_to: “{{ groups[‘masters’][0] }}”
run_once: true
– name: 加入集群
command: “{{ join_command.stdout }}”
register: join_result
failed_when: “‘already joined’ not in join_result.stdout and ‘has joined the cluster’ not in join_result.stdout”
changed_when: “‘already joined’ not in join_result.stdout”
– name: 等待节点就绪
command: kubectl get nodes {{ inventory_hostname }} -o jsonpath='{.status.conditions[?(@.type==”Ready”)].status}’
register: node_status
until: node_status.stdout == “True”
retries: 30
delay: 10
delegate_to: “{{ groups[‘masters’][0] }}”
6. Network角色：配置CNI网络
tasks/main.yml
:
—
– name: 安装Calico CNI
block:
– name: 下载Calico manifest
get_url:
url: “https://raw.githubusercontent.com/projectcalico/calico/v{{ calico_version }}/manifests/calico.yaml”
dest: /tmp/calico.yaml
– name: 修改Calico CIDR
replace:
path: /tmp/calico.yaml
regexp: ‘192.168.0.0/16’
replace: “{{ pod_cidr }}”
– name: 应用Calico manifest
command: kubectl apply -f /tmp/calico.yaml
register: calico_apply
changed_when: “‘created’ in calico_apply.stdout”
– name: 验证网络插件状态
command: kubectl get pods -n kube-system -l k8s-app=calico-node -o jsonpath='{.items[*].status.phase}’
register: calico_pods
until: “‘Running’ in calico_pods.stdout”
retries: 60
delay: 5
– name: 配置网络策略
k8s:
state: present
definition:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: default
spec:
podSelector: {}
policyTypes:
– Ingress
– Egress
解决初始化常见痛点
痛点1：证书过期管理
问题
：Kubernetes证书默认1年过期，过期后集群不可用。
解决方案
：在Playbook中添加证书续期任务。
– name: 检查证书过期时间
command: kubeadm certs check-expiration
register: cert_expiration
delegate_to: “{{ groups[‘masters’][0] }}”
run_once: true
– name: 自动续期证书
command: kubeadm certs renew all
when: “‘will expire within 365 days’ in cert_expiration.stdout”
delegate_to: “{{ groups[‘masters’][0] }}”
run_once: true
– name: 重启控制平面组件
systemd:
name: “{{ item }}”
state: restarted
loop:
– kube-apiserver
– kube-controller-manager
– kube-scheduler
when: “‘will expire within 365 days’ in cert_expiration.stdout”
delegate_to: “{{ groups[‘masters’][0] }}”
run_once: true
痛点2：内核参数调优
问题
：默认内核参数不适合大规模容器运行环境。
解决方案
：创建专门的内核调优角色。
– name: 优化内核参数
sysctl:
name: “{{ item.key }}”
value: “{{ item.value }}”
state: present
reload: yes
loop:
– { key: ‘net.core.somaxconn’, value: ‘32768’ }
– { key: ‘net.ipv4.tcp_max_syn_backlog’, value: ‘8192’ }
– { key: ‘net.core.netdev_max_backlog’, value: ‘16384’ }
– { key: ‘vm.swappiness’, value: ‘0’ }
– { key: ‘vm.overcommit_memory’, value: ‘1’ }
– { key: ‘fs.inotify.max_user_watches’, value: ‘524288’ }
– { key: ‘fs.file-max’, value: ‘2097152’ }
– name: 配置limits.conf
pam_limits:
domain: ‘*’
limit_type: “{{ item.type }}”
limit_item: “{{ item.item }}”
value: “{{ item.value }}”
loop:
– { type: ‘soft’, item: ‘nofile’, value: ‘65536’ }
– { type: ‘hard’, item: ‘nofile’, value: ‘65536’ }
– { type: ‘soft’, item: ‘nproc’, value: ‘65536’ }
– { type: ‘hard’, item: ‘nproc’, value: ‘65536’ }
– { type: ‘soft’, item: ‘memlock’, value: ‘unlimited’ }
– { type: ‘hard’, item: ‘memlock’, value: ‘unlimited’ }
痛点3：防火墙配置
问题
：防火墙规则复杂，容易遗漏端口。
解决方案
：使用Ansible统一配置防火墙。
– name: 开放Kubernetes必要端口
iptables:
chain: INPUT
protocol: “{{ item.protocol }}”
destination_port: “{{ item.port }}”
jump: ACCEPT
state: present
loop:
– { protocol: ‘tcp’, port: ‘6443’ } # API Server
– { protocol: ‘tcp’, port: ‘2379-2380’ } # etcd
– { protocol: ‘tcp’, port: ‘10250’ } # Kubelet
– { protocol: ‘tcp’, port: ‘10259’ } # Scheduler
– { protocol: ‘tcp’, port: ‘10257’ } # Controller Manager
– { protocol: ‘tcp’, port: ‘30000-32767’ } # NodePort Services
– { protocol: ‘tcp’, port: ‘8472’ } # Flannel VXLAN (if using Flannel)
– { protocol: ‘udp’, port: ‘8472’ }
– name: 保存iptables规则
command: iptables-save > /etc/iptables/rules.v4
when: ansible_os_family == “Debian”
– name: 安装iptables-persistent
package:
name: iptables-persistent
state: present
when: ansible_os_family == “Debian”
痛点4：高可用部署复杂性
问题
：多主节点配置复杂，需要处理负载均衡和etcd集群。
解决方案
：使用HAProxy + Keepalived实现API Server负载均衡。
roles/loadbalancer/tasks/main.yml
:
—
– name: 安装HAProxy
package:
name: haproxy
state: present
– name: 配置HAProxy
template:
src: haproxy.cfg.j2
dest: /etc/haproxy/haproxy.cfg
– name: 安装Keepalived
package:
name: keepalived
state: present
– name: 配置Keepalived
template:
src: keepalived.conf.j2
dest: /etc/keepalived/keepalived.conf
– name: 启动HAProxy
systemd:
name: haproxy
state: started
enabled: yes
– name: 启动Keepalived
systemd:
name: keepalived
state: started
enabled: yes
– name: 验证负载均衡器状态
command: systemctl is-active haproxy
register: haproxy_status
failed_when: haproxy_status.stdout != “active”
templates/haproxy.cfg.j2
:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
log global
mode tcp
option tcplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
frontend kubernetes-frontend
bind *:6443
default_backend kubernetes-backend
backend kubernetes-backend
balance roundrobin
option tcp-check
server master1 {{ hostvars[groups[‘masters’][0]].ansible_host }}:6443 check inter 2000 rise 2 fall 2
server master2 {{ hostvars[groups[‘masters’][1]].ansible_host }}:6443 check inter 2000 rise 2 fall 2
server master3 {{ hostvars[groups[‘masters’][2]].ansible_host }}:6443 check inter 2000 rise 2 fall 2
痛点5：网络插件配置错误
问题
：CNI插件配置错误导致Pod无法通信。
解决方案
：添加网络连通性测试任务。
– name: 创建测试Pod
k8s:
state: present
definition:
apiVersion: v1
kind: Pod
metadata:
name: network-test
namespace: default
spec:
containers:
– name: test
image: busybox
command: [‘sh’, ‘-c’, ‘sleep 3600′]
– name: 等待测试Pod就绪
command: kubectl get pod network-test -o jsonpath='{.status.phase}’
register: test_pod_status
until: test_pod_status.stdout == “Running”
retries: 30
delay: 5
– name: 测试Pod间网络连通性
command: kubectl exec network-test — ping -c 3 network-test
register: ping_result
failed_when: “‘0% packet loss’ not in ping_result.stdout”
– name: 测试DNS解析
command: kubectl exec network-test — nslookup kubernetes.default
register: dns_result
failed_when: “‘10.96.0.10’ not in dns_result.stdout”
– name: 清理测试资源
k8s:
state: absent
definition:
apiVersion: v1
kind: Pod
metadata:
name: network-test
namespace: default
完整的Playbook示例
主部署Playbook
playbooks/deploy.yml
:
—
– name: 部署Kubernetes基础环境
hosts: all
become: yes
roles:
– common
– container-runtime
– name: 部署etcd集群
hosts: etcd
become: yes
roles:
– etcd
– name: 初始化控制平面
hosts: masters
become: yes
roles:
– master
– name: 加入工作节点
hosts: workers
become: yes
roles:
– worker
– name: 配置网络插件
hosts: masters[0]
become: yes
roles:
– network
– name: 验证集群状态
hosts: masters[0]
become: yes
tasks:
– name: 检查所有节点状态
command: kubectl get nodes -o wide
register: nodes_status
changed_when: false
– name: 检查核心Pod状态
command: kubectl get pods -n kube-system
register: pods_status
changed_when: false
– name: 显示集群信息
debug:
msg: |
Kubernetes集群部署完成！
节点状态：
{{ nodes_status.stdout }}
核心Pod：
{{ pods_status.stdout }}
高可用部署Playbook
playbooks/deploy-ha.yml
:
—
– name: 部署负载均衡器
hosts: lb
become: yes
roles:
– loadbalancer
– name: 部署Kubernetes基础环境
hosts: all
become: yes
roles:
– common
– container-runtime
– name: 部署etcd集群
hosts: etcd
become: yes
roles:
– etcd
– name: 初始化第一个控制平面节点
hosts: masters[0]
become: yes
tasks:
– name: 初始化集群
command: kubeadm init –config=/tmp/kubeadm-config.yaml –upload-certs –control-plane-endpoint “{{ loadbalancer.vip }}:{{ loadbalancer.port }}”
register: init_result
– name: 获取证书key
command: kubeadm certs certificate-key
register: cert_key
when: init_result.changed
– name: 加入其他控制平面节点
hosts: masters[1:]
become: yes
tasks:
– name: 获取join命令
command: kubeadm token create –print-join-command
register: join_command
delegate_to: “{{ groups[‘masters’][0] }}”
run_once: true
– name: 加入控制平面
command: “{{ join_command.stdout }} –control-plane –certificate-key {{ hostvars[groups[‘masters’][0]].cert_key.stdout }}”
when: join_command.changed
– name: 加入工作节点
hosts: workers
become: yes
roles:
– worker
– name: 配置网络插件
hosts: masters[0]
become: yes
roles:
– network
执行和验证
执行部署
# 1. 配置SSH免密登录
ssh-keygen -t rsa -b 4096
ssh-copy-id user@master1
ssh-copy-id user@worker1
# 2. 测试Ansible连接
ansible all -i inventory/hosts.ini -m ping
# 3. 执行部署
ansible-playbook -i inventory/hosts.ini playbooks/deploy.yml
# 4. 高可用部署
ansible-playbook -i inventory/hosts.ini playbooks/deploy-ha.yml
验证集群状态
# 检查节点状态
kubectl get nodes -o wide
# 检查核心Pod
kubectl get pods -n kube-system
# 检查网络插件
kubectl get pods -n kube-system -l k8s-app=calico-node
# 测试集群功能
kubectl run nginx –image=nginx –port=80
kubectl expose pod nginx –type=NodePort –port=80
kubectl get svc nginx
curl http://
故障排查指南
常见问题及解决方案
1. Pod无法启动，CrashLoopBackOff
# 查看Pod详细信息
kubectl describe pod
# 查看Pod日志
kubectl logs
# 检查容器运行时
crictl logs
# 检查kubelet日志
journalctl -u kubelet -f
2. 节点NotReady状态
# 检查kubelet状态
systemctl status kubelet
# 查看kubelet日志
journalctl -u kubelet -f –since “1 hour ago”
# 检查网络插件
kubectl get pods -n kube-system
# 检查节点资源
kubectl describe node
3. DNS解析失败
# 检查CoreDNS Pod
kubectl get pods -n kube-system -l k8s-app=kube-dns
# 测试DNS解析
kubectl run -it –rm –image=busybox dns-test — nslookup kubernetes.default
# 检查CoreDNS配置
kubectl get configmap coredns -n kube-system -o yaml
4. 网络策略问题
# 检查Calico Pod状态
kubectl get pods -n kube-system -l k8s-app=calico-node
# 查看Calico日志
kubectl logs -n kube-system -l k8s-app=calico-node
# 检查Calico配置
kubectl get ippools -o yaml
# 测试网络连通性
kubectl run -it –rm –image=alpine network-test — ping 8.8.8.8
最佳实践
1. 版本控制
将Playbook和配置文件纳入Git版本控制：
git init
git add .
git commit -m “Initial commit: Kubernetes deployment automation”
2. 使用Vault加密敏感数据
# 创建vault密码文件
echo “my-vault-password” > .vault-password
chmod 600 .vault-password
# 加密敏感变量
ansible-vault encrypt_string ‘my-secret-token’ –name ‘vault_token’
# 在Playbook中使用
vars_files:
– secrets.yml
3. 集成CI/CD
在GitLab CI中集成部署：
# .gitlab-ci.yml
stages:
– deploy
deploy_k8s:
stage: deploy
script:
– ansible-playbook -i inventory/production playbooks/deploy.yml –vault-password-file .vault-password
only:
– main
environment: production
4. 监控和告警
部署后立即安装监控组件：
– name: 安装Prometheus和Grafana
command: kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup.yaml
register: prometheus_install
– name: 等待监控就绪
command: kubectl get pods -n monitoring
register: monitoring_pods
until: “‘Running’ in monitoring_pods.stdout”
retries: 60
delay: 5
总结
通过Ansible Playbook自动化部署Kubernetes集群，我们可以：
大幅减少部署时间
：从数小时缩短到几分钟
提高部署一致性
：避免手动操作的差异性
简化维护工作
：通过Playbook管理集群生命周期
降低故障风险
：内置的错误处理和验证机制
本文提供的Playbook示例涵盖了从基础环境配置到高可用部署的完整流程，并针对常见痛点提供了解决方案。在实际生产环境中，建议根据具体需求调整配置，并结合CI/CD流程实现完全自动化的集群管理。
记住，自动化部署只是第一步，持续的监控、维护和优化才是确保Kubernetes集群稳定运行的关键。