引言:为什么需要自定义Helm Charts
在Kubernetes生态系统中,Helm作为”包管理器”的地位已经不可动摇。然而,官方仓库中的通用Chart往往无法满足生产环境的复杂需求。自定义Helm Charts能够解决以下痛点:
业务适配性
:官方Chart通常只包含基础配置,缺乏业务特定的监控、告警、日志等集成
环境差异化
:开发、测试、生产环境需要不同的资源配置和副本数
安全合规
:生产环境需要满足特定的安全策略、网络策略和资源限制
成本优化
:通过合理的资源配比和自动扩缩容策略优化云资源使用
一、Helm Chart基础结构解析
1.1 标准目录结构
一个生产级的Helm Chart应该包含以下结构:
myapp/
├── Chart.yaml # Chart元数据
├── values.yaml # 默认配置值
├── templates/ # Kubernetes模板文件
│ ├── _helpers.tpl # 模板辅助函数
│ ├── deployment.yaml
│ ├── service.yaml
│ ├── configmap.yaml
│ ├── secret.yaml
│ ├── ingress.yaml
│ ├── pdb.yaml # PodDisruptionBudget
│ └── hpa.yaml # HorizontalPodAutoscaler
├── charts/ # 依赖的子Chart
├── crds/ # 自定义资源定义
├── templates/NOTES.txt # 部署后提示信息
└── .helmignore # 忽略文件
1.2 核心文件详解
Chart.yaml
– Chart的身份证:
apiVersion: v2
name: myapp
description: A High Availability Web Application
type: application
version: 1.2.0
appVersion: “1.16.0”
dependencies:
– name: redis
version: “16.0.0”
repository: “https://charts.bitnami.com/bitnami”
condition: redis.enabled
keywords:
– web
– production
maintainers:
– name: your-team
email: team@example.com
values.yaml
– 可配置的参数(生产级示例):
# 全局配置
global:
imageRegistry: “”
imagePullSecrets: []
# 应用配置
replicaCount: 3
image:
repository: myregistry.com/myapp
tag: “1.16.0”
pullPolicy: IfNotPresent
# 资源配置(生产级必须)
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 500m
memory: 512Mi
# 自动扩缩容
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 10
targetCPUUtilizationPercentage: 70
targetMemoryUtilizationPercentage: 80
# 健康检查
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 5
periodSeconds: 5
# 网络配置
service:
type: ClusterIP
port: 80
targetPort: 8080
ingress:
enabled: true
className: “nginx”
annotations:
cert-manager.io/cluster-issuer: “letsencrypt-prod”
hosts:
– host: myapp.example.com
paths:
– path: /
pathType: Prefix
tls:
– secretName: myapp-tls
hosts:
– myapp.example.com
# 安全配置
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
podSecurityContext:
seccompProfile:
type: RuntimeDefault
# 网络策略
networkPolicy:
enabled: true
ingress:
– from:
– namespaceSelector:
matchLabels:
name: ingress-nginx
ports:
– protocol: TCP
port: 8080
# 持久化存储
persistence:
enabled: true
storageClass: “fast-ssd”
size: 10Gi
accessMode: ReadWriteOnce
# 监控配置
monitoring:
enabled: true
serviceMonitor:
enabled: true
interval: 30s
prometheusRule:
enabled: true
rules:
– alert: HighErrorRate
expr: rate(http_requests_total{status=~”5..”}[5m]) > 0.1
for: 5m
labels:
severity: critical
annotations:
summary: “High error rate detected”
# 配置管理
config:
database:
host: postgres.example.com
port: 5432
name: myapp
redis:
host: redis-master
port: 6379
# 环境特定配置
environment: production
二、模板开发最佳实践
2.1 模板辅助函数 (_helpers.tpl)
创建可复用的模板函数:
{{/* 生成标准标签 */}}
{{- define “myapp.labels” -}}
app.kubernetes.io/name: {{ include “myapp.name” . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end -}}
{{/* 生成选择器标签 */}}
{{- define “myapp.selectorLabels” -}}
app.kubernetes.io/name: {{ include “myapp.name” . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end -}}
{{/* 生成完整名称 */}}
{{- define “myapp.fullname” -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix “-” -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix “-” -}}
{{- else -}}
{{- printf “%s-%s” .Release.Name $name | trunc 63 | trimSuffix “-” -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/* 镜像名称生成器 */}}
{{- define “myapp.image” -}}
{{- $registry := .Values.global.imageRegistry -}}
{{- $repository := .Values.image.repository -}}
{{- $tag := .Values.image.tag -}}
{{- if $registry -}}
{{- printf “%s/%s:%s” $registry $repository $tag -}}
{{- else -}}
{{- printf “%s:%s” $repository $tag -}}
{{- end -}}
{{- end -}}
{{/* 资源限制模板 */}}
{{- define “myapp.resources” -}}
{{- if .Values.resources -}}
resources:
{{- toYaml .Values.resources | nindent 2 }}
{{- else -}}
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 100m
memory: 128Mi
{{- end -}}
{{- end -}}
{{/* 安全上下文模板 */}}
{{- define “myapp.securityContext” -}}
{{- if .Values.securityContext -}}
securityContext:
{{- toYaml .Values.securityContext | nindent 2 }}
{{- end -}}
{{- if .Values.podSecurityContext -}}
podSecurityContext:
{{- toYaml .Values.podSecurityContext | nindent 2 }}
{{- end -}}
{{- end -}}
2.2 Deployment模板 (deployment.yaml)
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include “myapp.fullname” . }}
labels:
{{- include “myapp.labels” . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include “myapp.selectorLabels” . | nindent 6 }}
template:
metadata:
labels:
{{- include “myapp.selectorLabels” . | nindent 8 }}
annotations:
{{- if .Values.monitoring.enabled }}
prometheus.io/scrape: “true”
prometheus.io/port: “8080”
prometheus.io/path: “/metrics”
{{- end }}
checksum/config: {{ include (print $.Template.BasePath “/configmap.yaml”) . | sha256sum }}
checksum/secret: {{ include (print $.Template.BasePath “/secret.yaml”) . | sha256sum }}
spec:
{{- include “myapp.securityContext” . | nindent 6 }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include “myapp.fullname” . }}
containers:
– name: {{ .Chart.Name }}
image: {{ include “myapp.image” . }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
– name: http
containerPort: 8080
protocol: TCP
env:
– name: APP_ENV
value: {{ .Values.environment | quote }}
– name: DATABASE_URL
valueFrom:
secretKeyRef:
name: {{ include “myapp.fullname” . }}-db
key: connection-string
– name: REDIS_URL
value: {{ printf “redis://%s:%s” .Values.config.redis.host (toString .Values.config.redis.port) }}
envFrom:
– configMapRef:
name: {{ include “myapp.fullname” . }}-config
{{- if .Values.monitoring.enabled }}
livenessProbe:
{{- toYaml .Values.livenessProbe | nindent 12 }}
readinessProbe:
{{- toYaml .Values.readinessProbe | nindent 12 }}
{{- end }}
{{- include “myapp.resources” . | nindent 10 }}
volumeMounts:
{{- if .Values.persistence.enabled }}
– name: data
mountPath: /app/data
{{- end }}
{{- if .Values.config.extraConfig }}
– name: extra-config
mountPath: /app/config
readOnly: true
{{- end }}
{{- if .Values.persistence.enabled }}
volumes:
– name: data
persistentVolumeClaim:
claimName: {{ include “myapp.fullname” . }}-data
{{- if .Values.config.extraConfig }}
– name: extra-config
configMap:
name: {{ include “myapp.fullname” . }}-extra
items:
– key: config.yaml
path: config.yaml
{{- end }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
2.3 Service模板 (service.yaml)
apiVersion: v1
kind: Service
metadata:
name: {{ include “myapp.fullname” . }}
labels:
{{- include “myapp.labels” . | nindent 4 }}
{{- if .Values.service.annotations }}
annotations:
{{- toYaml .Values.service.annotations | nindent 4 }}
{{- end }}
spec:
type: {{ .Values.service.type }}
{{- if .Values.service.clusterIP }}
clusterIP: {{ .Values.service.clusterIP }}
{{- end }}
{{- if .Values.service.loadBalancerIP }}
loadBalancerIP: {{ .Values.service.loadBalancerIP }}
{{- end }}
{{- if .Values.service.loadBalancerSourceRanges }}
loadBalancerSourceRanges:
{{- toYaml .Values.service.loadBalancerSourceRanges | nindent 4 }}
{{- end }}
ports:
– port: {{ .Values.service.port }}
targetPort: {{ .Values.service.targetPort }}
protocol: TCP
name: http
selector:
{{- include “myapp.selectorLabels” . | nindent 4 }}
2.4 Ingress模板 (ingress.yaml)
{{- if .Values.ingress.enabled -}}
{{- $fullName := include “myapp.fullname” . -}}
{{- $svcPort := .Values.service.port -}}
{{- if and .Values.ingress.className (not (semverCompare “>=1.18-0” .Capabilities.KubeVersion.GitVersion)) }}
{{- if not (hasKey .Values.ingress.annotations “kubernetes.io/ingress.class”) }}
{{- $_ := set .Values.ingress.annotations “kubernetes.io/ingress.class” .Values.ingress.className }}
{{- end }}
{{- end }}
{{- if semverCompare “>=1.19-0” .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1
{{- else if semverCompare “>=1.14-0” .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
metadata:
name: {{ $fullName }}
labels:
{{- include “myapp.labels” . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- if .Values.ingress.className }}
ingressClassName: {{ .Values.ingress.className }}
{{- end }}
{{- if .Values.ingress.tls }}
tls:
{{- range .Values.ingress.tls }}
– hosts:
{{- range .hosts }}
– {{ . | quote }}
{{- end }}
secretName: {{ .secretName }}
{{- end }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
– host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
– path: {{ .path }}
{{- if and .pathType (semverCompare “>=1.18-0” $.Capabilities.KubeVersion.GitVersion) }}
pathType: {{ .pathType }}
{{- end }}
backend:
{{- if semverCompare “>=1.19-0” $.Capabilities.KubeVersion.GitVersion }}
service:
name: {{ $fullName }}
port:
number: {{ $svcPort }}
{{- else }}
serviceName: {{ $fullName }}
servicePort: {{ $svcPort }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
2.5 ConfigMap和Secret模板
ConfigMap (configmap.yaml)
:
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include “myapp.fullname” . }}-config
labels:
{{- include “myapp.labels” . | nindent 4 }}
data:
app-config.yaml: |
server:
port: 8080
database:
host: {{ .Values.config.database.host }}
port: {{ .Values.config.database.port }}
name: {{ .Values.config.database.name }}
redis:
host: {{ .Values.config.redis.host }}
port: {{ .Values.config.redis.port }}
logging:
level: {{ .Values.config.logging.level | default “info” }}
format: json
{{- if .Values.config.extra }}
extra:
{{- toYaml .Values.config.extra | nindent 6 }}
{{- end }}
Secret (secret.yaml)
:
{{- $dbPassword := .Values.config.database.password | default (randAlphaNum 16) -}}
apiVersion: v1
kind: Secret
metadata:
name: {{ include “myapp.fullname” . }}-db
labels:
{{- include “myapp.labels” . | nindent 4 }}
type: Opaque
data:
connection-string: {{ printf “postgres://%s:%s@%s:%d/%s” .Values.config.database.user $dbPassword .Values.config.database.host .Values.config.database.port .Values.config.database.name | b64enc | quote }}
password: {{ $dbPassword | b64enc | quote }}
2.6 高级资源模板
PodDisruptionBudget (pdb.yaml)
:
{{- if .Values.pdb.enabled -}}
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: {{ include “myapp.fullname” . }}
labels:
{{- include “myapp.labels” . | nindent 4 }}
spec:
{{- if .Values.pdb.minAvailable }}
minAvailable: {{ .Values.pdb.minAvailable }}
{{- else if .Values.pdb.maxUnavailable }}
maxUnavailable: {{ .Values.pdb.maxUnavailable }}
{{- end }}
selector:
matchLabels:
{{- include “myapp.selectorLabels” . | nindent 6 }}
{{- end }}
HorizontalPodAutoscaler (hpa.yaml)
:
{{- if .Values.autoscaling.enabled -}}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ include “myapp.fullname” . }}
labels:
{{- include “myapp.labels” . | nindent 4 }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include “myapp.fullname” . }}
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
{{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
– type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}
{{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
– type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
{{- end }}
{{- range .Values.autoscaling.customMetrics }}
– type: Pods
pods:
metric:
name: {{ .name }}
target:
type: AverageValue
averageValue: {{ .value }}
{{- end }}
behavior:
scaleDown:
stabilizationWindowSeconds: 300
policies:
– type: Percent
value: 50
periodSeconds: 60
scaleUp:
stabilizationWindowSeconds: 0
policies:
– type: Percent
value: 100
periodSeconds: 15
– type: Pods
value: 4
periodSeconds: 15
selectPolicy: Max
{{- end }}
三、生产级配置管理策略
3.1 环境分层配置
创建不同环境的values文件:
values-dev.yaml
(开发环境):
replicaCount: 1
resources:
limits:
cpu: 200m
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi
autoscaling:
enabled: false
ingress:
enabled: false
monitoring:
enabled: false
config:
logging:
level: debug
values-prod.yaml
(生产环境):
replicaCount: 3
resources:
limits:
cpu: 2000m
memory: 2Gi
requests:
cpu: 1000m
memory: 1Gi
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 15
targetCPUUtilizationPercentage: 60
ingress:
enabled: true
annotations:
cert-manager.io/cluster-issuer: “letsencrypt-prod”
nginx.ingress.kubernetes.io/rate-limit: “100”
nginx.ingress.kubernetes.io/rate-limit-window: “1m”
monitoring:
enabled: true
serviceMonitor:
enabled: true
prometheusRule:
enabled: true
pdb:
enabled: true
minAvailable: 2
nodeSelector:
workload-type: critical
tolerations:
– key: “critical”
operator: “Equal”
value: “true”
effect: “NoSchedule”
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
– labelSelector:
matchExpressions:
– key: app.kubernetes.io/name
operator: In
values:
– myapp
topologyKey: kubernetes.io/hostname
3.2 使用Helm Hooks进行生命周期管理
pre-install/pre-upgrade hook
(数据库迁移):
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include “myapp.fullname” . }}-migrate
labels:
{{- include “myapp.labels” . | nindent 4 }}
annotations:
“helm.sh/hook”: pre-install,pre-upgrade
“helm.sh/hook-weight”: “-5”
“helm.sh/hook-delete-policy”: before-hook-creation,hook-succeeded
spec:
template:
spec:
restartPolicy: Never
containers:
– name: migrate
image: “{{ .Values.image.repository }}:{{ .Values.image.tag }}”
command: [“npm”, “run”, “migrate”]
env:
– name: DATABASE_URL
valueFrom:
secretKeyRef:
name: {{ include “myapp.fullname” . }}-db
key: connection-string
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 200m
memory: 256Mi
{{- include “myapp.securityContext” . | nindent 6 }}
post-install hook
(初始化数据):
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include “myapp.fullname” . }}-init
labels:
{{- include “myapp.labels” . | nindent 4 }}
annotations:
“helm.sh/hook”: post-install
“helm.sh/hook-weight”: “1”
“helm.sh/hook-delete-policy”: before-hook-creation,hook-succeeded
spec:
template:
spec:
restartPolicy: Never
containers:
– name: init
image: “{{ .Values.image.repository }}:{{ .Values.image.tag }}”
command: [“npm”, “run”, “init-data”]
env:
– name: DATABASE_URL
valueFrom:
secretKeyRef:
name: {{ include “myapp.fullname” . }}-db
key: connection-string
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 200m
memory: 256Mi
{{- include “myapp.securityContext” . | nindent 6 }}
3.3 条件渲染和模板控制
使用if/else
:
# 在deployment.yaml中
{{- if .Values.extraContainers }}
containers:
– name: main
# 主容器配置
{{- toYaml .Values.extraContainers | nindent 2 }}
{{- else }}
containers:
– name: main
# 主容器配置
{{- end }}
使用range遍历
:
# 在configmap.yaml中
{{- range $key, $value := .Values.config.extra }}
{{ $key }}: {{ $value | quote }}
{{- end }}
使用with作用域
:
# 在secret.yaml中
{{- with .Values.secrets }}
{{- range $name, $value := . }}
– name: {{ $name }}
value: {{ $value | b64enc | quote }}
{{- end }}
{{- end }}
四、高级功能实现
4.1 子Chart依赖管理
Chart.yaml
:
dependencies:
– name: postgresql
version: “11.0.0”
repository: “https://charts.bitnami.com/bitnami”
condition: postgresql.enabled
tags:
– database
– name: redis
version: “16.0.0”
repository: “https://charts.bitnami.com/bitnami”
condition: redis.enabled
values.yaml
:
postgresql:
enabled: true
auth:
username: myapp
database: myapp
password: “”
primary:
persistence:
enabled: true
size: 8Gi
resources:
requests:
memory: 256Mi
cpu: 100m
redis:
enabled: true
auth:
enabled: true
password: “”
master:
persistence:
enabled: true
size: 2Gi
4.2 自定义验证
values.schema.json
:
{
“$schema”: “http://json-schema.org/draft-07/schema#”,
“type”: “object”,
“properties”: {
“replicaCount”: {
“type”: “integer”,
“minimum”: 1,
“maximum”: 50
},
“image”: {
“type”: “object”,
“properties”: {
“repository”: {
“type”: “string”,
“pattern”: “^[a-z0-9]+([._-][a-z0-9]+)*(/[a-z0-9]+([._-][a-z0-9]+)*)*$”
},
“tag”: {
“type”: “string”,
“pattern”: “^[a-zA-Z0-9._-]+$”
}
},
“required”: [“repository”, “tag”]
},
“resources”: {
“type”: “object”,
“properties”: {
“limits”: {
“type”: “object”,
“properties”: {
“cpu”: {
“type”: “string”,
“pattern”: “^[0-9]+m?$”
},
“memory”: {
“type”: “string”,
“pattern”: “^[0-9]+(Mi|Gi)$”
}
}
}
}
},
“ingress”: {
“type”: “object”,
“properties”: {
“enabled”: {“type”: “boolean”},
“hosts”: {
“type”: “array”,
“items”: {
“type”: “object”,
“properties”: {
“host”: {“type”: “string”, “format”: “hostname”},
“paths”: {“type”: “array”}
},
“required”: [“host”]
}
}
}
}
},
“required”: [“image”, “resources”]
}
4.3 模板库和共享组件
创建可复用的模板库:
templates/_ingress.tpl
:
{{- define “myapp.ingress” -}}
{{- if .Values.ingress.enabled -}}
{{- $fullName := include “myapp.fullname” . -}}
{{- $svcPort := .Values.service.port -}}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ $fullName }}
labels:
{{- include “myapp.labels” . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- if .Values.ingress.className }}
ingressClassName: {{ .Values.ingress.className }}
{{- end }}
{{- if .Values.ingress.tls }}
tls:
{{- range .Values.ingress.tls }}
– hosts:
{{- range .hosts }}
– {{ . | quote }}
{{- end }}
secretName: {{ .secretName }}
{{- end }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
– host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
– path: {{ .path }}
pathType: {{ .pathType | default “Prefix” }}
backend:
service:
name: {{ $fullName }}
port:
number: {{ $svcPort }}
{{- end }}
{{- end }}
{{- end }}
{{- end -}}
五、测试和验证
5.1 模板渲染测试
使用Helm模板测试:
# 渲染模板检查
helm template myapp ./myapp –values ./myapp/values-prod.yaml
# 检查特定资源
helm template myapp ./myapp –values ./myapp/values-prod.yaml | grep -A 20 “kind: Deployment”
# 使用helm unittest插件
helm plugin install https://github.com/helm-unittest/helm-unittest
# 创建测试文件 tests/deployment_test.yaml
tests/deployment_test.yaml
:
suite: test deployment
templates:
– deployment.yaml
tests:
– it: should create a deployment
asserts:
– hasKind:
of: Deployment
– hasDocument:
count: 1
– it: should have correct labels
asserts:
– equal:
path: metadata.labels[“app.kubernetes.io/name”]
value: myapp
– equal:
path: metadata.labels[“app.kubernetes.io/instance”]
value: myapp-test
– it: should use correct image
asserts:
– equal:
path: spec.template.spec.containers[0].image
value: myregistry.com/myapp:1.16.0
– it: should have resource limits
asserts:
– equal:
path: spec.template.spec.containers[0].resources.limits.cpu
value: 1000m
– it: should have liveness probe
asserts:
– equal:
path: spec.template.spec.containers[0].livenessProbe.httpGet.path
value: /health
5.2 集成测试
test-values.yaml
:
replicaCount: 1
image:
repository: nginx
tag: “1.21”
pullPolicy: IfNotPresent
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 50m
memory: 64Mi
ingress:
enabled: false
monitoring:
enabled: false
config:
database:
host: test-db
port: 5432
name: test
user: test
password: testpass
redis:
host: test-redis
port: 6379
运行测试:
# 渲染测试
helm template myapp ./myapp -f test-values.yaml –debug
# 安装测试
helm install myapp-test ./myapp -f test-values.yaml –dry-run
# 实际安装(在测试集群)
helm install myapp-test ./myapp -f test-values.yaml –namespace test
# 验证
kubectl get all -n test -l app.kubernetes.io/instance=myapp-test
# 检查日志
kubectl logs -n test deployment/myapp-test-myapp
# 端口转发测试
kubectl port-forward -n test svc/myapp-test-myapp 8080:80
curl http://localhost:8080/health
六、生产部署最佳实践
6.1 部署流程
# 1. 预检查
helm lint ./myapp
helm template myapp ./myapp -f values-prod.yaml –validate
# 2. 创建命名空间
kubectl create namespace production –dry-run=client -o yaml | kubectl apply -f –
# 3. 安装依赖(如果有)
helm dependency update ./myapp
# 4. 预览变更
helm diff upgrade myapp ./myapp -f values-prod.yaml –namespace production
# 5. 执行部署
helm upgrade –install myapp ./myapp \
-f values-prod.yaml \
–namespace production \
–create-namespace \
–wait \
–timeout 10m \
–atomic
# 6. 验证部署
helm status myapp -n production
# 7. 监控部署
kubectl get pods -n production -w
6.2 滚动更新策略
values-prod.yaml
:
# 滚动更新配置
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
# 蓝绿部署配置
blueGreen:
enabled: false
activeService: myapp-active
previewService: myapp-preview
autoPromotion:
enabled: true
seconds: 300
6.3 回滚策略
# 查看历史
helm history myapp -n production
# 回滚到上一个版本
helm rollback myapp -n production
# 回滚到指定版本
helm rollback myapp 2 -n production
# 预防性回滚
helm upgrade myapp ./myapp \
-f values-prod.yaml \
–namespace production \
–atomic \
–cleanup-on-fail \
–timeout 10m
七、监控和告警集成
7.1 ServiceMonitor配置
templates/servicemonitor.yaml
:
{{- if .Values.monitoring.enabled -}}
{{- if .Values.monitoring.serviceMonitor.enabled -}}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ include “myapp.fullname” . }}
labels:
{{- include “myapp.labels” . | nindent 4 }}
spec:
selector:
matchLabels:
{{- include “myapp.selectorLabels” . | nindent 6 }}
endpoints:
– port: http
interval: {{ .Values.monitoring.serviceMonitor.interval | default “30s” }}
path: /metrics
{{- if .Values.monitoring.serviceMonitor.honorLabels }}
honorLabels: true
{{- end }}
relabelings:
– sourceLabels: [__meta_kubernetes_pod_name]
targetLabel: pod
{{- end -}}
{{- end -}}
7.2 PrometheusRule配置
templates/prometheusrule.yaml
:
{{- if .Values.monitoring.enabled -}}
{{- if .Values.monitoring.prometheusRule.enabled -}}
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: {{ include “myapp.fullname” . }}
labels:
{{- include “myapp.labels” . | nindent 4 }}
role: alert-rules
spec:
groups:
– name: {{ include “myapp.fullname” . }}
interval: {{ .Values.monitoring.prometheusRule.interval | default “30s” }}
rules:
{{- range .Values.monitoring.prometheusRule.rules }}
– alert: {{ .alert }}
expr: {{ .expr }}
{{- if .for }}
for: {{ .for }}
{{- end }}
labels:
{{- toYaml .labels | nindent 12 }}
annotations:
{{- toYaml .annotations | nindent 12 }}
{{- end }}
{{- end -}}
{{- end -}}
7.3 Grafana Dashboard配置
templates/configmap-dashboard.yaml
:
{{- if .Values.monitoring.grafanaDashboard.enabled -}}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include “myapp.fullname” . }}-grafana-dashboard
labels:
{{- include “myapp.labels” . | nindent 4 }}
grafana_dashboard: “1”
annotations:
k8s-sidecar-target-directory: /tmp/dashboards/myapp
data:
dashboard.json: |
{
“dashboard”: {
“id”: null,
“title”: “MyApp – {{ .Release.Namespace }}”,
“tags”: [“myapp”, “{{ .Release.Namespace }}”],
“timezone”: “browser”,
“panels”: [
{
“id”: 1,
“title”: “Request Rate”,
“type”: “graph”,
“targets”: [
{
“expr”: “rate(http_requests_total{app=\”myapp\”}[5m])”,
“legendFormat”: “{{ ‘{{‘ }}{{ ‘{{‘ }} $labels.status {{ ‘}}’ }}{{ ‘}}’ }}”
}
]
},
{
“id”: 2,
“title”: “Error Rate”,
“type”: “stat”,
“targets”: [
{
“expr”: “rate(http_requests_total{app=\”myapp\”,status=~\”5..\”}[5m])”,
“legendFormat”: “Errors/sec”
}
]
}
]
}
}
{{- end -}}
八、安全最佳实践
8.1 Pod Security Standards
values.yaml
:
# 基础安全配置
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
– ALL
runAsNonRoot: true
# 网络策略
networkPolicy:
enabled: true
egress:
– to:
– namespaceSelector:
matchLabels:
name: monitoring
ports:
– protocol: TCP
port: 9090
– to:
– namespaceSelector:
matchLabels:
name: database
ports:
– protocol: TCP
port: 5432
– to:
– namespaceSelector:
matchLabels:
name: cache
ports:
– protocol: TCP
port: 6379
8.2 证书管理
templates/secret-tls.yaml
:
{{- if .Values.ingress.tls }}
{{- range .Values.ingress.tls }}
{{- if not .secretName }}
—
apiVersion: v1
kind: Secret
metadata:
name: {{ .secretName | default (printf “%s-tls” (include “myapp.fullname” $)) }}
labels:
{{- include “myapp.labels” $ | nindent 4 }}
type: kubernetes.io/tls
data:
tls.crt: {{ .crt | b64enc }}
tls.key: {{ .key | b64enc }}
{{- end }}
{{- end }}
{{- end }}
九、CI/CD集成
9.1 GitHub Actions工作流
.github/workflows/helm-chart-ci.yaml
:
name: Helm Chart CI
on:
push:
branches:
– main
paths:
– ‘charts/myapp/**’
pull_request:
branches:
– main
paths:
– ‘charts/myapp/**’
jobs:
lint:
runs-on: ubuntu-latest
steps:
– uses: actions/checkout@v3
– name: Set up Helm
uses: azure/setup-helm@v3
with:
version: ‘v3.10.0’
– name: Lint Chart
run: |
helm lint ./charts/myapp
helm template ./charts/myapp –validate
– name: Run Unit Tests
run: |
helm plugin install https://github.com/helm-unittest/helm-unittest || true
helm unittest ./charts/myapp
security-scan:
runs-on: ubuntu-latest
steps:
– uses: actions/checkout@v3
– name: Run kube-linter
uses: stackrox/kube-linter-action@v1
with:
directory: ./charts/myapp
– name: Run helm-schema
run: |
helm plugin install https://github.com/karuppiah7890/helm-schema || true
helm schema ./charts/myapp
release:
needs: [lint, security-scan]
runs-on: ubuntu-latest
if: github.ref == ‘refs/heads/main’
steps:
– uses: actions/checkout@v3
with:
fetch-depth: 0
– name: Configure Git
run: |
git config user.name “github-actions[bot]”
git config user.email “github-actions[bot]@users.noreply.github.com”
– name: Set up Helm
uses: azure/setup-helm@v3
with:
version: ‘v3.10.0’
– name: Add Dependencies
run: |
helm dependency update ./charts/myapp
– name: Package Chart
run: |
helm package ./charts/myapp
– name: Push to Chart Museum
env:
CHART_MUSEUM_URL: ${{ secrets.CHART_MUSEUM_URL }}
run: |
curl –data-binary “@myapp-*.tgz” $CHART_MUSEUM_URL/api/charts
– name: Update Chart Index
run: |
helm repo index . –url ${{ secrets.CHART_MUSEUM_URL }}
git add index.yaml
git commit -m “Update chart index [skip ci]”
git push
9.2 GitLab CI集成
.gitlab-ci.yml
:
stages:
– lint
– test
– release
variables:
HELM_VERSION: “3.10.0”
lint:
stage: lint
image: alpine/helm:$HELM_VERSION
script:
– helm lint ./charts/myapp
– helm template ./charts/myapp –validate
only:
changes:
– charts/myapp/**/*
test:
stage: test
image: alpine/helm:$HELM_VERSION
before_script:
– helm plugin install https://github.com/helm-unittest/helm-unittest
script:
– helm unittest ./charts/myapp
only:
changes:
– charts/myapp/**/*
release:
stage: release
image: alpine/helm:$HELM_VERSION
before_script:
– apk add –no-cache curl git
– git config user.name “GitLab CI”
– git config user.email “ci@gitlab.com”
script:
– helm dependency update ./charts/myapp
– helm package ./charts/myapp
– |
if [ -n “$CHART_MUSEUM_URL” ]; then
curl –data-binary “@myapp-*.tgz” $CHART_MUSEUM_URL/api/charts
helm repo index . –url $CHART_MUSEUM_URL
git add index.yaml
git commit -m “Release chart $(cat charts/myapp/Chart.yaml | grep version | cut -d: -f2 | tr -d ‘ ‘)”
git push
fi
only:
– main
when: manual
十、性能优化技巧
10.1 资源配置优化
动态资源计算
:
# 在_values.yaml中
resources:
limits:
cpu: “{{ .Values.replicaCount | mul 500 }}m”
memory: “{{ .Values.replicaCount | mul 256 }}Mi”
使用模板函数
:
{{- define “myapp.calculateResources” -}}
{{- $cpu := .Values.replicaCount | mul 500 -}}
{{- $memory := .Values.replicaCount | mul 256 -}}
limits:
cpu: {{ printf “%dm” $cpu }}
memory: {{ printf “%dMi” $memory }}
requests:
cpu: {{ printf “%dm” (div $cpu 2) }}
memory: {{ printf “%dMi” (div $memory 2) }}
{{- end -}}
10.2 镜像优化
多阶段构建
:
# Dockerfile
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci –only=production
COPY . .
RUN npm run build
FROM node:18-alpine AS runtime
WORKDIR /app
COPY –from=builder /app/node_modules ./node_modules
COPY –from=builder /app/dist ./dist
COPY –from=builder /app/package.json .
USER node
CMD [“node”, “dist/main.js”]
values.yaml
:
image:
repository: myapp
tag: “1.16.0”
pullPolicy: IfNotPresent
pullSecrets:
– name: my-registry-secret
10.3 模板渲染优化
使用tpl函数
:
# 在configmap.yaml中
data:
config.yaml: |
{{- .Values.configTemplate | nindent 4 }}
在values.yaml中
:
configTemplate: |
server:
port: 8080
database:
host: {{ .Values.config.database.host }}
十一、故障排查指南
11.1 常见问题诊断
# 1. 模板渲染失败
helm template myapp ./myapp –debug
# 2. 验证YAML语法
yamllint ./myapp/templates/
# 3. 检查资源是否存在
kubectl get all -n production -l app.kubernetes.io/instance=myapp
# 4. 查看Pod事件
kubectl describe pod -n production -l app.kubernetes.io/instance=myapp
# 5. 检查日志
kubectl logs -n production -l app.kubernetes.io/instance=myapp –tail=100
# 6. 检查ConfigMap/Secret
kubectl get configmap,secret -n production -l app.kubernetes.io/instance=myapp
# 7. 检查网络策略
kubectl get networkpolicy -n production -l app.kubernetes.io/instance=myapp
# 8. 检查PDB
kubectl get pdb -n production -l app.kubernetes.io/instance=myapp
# 9. 检查HPA
kubectl get hpa -n production -l app.kubernetes.io/instance=myapp
# 10. 检查事件
kubectl get events -n production –sort-by=’.lastTimestamp’
11.2 调试技巧
使用helm-secrets插件
:
helm plugin install https://github.com/jkroepke/helm-secrets
# 加密敏感数据
helm secrets encrypt values-prod.yaml
# 部署时解密
helm secrets upgrade myapp ./myapp -f values-prod.yaml
使用helm-diff插件
:
helm plugin install https://github.com/databus23/helm-diff
# 查看变更
helm diff upgrade myapp ./myapp -f values-prod.yaml –namespace production
# 三向合并
helm diff upgrade myapp ./myapp -f values-prod.yaml –namespace production –three-way-merge
十二、总结与最佳实践清单
12.1 生产级Chart检查清单
[ ]
结构完整性
[ ] Chart.yaml包含所有必要字段
[ ] values.yaml有详细注释
[ ] templates目录包含所有必需资源
[ ] .helmignore配置正确
[ ]
模板质量
[ ] 使用_helpers.tpl定义可复用函数
[ ] 所有资源都有标签和选择器
[ ] 使用条件渲染避免空资源
[ ] 支持所有必要的配置项
[ ]
安全性
[ ] 非root用户运行
[ ] 只读根文件系统
[ ] 网络策略配置
[ ] Pod安全上下文
[ ] Secret不硬编码
[ ]
可用性
[ ] 健康检查配置
[ ] PDB配置
[ ] HPA配置
[ ] 节点亲和性/反亲和性
[ ] 多副本部署
[ ]
可观察性
[ ] ServiceMonitor配置
[ ] PrometheusRule配置
[ ] Grafana Dashboard
[ ] 日志配置
[ ]
可维护性
[ ] 文档完善
[ ] 版本管理
[ ] 依赖管理
[ ] 测试覆盖
[ ] CI/CD集成
[ ]
性能
[ ] 资源限制配置
[ ] 镜像优化
[ ] 模板渲染效率
[ ] 自动扩缩容
12.2 持续改进
定期审查
:每月审查Chart配置,更新依赖版本
性能监控
:监控部署时间和资源使用情况
安全扫描
:集成安全扫描工具
文档更新
:保持README和values注释最新
社区反馈
:收集用户反馈,持续改进
通过遵循本指南,您将能够构建出高效、安全、可复用的生产级Helm Charts,为您的Kubernetes应用部署提供坚实的基础。