ECK在K8S中的部署与运维实践

1. 项目概述ECK在K8S生态中的定位Elastic Cloud on KubernetesECK是Elastic官方提供的Operator实现它彻底改变了传统Elasticsearch集群在Kubernetes中的部署和管理方式。与使用Helm chart或手动部署相比ECK通过Custom Resource DefinitionsCRD将Elasticsearch、Kibana、APM等组件抽象为Kubernetes原生资源实现了声明式的集群管理。在实际生产环境中我们团队经历过从StatefulSet直接部署到ECK的完整迁移过程。最直观的体验是原先需要人工干预的扩缩容、版本升级、证书轮换等操作现在只需修改YAML中的spec字段即可自动完成。例如将Elasticsearch节点从3个扩展到5个原本需要处理PVC绑定、服务发现等复杂操作现在只需要将nodeCount: 3改为nodeCount: 5并apply即可。2. 环境准备与ECK Operator安装2.1 前置条件检查在开始部署前必须确保Kubernetes集群满足以下要求Kubernetes版本≥1.21推荐1.25每个节点至少2核CPU和4GB内存默认StorageClass已正确配置集群有足够的资源配额特别是生产环境验证命令示例kubectl version --short kubectl get nodes -o wide kubectl get storageclass重要提示如果使用托管K8S服务如EKS、AKS需要确保IAM权限包含对PersistentVolume的创建权限。我们曾在AWS环境遇到因IAM角色缺失导致PVC一直处于Pending状态的案例。2.2 Operator安装最佳实践ECK Operator的安装推荐使用cert-manager进行证书管理以下是经过生产验证的安装流程安装cert-manager如果尚未安装kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml部署ECK Operator# eck-operator.yaml apiVersion: operator.k8s.io/v1 kind: Operator metadata: name: elastic-operator spec: channel: stable packageName: elastic-cloud-eck targetNamespaces: - elastic-system应用配置kubectl create namespace elastic-system kubectl apply -f eck-operator.yaml验证安装kubectl -n elastic-system get pods # 预期看到elastic-operator开头的Pod状态为Running3. Elasticsearch集群YAML配置详解3.1 基础集群配置以下是一个三节点Elasticsearch集群的完整YAML示例包含了生产环境必需的配置项# elasticsearch-cluster.yaml apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: production-cluster namespace: elastic-system spec: version: 8.11.4 nodeSets: - name: default count: 3 config: node.roles: [master, data, ingest] xpack.security.enabled: true podTemplate: spec: containers: - name: elasticsearch resources: requests: memory: 4Gi cpu: 2 limits: memory: 8Gi cpu: 4 volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: [ ReadWriteOnce ] storageClassName: gp3 resources: requests: storage: 100Gi关键参数解析node.roles明确指定节点角色生产环境建议分离master和data节点resourcesJVM堆内存建议设置为容器内存的50%这里配置了4GB请求/8GB限制volumeClaimTemplates使用gp3存储类AWS环境100GB容量3.2 安全配置进阶ECK会自动生成默认用户密码但生产环境应该自定义密码和证书spec: http: tls: selfSignedCertificate: disabled: true certificate: secretName: es-http-certs auth: roles: - secretName: es-roles fileRealm: - secretName: es-users需要提前创建对应的Secret# 生成PKCS#12格式的证书 openssl pkcs12 -export -out cert.p12 -inkey key.pem -in cert.pem # 创建K8S Secret kubectl -n elastic-system create secret generic es-http-certs \ --from-filetls.crtcert.pem \ --from-filetls.keykey.pem \ --from-fileca.crtca.pem \ --from-filekeystore.p12cert.p124. Kibana与APM集成配置4.1 Kibana部署配置与Elasticsearch配套的Kibana实例配置示例# kibana.yaml apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: production-kibana namespace: elastic-system spec: version: 8.11.4 count: 2 elasticsearchRef: name: production-cluster http: tls: selfSignedCertificate: disabled: true certificate: secretName: kibana-certs podTemplate: spec: containers: - name: kibana resources: requests: memory: 1Gi cpu: 0.5 limits: memory: 2Gi cpu: 24.2 APM Server集成对于需要应用性能监控的场景APM Server的配置如下# apm-server.yaml apiVersion: apm.k8s.elastic.co/v1 kind: ApmServer metadata: name: production-apm namespace: elastic-system spec: version: 8.11.4 count: 2 elasticsearchRef: name: production-cluster http: tls: selfSignedCertificate: disabled: false podTemplate: spec: containers: - name: apm-server resources: requests: memory: 1Gi cpu: 0.5 limits: memory: 2Gi cpu: 25. 运维与监控实践5.1 集群扩缩容操作横向扩展data节点从3个扩展到5个spec: nodeSets: - name: default count: 5 # 修改此处垂直扩展资源内存从4Gi增加到8GipodTemplate: spec: containers: - name: elasticsearch resources: requests: memory: 8Gi # 修改此处 limits: memory: 16Gi # 修改此处重要经验每次扩缩容操作后使用kubectl get elasticsearch观察health状态直到变为green再进行下一步操作。我们曾因连续操作导致集群状态异常。5.2 监控配置ECK内置了Prometheus指标导出配置示例spec: monitoring: metrics: elasticsearchRef: name: production-cluster logs: enabled: true对应的Prometheus scrape配置- job_name: elasticsearch metrics_path: /metrics static_configs: - targets: [production-cluster-es-http.elastic-system.svc:9200]6. 故障排查与常见问题6.1 Pod启动失败排查流程查看Operator日志kubectl -n elastic-system logs -l control-planeelastic-operator检查具体Pod事件kubectl -n elastic-system describe pod production-cluster-es-default-0常见错误解决方案错误现象可能原因解决方案Pod一直ContainerCreatingPVC无法绑定检查StorageClass和PV配置节点间无法通信网络策略限制配置合适的NetworkPolicyJVM内存不足资源限制过低增加resources.limits.memory6.2 证书过期处理ECK默认证书有效期为1年续期方法kubectl -n elastic-system delete secret production-cluster-es-http-certs-public kubectl -n elastic-system delete secret production-cluster-es-http-certs-internal # Operator会自动重新生成证书7. 版本升级策略ECK支持滚动升级以下是主要步骤检查兼容性矩阵确保当前版本与目标版本兼容修改YAML中的version字段spec: version: 8.12.0 # 新版本号分阶段升级先升级一个小型测试集群观察24小时无异常后再升级生产集群升级后验证curl -XGET https://localhost:9200 -u elastic -k # 检查返回的version.number字段8. 备份与灾难恢复8.1 快照仓库配置首先创建S3兼容的存储库apiVersion: elasticsearch.k8s.elastic.co/v1 kind: ElasticsearchSnapshotRepository metadata: name: s3-backup namespace: elastic-system spec: type: s3 settings: bucket: my-elastic-backups client: default endpoint: s3.amazonaws.com region: us-west-2然后创建定期快照策略apiVersion: elasticsearch.k8s.elastic.co/v1 kind: ElasticsearchSnapshotLifecyclePolicy metadata: name: daily-snapshots namespace: elastic-system spec: name: daily-snap-{now/d} schedule: 0 30 1 * * ? repository: s3-backup config: indices: [*] ignore_unavailable: true include_global_state: false8.2 恢复流程从快照恢复的YAML示例apiVersion: elasticsearch.k8s.elastic.co/v1 kind: ElasticsearchRestore metadata: name: disaster-recovery namespace: elastic-system spec: elasticsearchRef: name: production-cluster snapshotName: daily-snap-2024.06.01 repositoryName: s3-backup9. 性能优化实践9.1 JVM调优建议在resources中显式设置ES_JAVA_OPTSpodTemplate: spec: containers: - name: elasticsearch env: - name: ES_JAVA_OPTS value: -Xms4g -Xmx4g -XX:UseG1GC9.2 分片策略优化通过index模板控制分片PUT _index_template/production-template { index_patterns: [logs-*], template: { settings: { number_of_shards: 3, number_of_replicas: 1, index.routing.allocation.require.box_type: hot } } }10. 安全加固措施10.1 网络策略限制Elasticsearch API访问apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: es-api-access namespace: elastic-system spec: podSelector: matchLabels: elasticsearch.k8s.elastic.co/cluster-name: production-cluster policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: name: monitoring ports: - protocol: TCP port: 920010.2 审计日志配置启用安全审计spec: nodeSets: - config: xpack.security.audit.enabled: true xpack.security.audit.logfile.events.include: access_denied,anonymous_access_denied,authentication_failed