Prometheus Operator 数据持久化
来源:原创
时间:2020-04-04
作者:脚本小站
分类:DevOps
NFS方式
使用NFS作为持久化存储:
先在任意主机上创建好nfs服务,然后在k8s中部署如下几个资源,这里用到的是nfs-client-provisioner,官方文档如下:
https://github.com/kubernetes-retired/external-storage/tree/master/nfs-client
新地址:
https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner
需要部署的几个文件:
nfs-storageclass.yaml
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: managed-nfs-storage provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME' parameters: archiveOnDelete: "false"
rbac.yaml
apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default roleRef: kind: Role name: leader-locking-nfs-client-provisioner apiGroup: rbac.authorization.k8s.io
deployment.yaml:注意修改nfs服务地址和目录
apiVersion: apps/v1 kind: Deployment metadata: name: nfs-client-provisioner labels: app: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: default spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: nfs-client-provisioner template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: nfs-client-provisioner image: quay.io/external_storage/nfs-client-provisioner:latest volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: fuseim.pri/ifs # 如果有多个storageclass和多个nfs这里的值要不一样 - name: NFS_SERVER value: 192.168.0.71 - name: NFS_PATH value: /opt/nfs volumes: - name: nfs-client-root nfs: server: 192.168.0.71 path: /opt/nfs
prometheus-prometheus.yaml
apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: labels: prometheus: k8s name: k8s namespace: monitoring spec: retention: "30d" # 数据保存时间 alerting: alertmanagers: - name: alertmanager-main namespace: monitoring port: web storage: # 添加storage信息 volumeClaimTemplate: spec: storageClassName: managed-nfs-storage # 这个名称要和创建的storageclass名称一致 resources: requests: storage: 20Gi image: quay.io/prometheus/prometheus:v2.15.2 nodeSelector: kubernetes.io/os: linux podMonitorNamespaceSelector: {} podMonitorSelector: {} replicas: 2 resources: requests: memory: 400Mi ruleSelector: matchLabels: prometheus: k8s role: alert-rules securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 additionalScrapeConfigs: name: additional-configs key: prometheus-additional.yaml serviceAccountName: prometheus-k8s serviceMonitorNamespaceSelector: {} serviceMonitorSelector: {} version: v2.15.2
如果手动创建Pod测试需要手动创建pvc资源文件如下,如果想要删除pv则先要让应用停止使用存储,再删除pvc,删除pvc之后pv会自动删除。
nfs-pvc.yaml
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: prometheus-k8s-db-prometheus-k8s-0 namespace: monitoring annotations: volume.beta.kubernetes.io/storage-class: "managed-nfs-storage" spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: prometheus-k8s-db-prometheus-k8s-1 namespace: monitoring annotations: volume.beta.kubernetes.io/storage-class: "managed-nfs-storage" spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi
Ceph RBD方式
使用Ceph rbd作为持久化存储:
创建完StorageClass之后填写名称到prometheus-prometheus.yaml资源文件中即可。
apiVersion: v1 kind: Secret metadata: name: ceph-admin-secret namespace: monitoring data: key: QVFBbjVuVmVSZDJrS3hBQUlRZE9xcDkrSlQrVStzQUhIbVMzWGc9PQ== type: kubernetes.io/rbd --- apiVersion: v1 kind: Secret metadata: name: ceph-k8s-secret namespace: monitoring data: key: QVFCWEhuWmVSazlkSnhBQVJoenZEeUpnR1hFVDY4dzc0WW9KVmc9PQ== type: kubernetes.io/rbd --- apiVersion: storage.k8s.io/v1beta1 kind: StorageClass metadata: name: rbd-dynamic annotations: storageclass.beta.kubernetes.io/is-default-class: "true" provisioner: kubernetes.io/rbd reclaimPolicy: Retain parameters: monitors: 192.168.0.34:6789 adminId: admin adminSecretName: ceph-admin-secret adminSecretNamespace: monitoring pool: kube userId: k8s userSecretName: ceph-k8s-secret
配置存储时限:监控数据要保存多少天,修改资源文件 prometheus-prometheus.yaml
spec: retention: "30d" # [0-9]+(ms|s|m|h|d|w|y) (milliseconds seconds minutes hours days weeks years)
CephFS方式
使用Cephfs作为持久化存储:
示例地址:
https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/cephfs/deploy
storageclass.yaml
kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: cephfs provisioner: ceph.com/cephfs parameters: monitors: 192.168.0.34:6789 adminId: admin adminSecretName: ceph-admin-secret adminSecretNamespace: cephfs claimRoot: /pvc-volumes --- apiVersion: v1 kind: Secret metadata: name: ceph-admin-secret namespace: cephfs data: key: QVFBbjVuVmVSZDJrS3hBQUlRZE9xcDkrSlQrVStzQUhIbVMzWGc9PQ== type: kubernetes.io/rbd
deployment.yaml
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cephfs-provisioner namespace: cephfs rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] - apiGroups: [""] resources: ["services"] resourceNames: ["kube-dns","coredns"] verbs: ["list", "get"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cephfs-provisioner subjects: - kind: ServiceAccount name: cephfs-provisioner namespace: cephfs roleRef: kind: ClusterRole name: cephfs-provisioner apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: cephfs-provisioner namespace: cephfs rules: - apiGroups: [""] resources: ["secrets"] verbs: ["create", "get", "delete"] - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: cephfs-provisioner namespace: cephfs roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: cephfs-provisioner subjects: - kind: ServiceAccount name: cephfs-provisioner --- apiVersion: v1 kind: ServiceAccount metadata: name: cephfs-provisioner namespace: cephfs --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: cephfs-provisioner namespace: cephfs spec: replicas: 1 strategy: type: Recreate template: metadata: labels: app: cephfs-provisioner spec: containers: - name: cephfs-provisioner image: "quay.io/external_storage/cephfs-provisioner:latest" env: - name: PROVISIONER_NAME value: ceph.com/cephfs - name: PROVISIONER_SECRET_NAMESPACE value: cephfs command: - "/usr/local/bin/cephfs-provisioner" args: - "-id=cephfs-provisioner-1" serviceAccount: cephfs-provisioner
cephfs-pvc.yaml
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: prometheus-k8s-db-prometheus-k8s-0 namespace: monitoring spec: storageClassName: cephfs accessModes: - ReadWriteMany resources: requests: storage: 2Gi --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: prometheus-k8s-db-prometheus-k8s-1 namespace: monitoring spec: storageClassName: cephfs accessModes: - ReadWriteMany resources: requests: storage: 2Gi
注意:在使用Cephfs作为存储的时候如果prometheus-k8s-x出现 CrashLoopBackOff 的话可能是权限问题导致prometheus无权写入文件系统,只要修改 prometheus-prometheus.yaml文件中的runAsUser等选项即可。
securityContext: fsGroup: 0 runAsNonRoot: false runAsUser: 0
storage写法示例:
storage: volumeClaimTemplate: spec: storageClassName: cephfs accessModes: [ "ReadWriteMany" ] resources: requests: storage: 2Gi