Prometheus Operator 数据持久化

NFS方式

使用NFS作为持久化存储：

        先在任意主机上创建好nfs服务，然后在k8s中部署如下几个资源，这里用到的是nfs-client-provisioner，官方文档如下：

https://github.com/kubernetes-retired/external-storage/tree/master/nfs-client

新地址：

https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner

需要部署的几个文件：

nfs-storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"

rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

deployment.yaml：注意修改nfs服务地址和目录

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs # 如果有多个storageclass和多个nfs这里的值要不一样
            - name: NFS_SERVER
              value: 192.168.0.71
            - name: NFS_PATH
              value: /opt/nfs
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.0.71
            path: /opt/nfs

prometheus-prometheus.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  retention: "30d" # 数据保存时间
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  storage: # 添加storage信息
    volumeClaimTemplate:
      spec:
        storageClassName: managed-nfs-storage # 这个名称要和创建的storageclass名称一致
        resources:
          requests:
            storage: 20Gi
  image: quay.io/prometheus/prometheus:v2.15.2
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  additionalScrapeConfigs:
    name: additional-configs
    key: prometheus-additional.yaml
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.15.2

        如果手动创建Pod测试需要手动创建pvc资源文件如下，如果想要删除pv则先要让应用停止使用存储，再删除pvc，删除pvc之后pv会自动删除。

nfs-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: prometheus-k8s-db-prometheus-k8s-0
  namespace: monitoring
  annotations:
    volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: prometheus-k8s-db-prometheus-k8s-1
  namespace: monitoring
  annotations:
    volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

Ceph RBD方式

使用Ceph rbd作为持久化存储：

        创建完StorageClass之后填写名称到prometheus-prometheus.yaml资源文件中即可。

apiVersion: v1
kind: Secret
metadata:
  name: ceph-admin-secret
  namespace: monitoring
data:
  key: QVFBbjVuVmVSZDJrS3hBQUlRZE9xcDkrSlQrVStzQUhIbVMzWGc9PQ==
type: kubernetes.io/rbd
---
apiVersion: v1
kind: Secret
metadata:
  name: ceph-k8s-secret
  namespace: monitoring
data:
  key: QVFCWEhuWmVSazlkSnhBQVJoenZEeUpnR1hFVDY4dzc0WW9KVmc9PQ==
type: kubernetes.io/rbd
---
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: rbd-dynamic
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/rbd
reclaimPolicy: Retain
parameters:
  monitors: 192.168.0.34:6789
  adminId: admin
  adminSecretName: ceph-admin-secret
  adminSecretNamespace: monitoring
  pool: kube
  userId: k8s
  userSecretName: ceph-k8s-secret

配置存储时限：监控数据要保存多少天，修改资源文件 prometheus-prometheus.yaml

spec:
  retention: "30d" # [0-9]+(ms|s|m|h|d|w|y) (milliseconds seconds minutes hours days weeks years)

CephFS方式

使用Cephfs作为持久化存储：

示例地址：

https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/cephfs/deploy

storageclass.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: cephfs
provisioner: ceph.com/cephfs
parameters:
  monitors: 192.168.0.34:6789
  adminId: admin
  adminSecretName: ceph-admin-secret
  adminSecretNamespace: cephfs
  claimRoot: /pvc-volumes
---
apiVersion: v1
kind: Secret
metadata:
  name: ceph-admin-secret
  namespace: cephfs
data:
  key: QVFBbjVuVmVSZDJrS3hBQUlRZE9xcDkrSlQrVStzQUhIbVMzWGc9PQ==
type: kubernetes.io/rbd

deployment.yaml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-provisioner
  namespace: cephfs
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["kube-dns","coredns"]
    verbs: ["list", "get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-provisioner
subjects:
  - kind: ServiceAccount
    name: cephfs-provisioner
    namespace: cephfs
roleRef:
  kind: ClusterRole
  name: cephfs-provisioner
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cephfs-provisioner
  namespace: cephfs
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "delete"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cephfs-provisioner
  namespace: cephfs
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cephfs-provisioner
subjects:
- kind: ServiceAccount
  name: cephfs-provisioner
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cephfs-provisioner
  namespace: cephfs
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: cephfs-provisioner
  namespace: cephfs
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: cephfs-provisioner
    spec:
      containers:
      - name: cephfs-provisioner
        image: "quay.io/external_storage/cephfs-provisioner:latest"
        env:
        - name: PROVISIONER_NAME
          value: ceph.com/cephfs
        - name: PROVISIONER_SECRET_NAMESPACE
          value: cephfs
        command:
        - "/usr/local/bin/cephfs-provisioner"
        args:
        - "-id=cephfs-provisioner-1"
      serviceAccount: cephfs-provisioner

cephfs-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: prometheus-k8s-db-prometheus-k8s-0
  namespace: monitoring
spec:
  storageClassName: cephfs
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: prometheus-k8s-db-prometheus-k8s-1
  namespace: monitoring
spec:
  storageClassName: cephfs
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

注意：在使用Cephfs作为存储的时候如果prometheus-k8s-x出现 CrashLoopBackOff 的话可能是权限问题导致prometheus无权写入文件系统，只要修改 prometheus-prometheus.yaml文件中的runAsUser等选项即可。

  securityContext:
    fsGroup: 0
    runAsNonRoot: false
    runAsUser: 0

storage写法示例：

  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: cephfs
        accessModes: [ "ReadWriteMany" ]
        resources:
          requests:
            storage: 2Gi