Clever Geek Handbook

Contoh praktis menghubungkan penyimpanan berbasis Ceph ke cluster Kubernetes

Container Storage Interface (CSI) adalah antarmuka terpadu antara Kubernetes dan sistem penyimpanan. Kita telah membahasnya secara singkat , dan hari ini kita akan melihat lebih dekat kombinasi CSI dan Ceph: kita akan menunjukkan bagaimana menghubungkan penyimpanan Ceph ke cluster Kubernetes.

Artikel tersebut berisi contoh nyata, meskipun sedikit disederhanakan untuk memudahkan persepsi. Kami tidak akan mempertimbangkan untuk menginstal dan mengkonfigurasi cluster Ceph dan Kubernetes.



Apakah Anda bertanya-tanya bagaimana cara kerjanya?





, Kubernetes, , , kubespray. Ceph — , , . , , 10 /.



, !



Ceph , :



ceph health
ceph -s


RBD :



ceph osd pool create kube 32
ceph osd pool application enable kube rbd


Kubernetes. Ceph CSI RBD. , , Helm.

, ceph-csi-rbd:



helm repo add ceph-csi https://ceph.github.io/csi-charts
helm inspect values ceph-csi/ceph-csi-rbd > cephrbd.yml


cephrbd.yml. ID IP- Ceph:



ceph fsid  #    clusterID
ceph mon dump  #    IP-


cephrbd.yml. PSP (Pod Security Policies). nodeplugin provisioner , , :



csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "v2:172.18.8.5:3300/0,v1:172.18.8.5:6789/0"
      - "v2:172.18.8.6:3300/0,v1:172.18.8.6:6789/0"
      - "v2:172.18.8.7:3300/0,v1:172.18.8.7:6789/0"

nodeplugin:
  podSecurityPolicy:
    enabled: true

provisioner:
  podSecurityPolicy:
    enabled: true


— Kubernetes.



helm upgrade -i ceph-csi-rbd ceph-csi/ceph-csi-rbd -f cephrbd.yml -n ceph-csi-rbd --create-namespace


, RBD !

Kubernetes StorageClass. Ceph.



Ceph kube:



ceph auth get-or-create client.rbdkube mon 'profile rbd' osd 'profile rbd pool=kube'


:



ceph auth get-key client.rbdkube


:



AQCO9NJbhYipKRAAMqZsnqqS/T8OYQX20xIa9A==


Secret Kubernetes — , userKey:



---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: ceph-csi-rbd
stringData:
  #        ,   
  #  Ceph. ID      ,
  #   storage class
  userID: rbdkube
  userKey: <user-key>


: 



kubectl apply -f secret.yaml


StorageClass:



---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: <cluster-id>
   pool: kube

   imageFeatures: layering

   #       
   #   .
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-rbd
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-rbd

   csi.storage.k8s.io/fstype: ext4

reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - discard


clusterID, ceph fsid, Kubernetes: 



kubectl apply -f storageclass.yaml


, PVC (Persistent Volume Claim):



apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc


, Kubernetes Ceph :



kubectl get pvc
kubectl get pv


! Ceph?

:



rbd ls -p kube
rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653  # ,  ,   ID ,


, RBD.

pvc.yaml 2Gi :



kubectl apply -f pvc.yaml


, , .



rbd -p kube info csi-vol-eb3d257d-8c6c-11ea-bff5-6235e7640653

kubectl get pv
kubectl get pvc


, PVC . , Kubernetes PVC YAML: 



kubectl get pvc rbd-pvc -o yaml


:



message: Waiting for user to (re-)start a pod to finish file system resize of volume on node. type: FileSystemResizePending



, — .

, . PVC/PV .



Pod, :



---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx:1.17.6
      volumeMounts:
        - name: mypvc
          mountPath: /data
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false


PVC:



kubectl get pvc


, .



RBD ( – Rados Block Device), , . , , CephFS.

Ceph Kubernetes CSI CephFS.



Helm-:



helm inspect values ceph-csi/ceph-csi-cephfs > cephfs.yml


cephfs.yml. , Ceph:



ceph fsid
ceph mon dump


:



csiConfig:
  - clusterID: "bcd0d202-fba8-4352-b25d-75c89258d5ab"
    monitors:
      - "172.18.8.5:6789"
      - "172.18.8.6:6789"
      - "172.18.8.7:6789"

nodeplugin:
  httpMetrics:
    enabled: true
    containerPort: 8091
  podSecurityPolicy:
    enabled: true

provisioner:
  replicaCount: 1
  podSecurityPolicy:
    enabled: true


, address:port. cephfs , v2.

httpMetrics ( Prometheus ) , nginx-proxy, Kubespray’. , , .



Helm- Kubernetes:



helm upgrade -i ceph-csi-cephfs ceph-csi/ceph-csi-cephfs -f cephfs.yml -n ceph-csi-cephfs --create-namespace


Ceph, . , CephFS . fs :



ceph auth get-or-create client.fs mon 'allow r' mgr 'allow rw' mds 'allow rws' osd 'allow rw pool=cephfs_data, allow rw pool=cephfs_metadata'


, :



ceph auth get-key client.fs


Secret StorageClass.

, RBD:



---
apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-secret
  namespace: ceph-csi-cephfs
stringData:
  #     
  adminID: fs
  adminKey: <  >


:



kubectl apply -f secret.yaml


– StorageClass:



---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs-sc
provisioner: cephfs.csi.ceph.com
parameters:
  clusterID: <cluster-id>

  #    CephFS,     
  fsName: cephfs

  # ()  Ceph,      
  # pool: cephfs_data

  # ()      Ceph-fuse
  # :
  # fuseMountOptions: debug

  # ()     CephFS  
  # . man mount.ceph     . :
  # kernelMountOptions: readdir_max_bytes=1048576,norbytes

  #       /  Ceph.
  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/controller-expand-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-cephfs
  csi.storage.k8s.io/node-stage-secret-name: csi-cephfs-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-cephfs

  # ()     ceph-fuse (fuse), 
  #  ceph kernelclient (kernel).
  #   ,      ,
  #    ceph-fuse  mount.ceph
  # mounter: kernel
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
  - debug


clusterID Kubernetes:



kubectl apply -f storageclass.yaml




, , PVC:



---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-cephfs-sc


PVC/PV:



kubectl get pvc
kubectl get pv


CephFS, -. , .



Ceph :



#  
mkdir -p /mnt/cephfs

#     
ceph auth get-key client.admin >/etc/ceph/secret.key

#    /etc/fstab
# !!  ip     
echo "172.18.8.6:6789:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/secret.key,noatime,_netdev    0       2" >> /etc/fstab

mount /mnt/cephfs


, FS Ceph , . , - , .



, CephFS . Kubernetes PVC — , , 7Gi.



: 



kubectl apply -f pvc.yaml


, :



getfattr -n ceph.quota.max_bytes <-->


, , attr.



,



YAML , .

— . Ceph Kubernetes, :



Kubernetes c

RBD

RBD Kubernetes Ceph

RBD Kubernetes CSI

CephFS

CephFS Kubernetes CSI



Kubernetes Kubernetes , CephFS . GET/POST Ceph.



, Ceph. -, .



: , Southbridge, Certified Kubernetes Administrator, .



More articles:


All Articles