Skip to content

Use Local Persistent Volumes in a NebulaGraph cluster

Local Persistent Volumes, abbreviated as Local PVs in K8s store container data directly using the node's local disk directory. Compared with network storage, Local Persistent Volumes provide higher IOPS and lower read and write latency, which is suitable for data-intensive applications. This topic introduces how to use Local PVs in Google Kubernetes Engine (GKE) and Amazon Elastic Kubernetes Service (EKS) clusters, and how to enable automatic failover for Local PVs in the cloud.

While using Local Persistent Volumes can enhance performance, it's essential to note that, unlike network storage, local storage does not support automatic backup. In the event of a node failure, all data in local storage may be lost. Therefore, the utilization of Local Persistent Volumes involves a trade-off between service availability, data persistence, and flexibility.

Principles

NebulaGraph Operator implements a Storage Volume Provisioner interface to automatically create and delete PV objects. Utilizing the provisioner, you can dynamically generate Local PVs as required. Based on the PVC and StorageClass specified in the cluster configuration file, NebulaGraph Operator automatically generates PVCs and associates them with their respective Local PVs.

When a Local PV is initiated by the provisioner interface, the provisioner controller generates a local type PV and configures the nodeAffinity field. This configuration ensures that Pods using the local type PV are scheduled onto specific nodes. Conversely, when a Local PV is deleted, the provisioner controller eliminates the local type PV object and purges the node's storage resources.

Prerequisites

NebulaGraph Operator is installed. For details, see Install NebulaGraph Operator.

Steps

The resources in the following examples are all created in the default namespace.

  1. Create a node pool with local SSDs if not existing

    gcloud container node-pools create "pool-1" --cluster "gke-1" --region us-central1 --node-version "1.27.10-gke.1055000" --machine-type "n2-standard-2" --local-nvme-ssd-block count=2 --max-surge-upgrade 1 --max-unavailable-upgrade 0 --num-nodes 1 --enable-autoscaling --min-nodes 1 --max-nodes 2
    

    For information about the parameters to create a node pool with local SSDs, see Create a node pool with Local SSD.

  2. Format and mount the local SSDs using a DaemonSet.

    1. Download the gke-daemonset-raid-disks.yaml file.

    2. Deploy the RAID disks DaemonSet. The DaemonSet sets a RAID 0 array on all Local SSD disks and formats the device to an ext4 filesystem.

      kubectl apply -f gke-daemonset-raid-disks.yaml
      

  3. Deploy the Local PV provisioner.

    1. Download the local-pv-provisioner.yaml file.
    2. Run the provisioner.
      kubectl apply -f local-pv-provisioner.yaml
      
  4. In the NebulaGraph cluster configuration file, specify spec.storaged.dataVolumeClaims or spec.metad.dataVolumeClaim, and the StorageClass needs to be configured as local-nvme. For more information about cluster configurations, see Create a NebulaGraph cluster.

    Partial configuration of the NebulaGraph cluster
    ...
    metad: 
      dataVolumeClaim:
        resources:
          requests:
            storage: 2Gi
        storageClassName: local-nvme
    storaged:
      dataVolumeClaims:
      - resources:
          requests:
            storage: 2Gi
        storageClassName: local-nvme  
    ...
    

    After the NebulaGraph is deployed, the Local PVs are automatically created.

  5. View the PV list.

    kubectl get pv
    

    Return:

    NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                     STORAGECLASS   REASON   AGE
    pvc-01be9b75-9c50-4532-8695-08e11b489718   5Gi        RWO            Delete           Bound    default/storaged-data-nebula-storaged-0   local-nvme              3m35s
    pvc-09de8eb1-1225-4025-b91b-fbc0bcce670f   5Gi        RWO            Delete           Bound    default/storaged-data-nebula-storaged-1   local-nvme              3m35s
    pvc-4b2a9ffb-9000-4998-a7bb-edb825c872cb   5Gi        RWO            Delete           Bound    default/storaged-data-nebula-storaged-2   local-nvme              3m35s
    ...
    
  6. View the detailed information of the PV.

    kubectl get pv pvc-01be9b75-9c50-4532-8695-08e11b489718 -o yaml
    

    Return:

    apiVersion: v1
    kind: PersistentVolume
    metadata:
      annotations:
        local.pv.provisioner/selected-node: gke-snap-test-snap-test-591403a8-xdfc
        nebula-graph.io/pod-name: nebula-storaged-0
        pv.kubernetes.io/provisioned-by: nebula-cloud.io/local-pv
      creationTimestamp: "2024-03-05T06:12:32Z"
      finalizers:
      - kubernetes.io/pv-protection
      labels:
        app.kubernetes.io/cluster: nebula
        app.kubernetes.io/component: storaged
        app.kubernetes.io/managed-by: nebula-operator
        app.kubernetes.io/name: nebula-graph
      name: pvc-01be9b75-9c50-4532-8695-08e11b489718
      resourceVersion: "9999469"
      uid: ee28a4da-6026-49ac-819b-2075154b4724
    spec:
      accessModes:
      - ReadWriteOnce
      capacity:
        storage: 5Gi
      claimRef:
        apiVersion: v1
        kind: PersistentVolumeClaim
        name: storaged-data-nebula-storaged-0
        namespace: default
        resourceVersion: "9996541"
        uid: 01be9b75-9c50-4532-8695-08e11b489718
      local:
        fsType: ext4
        path: /mnt/disks/raid0
      nodeAffinity:
        required:
          nodeSelectorTerms:
          - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - gke-snap-test-snap-test-591403a8-xdfc
      persistentVolumeReclaimPolicy: Delete
      storageClassName: local-nvme
      volumeMode: Filesystem
    status:
      phase: Bound    
    
  1. Create a node pool with Instance Store if not existing.

    eksctl create nodegroup  --instance-types m5ad.2xlarge  --nodes 3  --cluster eks-1
    

    For more information about parameters to cluster node pools, see Creating a managed node group.

  2. Format and mount the local SSDs using a DaemonSet.

    1. Download the eks-daemonset-raid-disks.yaml file.

    2. Based on the node type created in step 1, modify the value of the nodeSelector.node.kubernetes.io/instance-type field in the eks-daemonset-raid-disks.yaml file as needed.

          spec:
            nodeSelector:
              node.kubernetes.io/instance-type: "m5ad.2xlarge"
      

    3. Install nvme-cli.

      • For Ubuntu and Debian systems
        sudo apt-get update
        sudo apt-get install -y nvme-cli
        
      • For CentOS and RHEL systems
        sudo yum install -y nvme-cli
        
    4. Deploy the RAID disk DaemonSet. The DaemonSet sets up a RAID 0 array on all local SSD disks and formats the devices as an ext4 file system.

      kubectl apply -f gke-daemonset-raid-disks.yaml
      

  3. Deploy the Local PV provisioner.

    1. Download the local-pv-provisioner.yaml file.
    2. Run the provisioner.
      kubectl apply -f local-pv-provisioner.yaml
      
  4. In the NebulaGraph cluster configuration file, specify spec.storaged.dataVolumeClaims or spec.metad.dataVolumeClaim, and the StorageClass needs to be configured as local-nvme. For more information about cluster configurations, see Create a NebulaGraph cluster.

    Partial configuration of the NebulaGraph cluster
    metad:
      dataVolumeClaim:
        resources:
          requests:
            storage: 2Gi
        storageClassName: local-nvme
    storaged:
      dataVolumeClaims:
      - resources:
          requests:
            storage: 2Gi
        storageClassName: local-nvme  
    
  5. View the PV list.

    kubectl get pv
    

    Return:

    NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                     STORAGECLASS   REASON   AGE
    pvc-290c15cc-a302-4463-a591-84b7217a6cd2   5Gi        RWO            Delete           Bound    default/storaged-data-nebula-storaged-0   local-nvme              3m40s
    pvc-fbb3167f-f556-4a16-ae0e-171aed0ac954   5Gi        RWO            Delete           Bound    default/storaged-data-nebula-storaged-1   local-nvme              3m40s
    pvc-6c7cfe80-0134-4573-b93e-9b259c6fcd63   5Gi        RWO            Delete           Bound    default/storaged-data-nebula-storaged-2   local-nvme              3m40s
    ...
    
  6. View the detailed information of the PV.

    kubectl get pv pvc-290c15cc-a302-4463-a591-84b7217a6cd2 -o yaml
    

    Return:

    apiVersion: v1
    kind: PersistentVolume
    metadata:
      annotations:
        local.pv.provisioner/selected-node: ip-192-168-77-60.ec2.internal
        nebula-graph.io/pod-name: nebula-storaged-0
        pv.kubernetes.io/provisioned-by: nebula-cloud.io/local-pv
      creationTimestamp: "2024-03-04T07:51:32Z"
      finalizers:
      - kubernetes.io/pv-protection
      labels:
        app.kubernetes.io/cluster: nebula
        app.kubernetes.io/component: storaged
        app.kubernetes.io/managed-by: nebula-operator
        app.kubernetes.io/name: nebula-graph
      name: pvc-290c15cc-a302-4463-a591-84b7217a6cd2
      resourceVersion: "7932689"
      uid: 66c0a2d3-2914-43ad-93b5-6d84fb62acef
    spec:
      accessModes:
      - ReadWriteOnce
      capacity:
        storage: 5Gi
      claimRef:
        apiVersion: v1
        kind: PersistentVolumeClaim
        name: storaged-data-nebula-storaged-0
        namespace: default
        resourceVersion: "7932688"
        uid: 8ecb5d96-004b-4672-bac4-1355ae15eae4
      local:
        fsType: ext4
        path: /mnt/disks/raid0
      nodeAffinity:
        required:
          nodeSelectorTerms:
          - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - ip-192-168-77-60.ec2.internal
      persistentVolumeReclaimPolicy: Delete
      storageClassName: local-nvme
      volumeMode: Filesystem
    status:
      phase: Bound    
    

Failover for Local Persistent Volumes in the cloud

When using network storage (e.g., AWS EBS, Google Cloud Persistent Disk, Azure Disk Storage, Ceph, NFS, etc.) as a PV, the storage resource is independent of any particular node. Therefore, the storage resource can be mounted and used by Pods regardless of the node to which the Pods are scheduled. However, when using a local storage disk as a PV, the storage resource can only be used by Pods on a specific node due to nodeAffinity.

The Storage service of NebulaGraph supports data redundancy, which allows you to set multiple odd-numbered partition replicas. When a node fails, the associated partition is automatically transferred to a healthy node. However, Storage Pods using Local Persistent Volumes cannot run on other nodes due to the node affinity setting and must wait for the node to recover. To run on another node, the Pods must be unbound from the associated Local Persistent Volume.

NebulaGraph Operator supports automatic failover in the event of a node failure while using Local Persistent Volumes in the cloud for elastic scaling. This is achieved by setting spec.enableAutoFailover to true in the cluster configuration file, which automatically unbinds the Pods from the Local Persistent Volume, allowing the Pods to run on another node.

Example configuration:

...
spec:
  # Enable automatic failover for Local PV.
  enableAutoFailover: true
  # The time to wait for the Storage service to be in the `OFFLINE` status
  # before automatic failover. 
  # The default value is 5 minutes.
  # If the Storage service recovers to the `ONLINE` status during this period,
  # failover will not be triggered.
  failoverPeriod: "2m"
  ...

Last update: March 6, 2024