Use Local Persistent Volumes in a NebulaGraph cluster¶

Local Persistent Volumes, abbreviated as Local PVs in K8s store container data directly using the node's local disk directory. Compared with network storage, Local Persistent Volumes provide higher IOPS and lower read and write latency, which is suitable for data-intensive applications. This topic introduces how to use Local PVs in Google Kubernetes Engine (GKE) and Amazon Elastic Kubernetes Service (EKS) clusters, and how to enable automatic failover for Local PVs in the cloud.

While using Local Persistent Volumes can enhance performance, it's essential to note that, unlike network storage, local storage does not support automatic backup. In the event of a node failure, all data in local storage may be lost. Therefore, the utilization of Local Persistent Volumes involves a trade-off between service availability, data persistence, and flexibility.

Principles¶

NebulaGraph Operator implements a Storage Volume Provisioner interface to automatically create and delete PV objects. Utilizing the provisioner, you can dynamically generate Local PVs as required. Based on the PVC and StorageClass specified in the cluster configuration file, NebulaGraph Operator automatically generates PVCs and associates them with their respective Local PVs.

When a Local PV is initiated by the provisioner interface, the provisioner controller generates a local type PV and configures the nodeAffinity field. This configuration ensures that Pods using the local type PV are scheduled onto specific nodes. Conversely, when a Local PV is deleted, the provisioner controller eliminates the local type PV object and purges the node's storage resources.

Prerequisites¶

NebulaGraph Operator is installed. For details, see Install NebulaGraph Operator.

Steps¶

The resources in the following examples are all created in the default namespace.

Use Local PV on GKEUse Local PV on EKS

Create a node pool with local SSDs if not existing

gcloud container node-pools create "pool-1" --cluster "gke-1" --region us-central1 --node-version "1.27.10-gke.1055000" --machine-type "n2-standard-2" --local-nvme-ssd-block count=2 --max-surge-upgrade 1 --max-unavailable-upgrade 0 --num-nodes 1 --enable-autoscaling --min-nodes 1 --max-nodes 2

For information about the parameters to create a node pool with local SSDs, see Create a node pool with Local SSD.

Format and mount the local SSDs using a DaemonSet.
1. Download the gke-daemonset-raid-disks.yaml file.
2. Deploy the RAID disks DaemonSet. The DaemonSet sets a RAID 0 array on all Local SSD disks and formats the device to an ext4 filesystem.
```
kubectl apply -f gke-daemonset-raid-disks.yaml
```
Deploy the Local PV provisioner.
1. Download the local-pv-provisioner.yaml file.
2. Run the provisioner.
```
kubectl apply -f local-pv-provisioner.yaml
```
In the NebulaGraph cluster configuration file, specify spec.storaged.dataVolumeClaims or spec.metad.dataVolumeClaim, and the StorageClass needs to be configured as local-nvme. For more information about cluster configurations, see Create a NebulaGraph cluster.
Partial configuration of the NebulaGraph cluster
```
...
metad: 
  dataVolumeClaim:
    resources:
      requests:
        storage: 2Gi
    storageClassName: local-nvme
storaged:
  dataVolumeClaims:
  - resources:
      requests:
        storage: 2Gi
    storageClassName: local-nvme  
...
```
After the NebulaGraph is deployed, the Local PVs are automatically created.

View the PV list.

kubectl get pv

Output:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                     STORAGECLASS   REASON   AGE
pvc-01be9b75-9c50-4532-8695-08e11b489718   5Gi        RWO            Delete           Bound    default/storaged-data-nebula-storaged-0   local-nvme              3m35s
pvc-09de8eb1-1225-4025-b91b-fbc0bcce670f   5Gi        RWO            Delete           Bound    default/storaged-data-nebula-storaged-1   local-nvme              3m35s
pvc-4b2a9ffb-9000-4998-a7bb-edb825c872cb   5Gi        RWO            Delete           Bound    default/storaged-data-nebula-storaged-2   local-nvme              3m35s
...

View the detailed information of the PV.

kubectl get pv pvc-01be9b75-9c50-4532-8695-08e11b489718 -o yaml

Output:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    local.pv.provisioner/selected-node: gke-snap-test-snap-test-591403a8-xdfc
    nebula-graph.io/pod-name: nebula-storaged-0
    pv.kubernetes.io/provisioned-by: nebula-cloud.io/local-pv
  creationTimestamp: "2024-03-05T06:12:32Z"
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    app.kubernetes.io/cluster: nebula
    app.kubernetes.io/component: storaged
    app.kubernetes.io/managed-by: nebula-operator
    app.kubernetes.io/name: nebula-graph
  name: pvc-01be9b75-9c50-4532-8695-08e11b489718
  resourceVersion: "9999469"
  uid: ee28a4da-6026-49ac-819b-2075154b4724
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: storaged-data-nebula-storaged-0
    namespace: default
    resourceVersion: "9996541"
    uid: 01be9b75-9c50-4532-8695-08e11b489718
  local:
    fsType: ext4
    path: /mnt/disks/raid0
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - gke-snap-test-snap-test-591403a8-xdfc
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-nvme
  volumeMode: Filesystem
status:
  phase: Bound

Create a node pool with Instance Store if not existing.
```
eksctl create nodegroup  --instance-types m5ad.2xlarge  --nodes 3  --cluster eks-1
```
For more information about parameters to cluster node pools, see Creating a managed node group.
Format and mount the local SSDs using a DaemonSet.
1. Download the eks-daemonset-raid-disks.yaml file.
2. Based on the node type created in step 1, modify the value of the nodeSelector.node.kubernetes.io/instance-type field in the eks-daemonset-raid-disks.yaml file as needed.
```
    spec:
      nodeSelector:
        node.kubernetes.io/instance-type: "m5ad.2xlarge"
```
3. Install nvme-cli.
  - For Ubuntu and Debian systems
```
sudo apt-get update
sudo apt-get install -y nvme-cli
```
  - For CentOS and RHEL systems
```
sudo yum install -y nvme-cli
```
4. Deploy the RAID disk DaemonSet. The DaemonSet sets up a RAID 0 array on all local SSD disks and formats the devices as an ext4 file system.
```
kubectl apply -f gke-daemonset-raid-disks.yaml
```
Deploy the Local PV provisioner.
1. Download the local-pv-provisioner.yaml file.
2. Run the provisioner.
```
kubectl apply -f local-pv-provisioner.yaml
```
In the NebulaGraph cluster configuration file, specify spec.storaged.dataVolumeClaims or spec.metad.dataVolumeClaim, and the StorageClass needs to be configured as local-nvme. For more information about cluster configurations, see Create a NebulaGraph cluster.
Partial configuration of the NebulaGraph cluster
```
metad:
  dataVolumeClaim:
    resources:
      requests:
        storage: 2Gi
    storageClassName: local-nvme
storaged:
  dataVolumeClaims:
  - resources:
      requests:
        storage: 2Gi
    storageClassName: local-nvme  
```

View the PV list.

kubectl get pv

Output:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                     STORAGECLASS   REASON   AGE
pvc-290c15cc-a302-4463-a591-84b7217a6cd2   5Gi        RWO            Delete           Bound    default/storaged-data-nebula-storaged-0   local-nvme              3m40s
pvc-fbb3167f-f556-4a16-ae0e-171aed0ac954   5Gi        RWO            Delete           Bound    default/storaged-data-nebula-storaged-1   local-nvme              3m40s
pvc-6c7cfe80-0134-4573-b93e-9b259c6fcd63   5Gi        RWO            Delete           Bound    default/storaged-data-nebula-storaged-2   local-nvme              3m40s
...

View the detailed information of the PV.

kubectl get pv pvc-290c15cc-a302-4463-a591-84b7217a6cd2 -o yaml

Output:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    local.pv.provisioner/selected-node: ip-192-168-77-60.ec2.internal
    nebula-graph.io/pod-name: nebula-storaged-0
    pv.kubernetes.io/provisioned-by: nebula-cloud.io/local-pv
  creationTimestamp: "2024-03-04T07:51:32Z"
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    app.kubernetes.io/cluster: nebula
    app.kubernetes.io/component: storaged
    app.kubernetes.io/managed-by: nebula-operator
    app.kubernetes.io/name: nebula-graph
  name: pvc-290c15cc-a302-4463-a591-84b7217a6cd2
  resourceVersion: "7932689"
  uid: 66c0a2d3-2914-43ad-93b5-6d84fb62acef
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: storaged-data-nebula-storaged-0
    namespace: default
    resourceVersion: "7932688"
    uid: 8ecb5d96-004b-4672-bac4-1355ae15eae4
  local:
    fsType: ext4
    path: /mnt/disks/raid0
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ip-192-168-77-60.ec2.internal
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-nvme
  volumeMode: Filesystem
status:
  phase: Bound

Failover for Local Persistent Volumes in the cloud¶

When using network storage (e.g., AWS EBS, Google Cloud Persistent Disk, Azure Disk Storage, Ceph, NFS, etc.) as a PV, the storage resource is independent of any particular node. Therefore, the storage resource can be mounted and used by Pods regardless of the node to which the Pods are scheduled. However, when using a local storage disk as a PV, the storage resource can only be used by Pods on a specific node due to nodeAffinity.

The Storage service of NebulaGraph supports data redundancy, which allows you to set multiple odd-numbered partition replicas. When a node fails, the associated partition is automatically transferred to a healthy node. However, Storage Pods using Local Persistent Volumes cannot run on other nodes due to the node affinity setting and must wait for the node to recover. To run on another node, the Pods must be unbound from the associated Local Persistent Volume.

NebulaGraph Operator supports automatic failover in the event of a node failure while using Local Persistent Volumes in the cloud for elastic scaling. This is achieved by setting spec.enableAutoFailover to true in the cluster configuration file, which automatically unbinds the Pods from the Local Persistent Volume, allowing the Pods to run on another node.

Example configuration:

...
spec:
  # Enable automatic failover for Local PV.
  enableAutoFailover: true
  # The time to wait for the Storage service to be in the `OFFLINE` status
  # before automatic failover. 
  # The default value is 5 minutes.
  # If the Storage service recovers to the `ONLINE` status during this period,
  # failover will not be triggered.
  failoverPeriod: "2m"
  ...

Last update: March 6, 2024