Skip to content

Backup and restore data using NebulaGraph Operator

This article introduces how to back up and restore data of the NebulaGraph cluster on Kubernetes.

Enterpriseonly

This feature is only for the enterprise edition NebulaGraph clusters on Kubernetes.

Overview

NebulaGraph BR (Enterprise Edition) is a command line tool for data backup and recovery of NebulaGraph enterprise edition. NebulaGraph Operator is based on the BR tool to achieve data backup and recovery for NebulaGraph clusters on Kubernetes.

When backing up data, NebulaGraph Operator creates a Job to back up the data in the NebulaGraph cluster to the specified storage service.

When restoring data, NebulaGraph Operator checks the specified backup NebulaGraph cluster for existence, and whether the access to remote storage is normally based on the information defined in the NebulaRestore resource object. It then creates a new cluster and restores the backup data to the new NebulaGraph cluster. For more information, see restore flowchart.

Prerequisites

To backup and restore data using NebulaGraph Operator, the following conditions must be met:

  • Nebula Operator version >= 1.4.0.
  • The enterprise edition NebulaGraph cluster deployed on Kubernetes is running.
  • In the YAML file used to create the cluster, spec.enableBR is set to true.

    // Partial content of a sample cluster YAML file.
    apiVersion: apps.nebula-graph.io/v1alpha1
    kind: NebulaCluster
    metadata:
      name: nebula
    spec:  
    enableBR: true // Set to true to enable the backup and restore function.
    ...
    
  • Only storage services that use the S3 protocol (such as AWS S3, Minio, etc.) can be used to back up and restore data.
  • Sufficient computing resources are available in the cluster to restore data.

Backup

Notes

  • NebulaGraph Operator supports full and incremental backups.
  • During data backup, DDL and DML statements in the specified graph space will be blocked. We recommend performing the operation during off-peak hours, such as from 2:00 am to 5:00 am.
  • The cluster executing incremental backups and the cluster specified for the last backup must be the same, and the (storage bucket) path for the last backup must be the same.
  • Ensure that the time between each incremental backup and the last backup is less than a wal_ttl.
  • Specifying the backup data of a specified graph space is not supported.
  • Before backing up data, you need to create a Secret to restore the credential for pulling the image of the BR-ent tool.

    kubectl - <nebula> create secret docker-registry <br-ent-secret> \
    --docker-server=REGISTRY_SERVER \
    --docker-username=REGISTRY_USERNAME \
    --docker-password=REGISTRY_PASSWORD \
    
    • <nebula>: The namespace where the Secret is located.
    • <br-ent-secret>: The name of the Secret.
    • REGISTRY_SERVER: The address of the private image repository server from which the image is pulled, for example, reg.example-inc.com.
    • REGISTRY_USERNAME: The username for logging in to the private image repository server.
    • REGISTRY_PASSWORD: The password for logging in to the private image repository server.

Full backup

When backing up data to a storage service compatible with the S3 protocol, you need to create a backup Job, which will back up the full NebulaGraph data to the specified storage location.

Here is an example of the YAML file for a full backup Job:

apiVersion: batch/v1
kind: Job
metadata:
  name: nebula-full-backup
spec:
  parallelism: 1
  ttlSecondsAfterFinished: 600
  template:
    spec:
      restartPolicy: OnFailure
      imagePullSecrets:
        - name: br-ent-secret  # The name of the Secret for pulling the image of the BR-ent tool.
      containers:
        - image: reg.vesoft-inc.com/cloud-dev/br-ent:v3.5.1
          imagePullPolicy: Always
          name: backup
          command:
            - /bin/sh
            - -ecx
            - 'exec /usr/local/bin/br-ent backup full 
            --meta nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559   # The address of the Metad service. 
            --storage s3://BUCKET                           # The storage location of the backup file.
            --s3.access_key ACCESS_KEY                      # The AccessKey for accessing the S3 protocol-compatible storage service.
            --s3.secret_key SECRET_KEY                      # The SecretKey for accessing the S3 protocol-compatible storage service.
            --s3.region REGION                              # The region of the S3 protocol-compatible storage service.
            --s3.endpoint https://s3.REGION.amazonaws.com'  # The endpoint of the S3 protocol-compatible storage service.

Incremental backup

Except for the name of the Job and the command specified in spec.template.spec.containers[0].command, the YAML file for incremental backup is the same as that for a full backup. Here is an example of the YAML file for incremental backup:

apiVersion: batch/v1
kind: Job
metadata:
  name: nebula-incr-backup
spec:
  parallelism: 1
  ttlSecondsAfterFinished: 60
  template:
    spec:
      restartPolicy: OnFailure
      imagePullSecrets:
        - name: br-ent-secret
      containers:
        - image: reg.vesoft-inc.com/cloud-dev/br-ent:v3.5.1
          imagePullPolicy: Always
          name: backup
          command:
            - /bin/sh
            - -ecx
            - 'exec /usr/local/bin/br-ent backup incr
            --meta nebula-metad-0.nebula-metad-headless.nebula.svc.cluster.local:9559 # The address of the Metad service. 
            --storage s3://BUCKET                           # The storage location of the backup file.
            --s3.access_key ACCESS_KEY                      # The AccessKey for accessing the S3 protocol-compatible storage service.
            --s3.secret_key SECRET_KEY                      # The SecretKey for accessing the S3 protocol-compatible storage service.
            --s3.region REGION                              # The region of the S3 protocol-compatible storage service.
            --s3.endpoint https://s3.REGION.amazonaws.com'  # The endpoint of the S3 protocol-compatible storage service.

Parameter description

The main parameters are described as follows:

Parameter Default value Description
spec.parallelism 1 The number of tasks executed in parallel.
spec.ttlSecondsAfterFinished 60 The time to keep task information after the task is completed.
spec.template.spec.containers[0].image vesoft/br-ent:3.5.1 The image address of the NebulaGraph BR Enterprise Edition tool.
spec.template.spec.containers[0].command - The command for backing up data to the storage service compatible with the S3 protocol.
For descriptions of the options in the command, see Parametr description.

For more settings of the Job, see Kubernetes Jobs.

After the YAML file for the backup Job is set, run the following command to start the backup Job:

kubectl apply -f <backup_file_name>.yaml

When the data backup succeeds, a backup file is generated in the specified storage location. For example, the backup file name is BACKUP_2023_02_12_10_04_16.

Restore

Notes

  • After the data recovery is successful, a new cluster will be created and the old cluster will not be deleted by default. You can decide whether to delete the old cluster themselves. The name of the new cluster is automatically generated by the Operator.
  • There will be a period of service unavailability during the data recovery process, so it is recommended to perform the operation during a low period of business activity.

Process

When restoring data from a compatible S3 protocol service, you need to create a Secret to store the credentials for accessing the compatible S3 protocol service. Then create a resource object (NebulaRestore) for restoring the data, which will instruct the Operator to create a new NebulaGraph cluster based on the information defined in this resource object and restore the backup data to the newly created cluster.

Here is an example YAML for restoring data based on the backup file BACKUP_2023_02_12_10_04_16:

apiVersion: v1
kind: Secret
metadata:
  name: aws-s3-secret
type: Opaque
data:
  access-key: QVNJQVE0WFlxxx
  secret-key: ZFJ6OEdNcDdxenMwVGxxx
---
apiVersion: apps.nebula-graph.io/v1alpha1
kind: NebulaRestore
metadata:
  name: restore1
spec:
  br:
    clusterName: nebula
    backupName: "BACKUP_2023_02_12_10_04_16"
    concurrency: 5
    s3:
      region: "us-west-2"
      bucket: "nebula-br-test"
      endpoint: "https://s3.us-west-2.amazonaws.com"
      secretName: "aws-s3-secret"

Parameter Description

  • Secret

    Parameter Default Value Description
    metadata.name - The name of the Secret.
    type Opaque The type of the Secret. See Types of Secret for more information.
    data.access-key - The AccessKey for accessing the S3 protocol-compatible storage service.
    data.secret-key - The SecretKey for accessing the S3 protocol-compatible storage service.
  • NebulaRestore

    Parameter Default Value Description
    metadata.name - The name of the resource object NebulaRestore.
    spec.br.clusterName - The name of the backup cluster. Backup based on this cluster, then create a new cluster with a name automatically generated.
    spec.br.backupName - The name of the backup file. Restore data based on this backup file.
    spec.br.concurrency 5 The number of concurrent downloads when restoring data. The default value is 5.
    spec.br.s3.region - The geographical region where the S3 storage bucket is located.
    spec.br.s3.bucket - The path of the S3 storage bucket where backup data is stored.
    spec.br.s3.endpoint - The access address of the S3 storage bucket.
    spec.br.s3.secretName - The name of the Secret that is used to access the S3 storage bucket.

After setting up the YAML file for restoring the data, run the following command to start the restore job:

kubectl apply -f <restore_file_name>.yaml

Run the following command to check the status of the NebulaRestore object.

kubectl get rt <NebulaRestore_name> -n <namespace>

# Output example:
NAME       STATUS     STARTED   COMPLETED   AGE
restore1   Complete   67m       59m         67m

After the restore job is completed, a new NebulaGraph cluster is created with the name automatically generated by the Operator. To check the status of the new cluster:

kubectl get nc -n <namespace>

# Output example:

NAME     GRAPHD-DESIRED   GRAPHD-READY   METAD-DESIRED   METAD-READY   STORAGED-DESIRED   STORAGED-READY   AGE
nebula   1                1              1               1             3                  3                2d3h
ngxvsm   1                1              1               1             3                  3                92m  # The newly created cluster.

Last update: July 27, 2023