Backup with mTLS enabled¶

For a NebulaGraph cluster with mTLS enabled, NebulaGraph Operator supports backup and restore operations on the data. This topic describes how to back up data from a NebulaGraph cluster with mTLS enabled to Google Cloud Storage (GCS) and S3-compatible cloud storage services (such as AWS S3 and Minio) in a K8s environment.

Cautions¶

During the backup operation, DDL and DML statements in the specified graph space are blocked. We recommend that you perform operations during off-peak business hours, such as between 2 a.m. and 5 a.m.
The cluster performing the incremental backup and the cluster specified for the previous backup must be the same, and the storage bucket specified must be the same as the previous backup.
Ensure that the time between each incremental backup and the previous backup is less than one wal_ttl duration.
The backup of specified graph space data is not supported.

Prerequisites¶

To perform data backup with mTLS enabled using NebulaGraph Operator, the following conditions must be met:

NebulaGraph Operator version is 1.8.0 or higher.
A NebulaGraph cluster is running on Kubernetes.
mTLS authentication is enabled in the NebulaGraph cluster. For more information, see Enable TLS encryption.
Access credentials for Google Cloud Storage (GCS) or cloud storage services compatible with the S3 protocol are prepared for data backup.

Steps¶

If mTLS authentication is enabled for the NebulaGraph cluster, you need to provide certificates to the NebulaGraph Backup&Restore tool to access the cluster. The following example is a full backup process with mTLS enabled and all created resource objects in the default namespace default.

Execute kubectl edit nc <cluster_name> to edit the cluster's YAML file.
- <cluster_name> is the name of your cluster.

Set spec.enableBR to true and configure the certificate for Agent by spec.agent.env to enable mTLS encrypted backup. For information about complete cluster configuration, see Enable mTLS in NebulaGraph.

Partial configuration of the cluster

apiVersion: apps.nebula-graph.io/v1alpha1
kind: NebulaCluster
metadata:
  name: nebula
spec:  
  enableBR: true # Set to true to enable the backup and recovery feature. The default value is false.
  agent:         # A component used for backup and recovery.
    env:         # Configure the certificate for Agent.
    - name: CA_CERT_PATH
      value: /usr/local/certs/root.crt
    - name: CLIENT_CERT_PATH
      value: /usr/local/certs/client.crt
    - name: CLIENT_KEY_PATH
      value: /usr/local/certs/client.key
    image: reg.vesoft-inc.com/cloud-dev/nebula-agent # Agent image address. The default value is vesoft/nebula-agent.
    version: latest                                  # Agent image version. The default value is latest.
    resources:                  
      requests:
        cpu: "100m"                                  # Minimum CPU usage.
        memory: "128Mi"                              # Minimum memory usage.
      limits:
        cpu: "1"                                     # Maximum CPU usage.
        memory: "1Gi"                                # Maximum memory usage.
  # Limit the speed of file upload and download, in Mbps. The default value is 0, indicating no limit.
  # rateLimit: 0
  # The connection timeout between the Agent and metad, in seconds. The default value is 60.
  # heartbeatInterval: 60
    volumeMounts:                                    # Mount path for the certificate.
    - mountPath: /usr/local/certs
      name: credentials   
...

Create a Secret for pulling the NebulaGraph Backup&Restore image from the private repository.
```
kubectl -n <nebula> create secret docker-registry <br_secret_name> \
--docker-server=<registry_server> \
--docker-username=<registry_username> \
--docker-password=<registry_password> \
```
- <nebula>: The namespace where the Secret is stored. Ensure that the namespace is the same as the namespace of the NebulaBackup object. This example does not set the namespace, so the default namespace default is used.
- <br_secret_name>: Name of the Secret.
- <registry_server>: The private repository server address for pulling the image, such as reg.example-inc.com.
- <registry_username>: The image repository username.
- <registry_password>: The image repository password.

Create a YAML file for the backup with mTLS enabled, for example, backup_file_name.yaml.

The YAML file example for an incremental backup with mTLS encryption is similar to that of a full backup with mTLS encryption. The difference is that you need to add the baseBackupName parameter in the spec.config of the incremental backup YAML file to specify the name of the previous backup.

Example YAML content for backup to GCS Example YAML content for backup to S3-compatible storage service

apiVersion: v1
kind: Secret                                                       
metadata:
  name: gcs-secret                                     # Name of the Secret for the credential to access GCS storage service.
type: Opaque
data:
  credentials: <GOOGLE_APPLICATION_CREDENTIALS_JSON>   # JSON string representing Google service account key.
---
apiVersion: apps.nebula-graph.io/v1alpha1
kind: NebulaBackup
metadata:
  name: mtls-backup
spec:
  image: reg.vesoft-inc.com/cloud-dev/br-ent           # Image URL of the NebulaGraph Backup&Restore tool.
  version: v3.7.1                              # Version of the NebulaGraph Backup&Restore tool.
  env:                                                 # Environment variables to obtain the certificate path for NebulaGraph Backup&Restore.
  - name: CA_CERT_PATH
    value: /usr/local/certs/root.crt
  - name: CLIENT_CERT_PATH
    value: /usr/local/certs/client.crt
  - name: CLIENT_KEY_PATH
    value: /usr/local/certs/client.key
  resources:
    limits:
      cpu: "1"
      memory: 300Mi
    requests:
      cpu: 100m
      memory: 200Mi
  imagePullSecrets:                                    # Name of the Secret for the credential to pull the NebulaGraph Backup&Restore image from a private repository.
  - name: br_secret_name
  initContainers:                                      # The initialization container responsible for generating the certificate.
  - name: init-auth-sidecar
    command:
    - /bin/sh
    - -c
    args:
    - cp -r /certs/* /credentials/
    imagePullPolicy: Always
    image: reg.vesoft-inc.com/cloud-dev/nebula-certs:latest
    volumeMounts:
    - name: credentials
      mountPath: /credentials
  volumes:                                             # The volume to obtain the certificate.
  - name: credentials           
    emptyDir:
      medium: Memory
  volumeMounts:                                        # The mount path of the certificate in the container.
  - name: credentials
    mountPath: /usr/local/certs
  autoRemoveFinished: false                            # Whether to automatically delete the Job after the backup job is completed or failed. The default value is false.
  cleanBackupData: false                               # Whether to delete the backup data in the cloud storage service when deleting the backup object. The default value is false.
  config:   
    concurrency: 5                                     # The concurrency of the backup jobs. The default value is 5.                                        
    clusterName: nebula                                # Name of the cluster to back up.       
    gs:                                                # Configuration information for the GCS storage service.
      location: "us-central1"                          # Geographical region of the GCS storage bucket.
      bucket: "ng-2025"                                # Name of the GCS storage bucket to store the backup data.
      secretName: "gcs-secret"                         # Name of the Secret for the credential to access the GCS storage bucket.

apiVersion: v1
kind: Secret                                   
metadata:
  name: aws-s3-secret                            # Name of the Secret for the credential to access the S3-compatible storage service.
type: Opaque
data:                                          
  access_key: QVNJQVE0WFlxxx                     # AccessKey for the S3-compatible storage service.
  secret_key: ZFJ6OEdNcDdxenMwVGxxx              # SecretKey for the S3-compatible storage service.
---
apiVersion: apps.nebula-graph.io/v1alpha1
kind: NebulaBackup
metadata:
  name: mtls-backup
spec:
  image: reg.vesoft-inc.com/cloud-dev/br-ent           # Image URL of the NebulaGraph Backup&Restore tool.
  version: v3.7.1                              # Version of the NebulaGraph Backup&Restore tool.
  env:                                                 # Environment variables to obtain the certificate path for NebulaGraph Backup&Restore.
  - name: CA_CERT_PATH
    value: /usr/local/certs/root.crt
  - name: CLIENT_CERT_PATH
    value: /usr/local/certs/client.crt
  - name: CLIENT_KEY_PATH
    value: /usr/local/certs/client.key
  resources:
    limits:
      cpu: "1"
      memory: 300Mi
    requests:
      cpu: 100m
      memory: 200Mi
  imagePullSecrets:                                    # Name of the Secret for the credential to pull the NebulaGraph Backup&Restore image from a private repository.
  - name: br_secret_name
  initContainers:                                      # The initialization container responsible for generating the certificate.
  - name: init-auth-sidecar
    command:
    - /bin/sh
    - -c
    args:
    - cp -r /certs/* /credentials/
    imagePullPolicy: Always
    image: reg.vesoft-inc.com/cloud-dev/nebula-certs:latest
    volumeMounts:
    - name: credentials
      mountPath: /credentials
  volumes:                                             # The volume to obtain the certificate.
  - name: credentials           
    emptyDir:
      medium: Memory
  volumeMounts:                                        # The mount path of the certificate in the container.
  - name: credentials
    mountPath: /usr/local/certs
  autoRemoveFinished: false                             # Whether to automatically delete the Job after the backup job is completed or failed. The default value is false.
  cleanBackupData: false                                # Whether to delete the backup data in the cloud storage service when deleting the backup object. The default value is false.
  config:                    
    concurrency: 5                             # The concurrency of the backup jobs. The default value is 5.                          
    clusterName: nebula                        # Name of the cluster to back up.
    s3:                                        # Configuration information for the S3-compatible storage service.
      region: "us-east-2"                      # Geographical region of the S3 storage bucket.
      bucket: "nebula-test"                    # Name of the S3 storage bucket to store the backup data.
      endpoint: "https://s3.us-east-2.amazonaws.com"  # Access address of the S3 storage bucket.
      secretName: "aws-s3-secret"              # Name of the Secret for the credential to access the S3 storage bucket.

In addition to the parameters in the above YAML files, the following parameters can also be set:

Parameter	Default Value	Description
`spec.imagePullPolicy`	`Always`	Image pull policy for the backup job.
`spec.nodeSelector`	`{}`	Node selector for the backup job.
`spec.affinity`	`{}`	Affinity settings for the backup job.
`spec.tolerations`	`[]`	Toleration settings for the backup job.
`spec.config.clusterNamespace`	`default`	Namespace where the cluster to be backed up is located. Cross-namespace cluster backup is supported, which means the namespace of the cluster may be different from that of the `NebulaBackup` object.

Start the backup job.
```
kubectl create -f backup_file_name.yaml
```
Upon initiating a backup job, the system automatically creates a backup instance nb, which in turn creates a Job object. This Job object is responsible for creating and managing a Pod object. The Pod object runs the NebulaGraph Backup&Restore tool to execute the backup job. If the spec.autoRemoveFinished configuration of the NebulaBackup resource object is set to true, the system automatically deletes the Job object upon completion of the backup job.

View the status of the backup object.

View the status of the NebulaBackup resource object.

kubectl get nb

Output:

NAME          TYPE   BACKUP                       STATUS     STARTED   COMPLETED   AGE
mtls-backup   full   BACKUP_2024_02_05_08_05_13   Complete   71s       1s          71s

When spec.autoRemoveFinished is set to false, you can execute the following commands to check the status of the Job and Pod objects.

View the status of the Job object.

kubectl get job

Output:

NAME                     COMPLETIONS   DURATION   AGE
backup-full-nb202402v5   1/1           96s        96s

View the status of the Pod object.

kubectl get pod

Output:

NAME                                 READY   STATUS    RESTARTS   AGE
backup-full-nb202402v5-67884k        1/1     Running   0          109s

Last update: March 6, 2024