Enable mTLS in NebulaGraph¶

Transport Layer Security (TLS) is an encryption protocol in wide use on the Internet. TLS, which was formerly called SSL, authenticates the server in a client-server connection and encrypts communications between client and server so that external parties cannot spy on the communications. Its working principle is mainly to protect data transmitted over the network by using encryption algorithms to prevent data interception or tampering during transmission. During the TLS connection establishment process, the server sends a digital certificate containing a public key and some identity information to the client. This certificate is issued by a trusted third-party Certification Authority (CA). The client verifies this digital certificate to confirm the identity of the server.

In the NebulaGraph environment running in Kubernetes, mutual TLS (mTLS) is used to encrypt the communication between the client and server by default, which means both the client and server need to verify their identities. This article explains how to enable mTLS encryption in NebulaGraph running in K8s.

Prerequisites¶

NebulaGraph Operator is installed.
A NebulaGraph cluster is created. For details, see Create a NebulaGraph cluster .
Certificates and their corresponding private keys have been generated for the client and server, and the CA certificate has been generated. For details, see Generate Certificates Manually.

Note

In the cluster created using Operator, the client and server use the same CA root certificate by default.

Encryption scenarios¶

The following two scenarios are commonly used for encryption:

Encrypting communication between the client and the Graph service.

Encrypting communication between services, such as communication between the Graph service and the Meta service, communication between the Graph service and the Storage service, and communication between the Meta service and the Storage service.
Note
- The Graph service in NebulaGraph is the entry point for all client requests. The Graph service communicates with the Meta service and the Storage service to complete the client requests. Therefore, the Graph service needs to be able to communicate with the Meta service and the Storage service.
- The Storage and Meta services in NebulaGraph communicate with each other through heartbeat messages to ensure their availability and health. Therefore, the Storage service needs to be able to communicate with the Meta service and vice versa.

For all encryption scenarios, see Authentication policies.

mTLS with certificate hot-reloading¶

NebulaGraph Operator supports enabling mTLS with certificate hot-reloading.

Examples¶

The following provides examples of the cluster configuration file to enable mTLS between the client and the Graph service, and between services.

View sample configurations of mTLS between services

apiVersion: apps.nebula-graph.io/v1alpha1
kind: NebulaCluster
metadata:
  name: nebula
spec:
  # # Required to enable backup and restore.
  # enableBR: true
  # # Customize Agent configurations for backup and restore.
  # agent:
  #   # Configure environment variables for the Agent component to obtain certificates.
  #   env:
  #   - name: CA_CERT_PATH
  #     value: /usr/local/certs/root.crt
  #   - name: CLIENT_CERT_PATH
  #     value: /usr/local/certs/client.crt
  #   - name: CLIENT_KEY_PATH
  #     value: /usr/local/certs/client.key
  #   # Customize the Agent image and version.
  #   image: reg.vesoft-inc.com/cloud-dev/nebula-agent
  #   resources: {}
  #   version: snap
  #   # Limit the speed of file upload and download, in Mbps. The default value is 0, indicating no limit.
  #   rateLimit: 0
  #   # The connection timeout between the Agent and metad, in seconds. The default value is 60.
  #   heartbeatInterval: 60
  #   volumeMounts:
  #   - mountPath: /usr/local/certs
  #     name: credentials
  # # Used to obtain the zone information about the node.
  # alpineImage: "reg.vesoft-inc.com/vesoft/nebula-alpine:latest"
  exporter:
    image: vesoft/nebula-stats-exporter
    replicas: 1
    maxRequests: 20
  # The certificate files for NebulaGraph Operator to access Storage and Meta services.
  sslCerts:
    # Certificates can be passed in by creating a Secret or 
    # by configuring environment variables in the Operator chart configuration file.
    # The following fields are example configurations for passing in certificates by creating a Secret.
    # clientSecret: "client-cert"
    # clientCert: "tls.crt"
    # clientKey: "tls.key"
    # caSecret: "ca-cert" 
    # caCert: "tls.crt"
    insecureSkipVerify: false # Whether the client verifies the server's certificate chain and hostname. Default is false.
    # If `insecureSkipVerify` is set to `false`,
    # either specify the server's hostname or IP of the server in the `subjectAltName` field of the server's certificate, 
    # or specify the Subject Alternative Name (SAN) of the server in the following `serverName` field.
    serverName: managed-graph-http2.nebula # The SAN of the server.
  graphd:
    config:
      # The following fields are used to enable mTLS between services.
      ca_client_path: certs/root.crt
      ca_path: certs/root.crt
      cert_path: certs/server.crt
      key_path: certs/server.key
      enable_meta_ssl: "true"
      enable_storage_ssl: "true" 
      # The following fields are required for creating a cluster with zones.
      # accept_partial_success: "true"
      # prioritize_intra_zone_reading: "true"
      # sync_meta_when_use_space: "true"
      # stick_to_intra_zone_on_failure: "false" 
      # session_reclaim_interval_secs: "300"
      # The SAN of the server, which is used by other services to verify the server's hostname.
      ssl_server_SAN: managed-graph-http2.nebula 
      # The parent directory of the SSL certificate files that are automatically reloaded.
      ssl_watch_path: certs 
    initContainers:
      - name: init-auth-sidecar
        command:
          - /bin/sh
          - -c
        args:
          - cp /certs/* /credentials/
        imagePullPolicy: Always
        image: reg.vesoft-inc.com/xxx/xxx:latest
        volumeMounts:
          - name: credentials
            mountPath: /credentials
    sidecarContainers:
      - name: auth-sidecar
        imagePullPolicy: Always
        image: reg.vesoft-inc.com/xxx/xxx:latest
        volumeMounts:
          - name: credentials
            mountPath: /credentials
    volumes:
      - name: credentials
        emptyDir:
          medium: Memory
    volumeMounts:
      - name: credentials
        mountPath: /usr/local/nebula/certs
    logVolumeClaim:
      resources:
        requests:
          storage: 1Gi
      storageClassName: local-path
    resources:
      requests:
        cpu: "200m"
        memory: "500Mi"
      limits:
        cpu: "1"
        memory: "1Gi"
    replicas: 1
    image: reg.vesoft-inc.com/xxx/xxx
    version: v3.5.0-sc
  metad:
    config:
      # Zone names CANNOT be modified once set.
      # It's suggested to set an odd number of zones.
      # zone_list: az1,az2,az3 
      # Whether to enable the timestamp validation of the session.
      validate_session_timestamp: "true"
      # The following fields are used to enable mTLS between services.
      ca_client_path: certs/root.crt
      ca_path: certs/root.crt
      cert_path: certs/server.crt
      key_path: certs/server.key
      enable_meta_ssl: "true"
      enable_storage_ssl: "true"  
      # The SAN of the server, which is used by other services to verify the server's hostname.
      ssl_server_SAN: managed-graph-http2.nebula
      # The parent directory of the SSL certificate files that are automatically reloaded.
      ssl_watch_path: certs
    initContainers:
      - name: init-auth-sidecar
        command:
          - /bin/sh
          - -c
        args:
          - cp /certs/* /credentials/
        imagePullPolicy: Always
        image: reg.vesoft-inc.com/xxx/xxx:latest
        volumeMounts:
          - name: credentials
            mountPath: /credentials
    sidecarContainers:
      - name: auth-sidecar
        imagePullPolicy: Always
        image: reg.vesoft-inc.com/xxx/xxx:latest
        volumeMounts:
          - name: credentials
            mountPath: /credentials
    volumes:
      - name: credentials
        emptyDir:
          medium: Memory
    volumeMounts:
      - name: credentials
        mountPath: /usr/local/nebula/certs
    licenseManagerURL: "192.168.8.xx:9119"
    resources:
      requests:
        cpu: "300m"
        memory: "500Mi"
      limits:
        cpu: "1"
        memory: "1Gi"
    replicas: 1
    image: reg.vesoft-inc.com/xxx/xxx
    version: v3.5.0-sc
    dataVolumeClaim:
      resources:
        requests:
          storage: 2Gi
      storageClassName: local-path
    logVolumeClaim:
      resources:
        requests:
          storage: 1Gi
      storageClassName: local-path
  storaged:
    config:
      # The following fields are used to enable mTLS between services.
      ca_client_path: certs/root.crt
      ca_path: certs/root.crt
      cert_path: certs/server.crt
      key_path: certs/server.key
      enable_meta_ssl: "true"
      enable_storage_ssl: "true"  
      # The SAN of the server, which is used by other services to verify the server's hostname.
      ssl_server_SAN: managed-graph-http2.nebula
      # The parent directory of the SSL certificate files that are automatically reloaded.
      ssl_watch_path: certs
    initContainers:
      - name: init-auth-sidecar
        command:
          - /bin/sh
          - -c
        args:
          - cp /certs/* /credentials/
        imagePullPolicy: Always
        image: reg.vesoft-inc.com/xxx/xxx:latest
        volumeMounts:
          - name: credentials
            mountPath: /credentials
    sidecarContainers:
      - name: auth-sidecar
        imagePullPolicy: Always
        image: reg.vesoft-inc.com/xxx/xxx:latest
        volumeMounts:
          - name: credentials
            mountPath: /credentials
    volumes:
      - name: credentials
        emptyDir:
          medium: Memory
    volumeMounts:
      - name: credentials
        mountPath: /usr/local/nebula/certs
    resources:
      requests:
        cpu: "300m"
        memory: "500Mi"
      limits:
        cpu: "1"
        memory: "1Gi"
    replicas: 1
    image: reg.vesoft-inc.com/xxx/xxx
    version: v3.5.0-sc
    dataVolumeClaims:
      - resources:
          requests:
            storage: 2Gi
        storageClassName: local-path
    logVolumeClaim:
      resources:
        requests:
          storage: 1Gi
      storageClassName: local-path
    # Automatically balance storage data after scaling out. 
    # enableAutoBalance: true
  reference:
    name: statefulsets.apps
    version: v1
  schedulerName: nebula-scheduler
  imagePullPolicy: Always
  imagePullSecrets:
    - name: nebula-image
  # Whether to automatically delete PVCs when deleting a cluster.
  # enablePVReclaim: true
  # Used to evenly distribute Pods across zones.
  # topologySpreadConstraints:
  # - topologyKey: "kubernetes.io/zone"
  #   whenUnsatisfiable: "DoNotSchedule"

View sample configurations of mTLS between the client and the Graph service

apiVersion: apps.nebula-graph.io/v1alpha1
kind: NebulaCluster
metadata:
  name: nebula
spec:
  # # Required to enable backup and restore.
  # enableBR: true
  # # Customize Agent configurations for backup and restore.
  # agent:
  #   # Configure environment variables for the Agent component to obtain certificates.
  #   env:
  #   - name: CA_CERT_PATH
  #     value: /usr/local/certs/root.crt
  #   - name: CLIENT_CERT_PATH
  #     value: /usr/local/certs/client.crt
  #   - name: CLIENT_KEY_PATH
  #     value: /usr/local/certs/client.key
  #   # Customize the Agent image and version.
  #   image: reg.vesoft-inc.com/cloud-dev/nebula-agent
  #   resources: {}
  #   version: snap
  #   # Limit the speed of file upload and download, in Mbps. The default value is 0, indicating no limit.
  #   rateLimit: 0
  #   # The connection timeout between the Agent and metad, in seconds. The default value is 60.
  #   heartbeatInterval: 60
  #   volumeMounts:
  #   - mountPath: /usr/local/certs
  #     name: credentials
  # # Used to obtain the zone information about the node.    
  # alpineImage: "reg.vesoft-inc.com/vesoft/nebula-alpine:latest"    
  exporter:
    image: vesoft/nebula-stats-exporter
    replicas: 1
    maxRequests: 20
  graphd:
    config:
      # The following fields are used to enable mTLS between the client and the Graph service.
      ca_client_path: certs/root.crt
      ca_path: certs/root.crt
      cert_path: certs/server.crt
      key_path: certs/server.key
      enable_graph_ssl: "true"
      # The following fields are required for creating a cluster with zones.
      # accept_partial_success: "true"
      # prioritize_intra_zone_reading: "true"
      # sync_meta_when_use_space: "true"
      # stick_to_intra_zone_on_failure: "false" 
      # session_reclaim_interval_secs: "300"
      # The parent directory of the SSL certificate files that are automatically reloaded.
      ssl_watch_path: certs
    initContainers:
      - name: init-auth-sidecar
        command:
          - /bin/sh
          - -c
        args:
          - cp /certs/* /credentials/
        imagePullPolicy: Always
        image: reg.vesoft-inc.com/xxx/xxx:latest
        volumeMounts:
          - name: credentials
            mountPath: /credentials
    sidecarContainers:
      - name: auth-sidecar
        imagePullPolicy: Always
        image: reg.vesoft-inc.com/xxx/xxx:latest
        volumeMounts:
          - name: credentials
            mountPath: /credentials
    volumes:
      - name: credentials
        emptyDir:
          medium: Memory
    volumeMounts:
      - name: credentials
        mountPath: /usr/local/nebula/certs
    logVolumeClaim:
      resources:
        requests:
          storage: 1Gi
      storageClassName: local-path
    resources:
      requests:
        cpu: "200m"
        memory: "500Mi"
      limits:
        cpu: "1"
        memory: "1Gi"
    replicas: 1
    image: reg.vesoft-inc.com/xxx/xxx
    version: v3.5.0-sc
  metad:
    # Zone names CANNOT be modified once set.
    # It's suggested to set an odd number of zones.
    # zone_list: az1,az2,az3 
    # Whether to enable the timestamp validation of the session.
    validate_session_timestamp: "true"
    licenseManagerURL: "192.168.8.xxx:9119"
    resources:
      requests:
        cpu: "300m"
        memory: "500Mi"
      limits:
        cpu: "1"
        memory: "1Gi"
    replicas: 1
    image: reg.vesoft-inc.com/xxx/xxx
    version: v3.5.0-sc
    dataVolumeClaim:
      resources:
        requests:
          storage: 2Gi
      storageClassName: local-path
    logVolumeClaim:
      resources:
        requests:
          storage: 1Gi
      storageClassName: local-path
  storaged:
    resources:
      requests:
        cpu: "300m"
        memory: "500Mi"
      limits:
        cpu: "1"
        memory: "1Gi"
    replicas: 1
    image: reg.vesoft-inc.com/xxx/xxx
    version: v3.5.0-sc
    dataVolumeClaims:
      - resources:
          requests:
            storage: 2Gi
        storageClassName: local-path
    logVolumeClaim:
      resources:
        requests:
          storage: 1Gi
      storageClassName: local-path
    enableAutoBalance: true
  reference:
    name: statefulsets.apps
    version: v1
  schedulerName: nebula-scheduler
  imagePullPolicy: Always
  imagePullSecrets:
    - name: nebula-image
  enablePVReclaim: true
  # Used to evenly distribute Pods across zones.
  # topologySpreadConstraints:
  # - topologyKey: "kubernetes.io/zone"
  #   whenUnsatisfiable: "DoNotSchedule"

Must-have fields¶

The following only lists the fields that must be configured to enable mTLS with certificate hot-reloading. For details about other fields, see the above examples.

mTLS between servicesmTLS between the client and the Graph service

`spec.sslCerts`¶

The spec.sslCerts field specifies the encryption certificates for NebulaGraph Operator and nebula-agent (if you do not use the default nebula-agent image). As NebulaGraph Operator needs to communicate with the Meta and Storage services, you must configure spec.sslCerts in the cluster configuration file to set certificates for it when enabling mTLS between services.

You can configure the environment variables of NebulaGraph Operator in the nebula-operator chart configuration file to obtain local certificates or create a Secret and specify it in the cluster configuration file to pass in the certificates. Note that when environment variables are used, the secrets under the spec.sslCerts section are ignored.

To obtain certificates locally through environment variables, first update the controllerManager.env value to set the certificate environment variables for NebulaGraph Operator in the chart configuration file of NebulaGraph Operator.

The nebula-operator chart allows configuration of the init container, sidecar container, volumes, and volumeMounts within the controller-manager Pod for automatically generating and updating certificates for NebulaGraph Operator. It accesses the certificates generated by the sidecar-container through the environment variables CA_CERT_PATH, CLIENT_CERT_PATH, and CLIENT_KEY_PATH. To generate certificates, you need to place the sidecar-container and the controller-manager container in the same Pod and share the certificate directory credentials. The controller-manager then reads the client-side certificates from the environment variables and establishes a mutual TLS connection with the Meta and Storage services. Example:

Note

When the following certificate environment variables are configured, the Secrets under spec.sslCerts in the cluster configuration file are ignored.
nebula-operator_chart_configuration_file.yaml
```
...
controllerManager:
  env:
  - name: CA_CERT_PATH
    value: /credentials/root.crt  # Must use absolute paths.
  - name: CLIENT_CERT_PATH
    value: /credentials/client.crt # Must use absolute paths.
  - name: CLIENT_KEY_PATH
    value: /credentials/client.key # Must use absolute paths.

  extraInitContainers:
  - name: init-auth-sidecar
    command:
    - /bin/sh
    - -c
    args:
    - cp -R /certs/* /credentials/
    imagePullPolicy: Always
    image: reg.vesoft-inc.com/cloud-dev/nebula-certs:latest
    volumeMounts:
    - name: credentials
      mountPath: /credentials

  sidecarContainers:
    auth-sidecar:
      imagePullPolicy: Always
      image: reg.vesoft-inc.com/cloud-dev/nebula-certs:latest
      volumeMounts:
        - name: credentials
          mountPath: /credentials

  extraVolumes:
    - name: credentials
      emptyDir:
        medium: Memory

  extraVolumeMounts:
    - name: credentials
      mountPath: /credentials
```
Then, explicitly specify the spec.sslCerts field in the cluster configuration file. Example:
cluster_configuration_file.yaml
```
spec:
  sslCerts:
```

Alternatively, to pass in the certificates through a Secret, add the following fields under spec.sslCerts in the cluster configuration file.

Caution

When the certificate environment variables mentioned above are configured, the following secrets under spec.sslCerts are ignored.
cluster_configuration_file.yaml
```
spec:
  sslCerts:
    clientSecret: "client-cert" 
    clientCert: "tls.crt"  # The key name of the certificate in the client certificate Secret, default is "tls.crt".
    clientKey: "tls.key"   # The key name of the private key in the client certificate Secret, default is "tls.key".
    caSecret: "ca-cert" # The name of the CA certificate Secret.
    caCert: "tls.crt"  # The key name of the certificate in the CA certificate Secret, default is "ca.crt".
```
Ensure that the keys for the certificates and private keys in the created Secret match the values of the clientCert, clientKey, and caCert fields in spec.sslCerts. If you create a Secret of the TLS type, the default key names are tls.crt and tls.key for the certificate and private key. In this case, you need to change the value of the caCert field from ca.crt to tls.crt.

`spec.sslCerts.insecureSkipVerify`¶

The insecureSkipVerify field is used to determine whether the client verifies the server's certificate chain and hostname. The default value is false.

It is recommended that you use the default value in production environments. In this case, you need to either specify the hostname or IP of the server in the subjectAltName field of the server's certificate or specify the Subject Alternative Name (SAN) of the server in the spec.sslCerts.serverName field. Otherwise, an error occurs when the client verifies the server's certificate chain and hostname. For more information, see openssl.

Example:

cluster_configuration_file.yaml

spec:
  sslCerts:
    insecureSkipVerify: false
    serverName: managed-graph-http2.nebula

If you set the value of insecureSkipVerify to true, the client does not verify the server's certificate chain and hostname.

`spec.<meta|graph|storage>.config`¶

To enable mTLS between services (Graph, Meta, and Storage), add the following fields under the spec.metad.config, spec.graphd.config, and spec.storaged.config respectively in the cluster configuration file.

spec:
  graph:
    config:
      ca_client_path: certs/root.crt
      ca_path: certs/root.crt
      cert_path: certs/server.crt
      key_path: certs/server.key
      enable_meta_ssl: "true"
      enable_storage_ssl: "true"
      ssl_server_SAN: managed-graph-http2.nebula
      ssl_watch_path: certs
  metad:
    config:
      ca_client_path: certs/root.crt
      ca_path: certs/root.crt
      cert_path: certs/server.crt
      key_path: certs/server.key
      enable_meta_ssl: "true"
      enable_storage_ssl: "true"
      ssl_server_SAN: managed-graph-http2.nebula
      ssl_watch_path: certs
  storaged:
    config:
      ca_client_path: certs/root.crt
      ca_path: certs/root.crt
      cert_path: certs/server.crt
      key_path: certs/server.key
      enable_meta_ssl: "true"
      enable_storage_ssl: "true"
      ssl_server_SAN: managed-graph-http2.nebula
      ssl_watch_path: certs

`initContainers`, `sidecarContainers`, `volumes`, and `volumeMounts`¶

initContainers, sidecarContainers, volumes, and volumeMounts fields are essential for implementing mTLS certificate online hot-reloading.

For the encryption scenario where the Graph service, Meta service, and Storage service need to be encrypted, configure these fields under spec.graph.config, spec.storage.config, and spec.meta.config respectively.

Details about these fields are as follows:

initContainers

The initContainers field is utilized to configure an init-container responsible for generating certificate files. Note that the volumeMounts field specifies how a volume specified by volumes, shared with the NebulaGraph container, is mounted, providing read and write access.

In the following example, init-auth-sidecar performs the task of copying files from the certs directory within the image to /credentials. After this task is completed, the init-container exits.

Example:
```
initContainers:
- name: init-auth-sidecar
    command:
      - /bin/sh
      - -c
    args:
      - cp /certs/* /credentials/
    imagePullPolicy: Always
    image: reg.vesoft-inc.com/xxx/xxx:latest
    volumeMounts:
      - name: credentials
        mountPath: /credentials
```

sidecarContainers

The sidecarContainers field is dedicated to periodically monitoring the expiration time of certificates and, when they are near expiration, generating new certificates to replace the existing ones. This process ensures seamless online certificate hot-reloading without any service interruptions.

In the example provided, the auth-sidecar container employs the crond process, which runs a crontab script every minute. This script checks the certificate's expiration status using the openssl x509 -noout -enddate command. The volumeMounts field specifies how a volume is mounted, and this volume is shared with the NebulaGraph container.

Example:
```
sidecarContainers:
- name: auth-sidecar
  imagePullPolicy: Always
  image: reg.vesoft-inc.com/xxx/xxx:latest
  volumeMounts:
    - name: credentials
      mountPath: /credentials
```

volumes

The volumes field defines the storage volumes that need to be attached to the NebulaGraph pod. In the following example, the credentials volume is of type emptyDir, which is a temporary storage volume. Multiple containers can mount the emptyDir volume, and they all have read and write access to the same files within the volume.

Example:
```
volumes:
- name: credentials
  emptyDir:
    medium: Memory
```

volumeMounts

The volumeMounts field specifies the location within the container where the storage volume is mounted. In the example below, the credentials storage volume is mounted at the path /usr/local/nebula/certs within the NebulaGraph container.

Example:
```
volumeMounts:
- name: credentials
  mountPath: /usr/local/nebula/certs
```

`spec.<meta|graph|storage>.config`¶

To enable mTLS between the client and the Graph service, add the following fields under the spec.graphd.config in the cluster configuration file. The paths specified in fields with *_path correspond to file paths relative to /user/local/nebula. It's important to avoid using absolute paths to prevent path recognition errors.

spec:
  graph:
    config:
      ca_client_path: certs/root.crt
      ca_path: certs/root.crt
      cert_path: certs/server.crt
      key_path: certs/server.key
      enable_graph_ssl: "true"
      ssl_watch_path: certs

`initContainers`, `sidecarContainers`, `volumes`, and `volumeMounts`¶

initContainers, sidecarContainers, volumes, and volumeMounts fields are essential for implementing mTLS certificate online hot-reloading.

For the encryption scenario where only the Graph service needs to be encrypted, configure these fields under spec.graph.config.

Details about these fields are as follows:

initContainers

The initContainers field is utilized to configure an init-container responsible for generating certificate files. Note that the volumeMounts field specifies how a volume specified by volumes, shared with the NebulaGraph container, is mounted, providing read and write access.

In the following example, init-auth-sidecar performs the task of copying files from the certs directory within the image to /credentials. After this task is completed, the init-container exits.

Example:
```
initContainers:
- name: init-auth-sidecar
    command:
      - /bin/sh
      - -c
    args:
      - cp /certs/* /credentials/
    imagePullPolicy: Always
    image: reg.vesoft-inc.com/xxx/xxx:latest
    volumeMounts:
      - name: credentials
        mountPath: /credentials
```

sidecarContainers

The sidecarContainers field is dedicated to periodically monitoring the expiration time of certificates and, when they are near expiration, generating new certificates to replace the existing ones. This process ensures seamless online certificate hot-reloading without any service interruptions.

In the example provided, the auth-sidecar container employs the crond process, which runs a crontab script every minute. This script checks the certificate's expiration status using the openssl x509 -noout -enddate command. The volumeMounts field specifies how a volume is mounted, and this volume is shared with the NebulaGraph container.

Example:
```
sidecarContainers:
- name: auth-sidecar
  imagePullPolicy: Always
  image: reg.vesoft-inc.com/xxx/xxx:latest
  volumeMounts:
    - name: credentials
      mountPath: /credentials
```

volumes

The volumes field defines the storage volumes that need to be attached to the NebulaGraph pod. In the following example, the credentials volume is of type emptyDir, which is a temporary storage volume. Multiple containers can mount the emptyDir volume, and they all have read and write access to the same files within the volume.

Example:
```
volumes:
- name: credentials
  emptyDir:
    medium: Memory
```

volumeMounts

The volumeMounts field specifies the location within the container where the storage volume is mounted. In the example below, the credentials storage volume is mounted at the path /usr/local/nebula/certs within the NebulaGraph container.

Example:
```
volumeMounts:
- name: credentials
  mountPath: /usr/local/nebula/certs
```

Connect to the Graph service¶

After applying the cluster configuration file by running kubectl apply -f, you can use NebulaGraph Console to connect to the Graph service with the following command.

Note

When mTLS is required for external clients to connect to the Graph service, you need to set the relevant SSL fields depending on different clients.

You can configure spec.console to start a NebulaGraph Console container in the cluster. For details, see nebula-console.

spec:
  console:
    version: "nightly"

Then enter the console container and run the following command to connect to the Graph service.

nebula-console -addr nebula-graphd-svc.default.svc.cluster.local -port 9669 -u root  -p nebula -enable_ssl -ssl_root_ca_path /home/xxx/cert/root.crt -ssl_cert_path /home/xxx/cert/client.crt -ssl_private_key_path /home/xxx/cert/client.key

mTLS without hot-reloading¶

If you don't need to perform TLS certificate hot-reloading and prefer to use TLS certificates stored in a Secret when deploying Kubernetes applications, you can follow the steps below to enable mTLS in NebulaGraph.

Create a TLS-type Secret¶

In a K8s cluster, you can create Secrets to store sensitive information, such as passwords, OAuth tokens, and TLS certificates. In NebulaGraph, you can create a Secret to store TLS certificates and private keys. When creating a Secret, the type tls should be specified. A tls Secret is used to store TLS certificates.

For example, to create a Secret for storing server certificates and private keys:

kubectl create secret tls <server-cert> --key=<path/to/tls.key> --cert=<path/to/tls.cert> --namespace=<namespace>

<server-cert>: The name of the Secret storing the server certificate and private key.
<path/to/tls.key>: The path to the server private key file.
<path/to/tls.cert>: The path to the server certificate file.
<namespace>: The namespace where the Secret is located. If --namespace is not specified, it defaults to the default namespace.

You can follow the above steps to create Secrets for the client certificate and private key, and the CA certificate.

To view the created Secrets:

kubectl get secret --namespace=<namespace>

Configure certifications¶

NebulaGraph Operator provides the sslCerts field to specify the encrypted certificates. The sslCerts field contains four subfields. These three fields serverSecret, clientSecret, and caSecret are used to specify the Secret names of the NebulaGraph server certificate, client certificate, and CA certificate, respectively. When you specify these three fields, NebulaGraph Operator reads the certificate content from the corresponding Secret and mounts it into the cluster's Pod. The autoMountServerCerts must be set to true if you want to automatically mount the server certificate and private key into the Pod. The default value is false.

sslCerts:
  autoMountServerCerts: "true"    # Automatically mount the server certificate and private key into the Pod.
  serverSecret: "server-cert"     # The name of the server certificate Secret.
  serverCert: ""                  # The key name of the certificate in the server certificate Secret, default is tls.crt.
  serverKey: ""                   # The key name of the private key in the server certificate Secret, default is tls.key.
  clientSecret: "client-cert"     # The name of the client certificate Secret.
  clientCert: ""                  # The key name of the certificate in the client certificate Secret, default is tls.crt.
  clientKey: ""                   # The key name of the private key in the client certificate Secret, default is tls.key.
  caSecret: "ca-cert"             # The name of the CA certificate Secret.
  caCert: ""                      # The key name of the certificate in the CA certificate Secret, default is ca.crt.

The serverCert and serverKey, clientCert and clientKey, and caCert are used to specify the key names of the certificate and private key of the server Secret, the key names of the certificate and private key of the client Secret, and the key name of the CA Secret certificate. If you do not customize these field values, Operator defaults serverCert and clientCert to tls.crt, serverKey and clientKey to tls.key, and caCert to ca.crt. However, in the K8s cluster, the TLS type Secret uses tls.crt and tls.key as the default key names for the certificate and private key. Therefore, after creating the NebulaGraph cluster, you need to manually change the caCert field from ca.crt to tls.crt in the cluster configuration, so that the Operator can correctly read the content of the CA certificate. Before you customize these field values, you need to specify the key names of the certificate and private key in the Secret when creating it. For how to create a Secret with the key name specified, run the kubectl create secret generic -h command for help.

The insecureSkipVerify field is used to determine whether the client verifies the server's certificate chain and hostname. The default value is false.

It is recommended that you use the default value in production environments. In this case, you need to either specify the hostname or IP of the server in the subjectAltName field of the server's certificate or specify the SAN of the server in the spec.sslCerts.serverName field. Otherwise, an error occurs when the client verifies the server's certificate chain and hostname. For more information, see openssl.

spec:
  sslCerts:
    insecureSkipVerify: false
    serverName: managed-graph-http2.nebula

If you set the value of insecureSkipVerify to true, the client does not verify the server's certificate chain and hostname.

When the certificates are approaching expiration, they can be automatically updated by installing cert-manager. NebulaGraph will monitor changes to the certificate directory files, and once a change is detected, it will load the new certificate content into memory.

Encryption strategies¶

NebulaGraph offers three encryption strategies that you can choose and configure according to your needs.

Encryption of client-graph and all inter-service communications

If you want to encrypt all data transmission between the client, Graph service, Meta service, and Storage service, you need to add the enable_ssl = true field to each service.

Here is an example configuration:

apiVersion: apps.nebula-graph.io/v1alpha1
kind: NebulaCluster
metadata:
  name: nebula
  namespace: default
spec:
  sslCerts:
    autoMountServerCerts: "true"  # Automatically mount the server certificate and private key into the Pod.
    serverSecret: "server-cert"   # The Secret name of the server certificate and private key.
    clientSecret: "client-cert"   # The Secret name of the client certificate and private key.
    caSecret: "ca-cert"           # The Secret name of the CA certificate.
    insecureSkipVerify: "false" # Whether the client verifies the server's certificate chain and hostname. Default is "false".
    # If `insecureSkipVerify` is set to `false`,
    # either specify the server's hostname or IP of the server in the `subjectAltName` field of the server's certificate, 
    # or specify the Subject Alternative Name (SAN) of the server in the following `serverName` field.
    serverName: managed-graph-http2.nebula # The SAN of the server.
  graphd:
    config:
      enable_ssl: "true"
  metad:
    config:
      enable_ssl: "true"
  storaged:
    config:
      enable_ssl: "true"

Encryption of only Graph service communication

If the K8s cluster is deployed in the same data center and only the port of the Graph service is exposed externally, you can choose to encrypt only data transmission between the client and the Graph service. In this case, other services can communicate internally without encryption. Just add the enable_graph_ssl = true field to the Graph service.

Here is an example configuration:
```
apiVersion: apps.nebula-graph.io/v1alpha1
kind: NebulaCluster
metadata:
  name: nebula
  namespace: default
spec:
  sslCerts:
    autoMountServerCerts: "true"  
    serverSecret: "server-cert"
    caSecret: "ca-cert"
    insecureSkipVerify: "true"
  graphd:
    config:
      enable_graph_ssl: "true"
```
Note

Because NebulaGraph Operator doesn't need to call the Graph service through an interface, it's not necessary to set clientSecret in sslCerts.

Encryption of only Meta service communication

If you need to transmit confidential information to the Meta service, you can choose to encrypt data transmission related to the Meta service. In this case, you need to add the enable_meta_ssl = true configuration to each component.

Here is an example configuration:
```
apiVersion: apps.nebula-graph.io/v1alpha1
kind: NebulaCluster
metadata:
  name: nebula
  namespace: default
spec:
  sslCerts:
    autoMountServerCerts: "true"
    serverSecret: "server-cert"
    clientSecret: "client-cert"
    caSecret: "ca-cert"
    insecureSkipVerify: "true"
  graphd:
    config:
      enable_meta_ssl: "true"
  metad:
    config:
      enable_meta_ssl: "true"
  storaged:
    config:
      enable_meta_ssl: "true"
```
After setting up the encryption policy, when an external client needs to connect to the Graph service with mutual TLS, you still need to set the relevant TLS fields according to the different clients. See the Use NebulaGraph Console to connect to Graph service section below for examples.

Example of enabling mTLS without hot-reloading¶

Use the pre-generated server and client certificates and private keys, and the CA certificate to create a Secret for each.
```
kubectl create secret tls <client/server/ca-cert-secret> --key=<client/server/ca.key> --cert=<client/server/ca.crt>
```
- tls: Indicates that the type of secret being created is TLS, which is used to store TLS certificates.
- <client/server/ca-cert-secret>: Specifies the name of the new secret being created, which can be customized.
- --key=<client/server/ca.key>: Specifies the path to the private key file of the TLS certificate to be stored in the secret.
- --cert=<client/server/ca.crt>: Specifies the path to the public key certificate file of the TLS certificate to be stored in the secret.

Add server certificate, client certificate, CA certificate configuration, and encryption policy configuration in the corresponding cluster instance YAML file. For details, see Encryption strategies.

For example, add encryption configuration for transmission data between client, Graph service, Meta service, and Storage service.

apiVersion: apps.nebula-graph.io/v1alpha1
kind: NebulaCluster
metadata:
  name: nebula
  namespace: default
spec:
  sslCerts:
    autoMountServerCerts: "true"
    serverSecret: "server-cert"  # The name of the server Certificate Secret.
    clientSecret: "client-cert"  # The name of the client Certificate Secret.
    caSecret: "ca-cert"          # The name of the CA Certificate Secret.
    insecureSkipVerify: "false"  # Whether the client verifies the server's certificate chain and hostname. Default is "false".
    serverName: managed-graph-http2.nebula # The SAN of the server.
  graphd:
    config:
      enable_ssl: "true"
  metad:
    config:
      enable_ssl: "true"
  storaged:
    config:
      enable_ssl: "true"

Use kubectl apply -f to apply the file to the Kubernetes cluster.
Verify that the values of serverCert, serverKey, clientCert, clientKey, caCert under the sslCerts field in the cluster configuration match the key names of the certificates and private keys stored in the created Secret.
```
# Check the key names of the certificate and private key stored in the Secret. For example, check the key name of the CA certificate stored in the Secret.
kubectl get secret ca-cert -o yaml
```
```
# Check the cluster configuration file.
kubectl get nebulacluster nebula -o yaml
```
Example output:
```
...
spec:
  sslCerts:
    autoMountServerCerts: "true"
    serverSecret: server-cert
    serverCert: tls.crt
    serverKey: tls.key
    clientSecret: client-cert
    clientCert: tls.crt
    clientKey: tls.key
    caSecret: ca-cert
    caCert: ca.crt    
...
```
If the key names of the certificates and private keys stored in the Secret are different from the values of serverCert, serverKey, clientCert, clientKey, caCert under the sslCerts field in the cluster configuration, you need to execute kubectl edit nebulacluster <cluster_name> to manually modify the cluster configuration file.

In the example output, the key name of the CA certificate in the TLS-type Secret is tls.crt, so you need to change the value of caCert from ca.crt to tls.crt.
Use NebulaGraph Console to connect to the Graph service and establish a secure TLS connection.

Example:
```
kubectl run -ti --image vesoft/nebula-console:v3.5.0 --restart=Never -- nebula-console -addr 10.98.xxx.xx  -port 9669 -u root -p nebula -enable_ssl -ssl_root_ca_path /path/to/cert/root.crt -ssl_cert_path /path/to/cert/client.crt -ssl_private_key_path /path/to/cert/client.key
```
- -enable_ssl: Use mTLS when connecting to NebulaGraph.
- -ssl_root_ca_path: Specify the storage path of the CA root certificate.
- -ssl_cert_path: Specify the storage path of the TLS public key certificate.
- -ssl_private_key_path: Specify the storage path of the TLS private key.
- For details on using NebulaGraph Console to connect to the Graph service, see Connect to NebulaGraph.
Note

If you set spec.console to start a NebulaGraph Console container in the cluster, you can enter the console container and run the following command to connect to the Graph service.
```
nebula-console -addr 10.98.xxx.xx  -port 9669 -u root -p nebula -enable_ssl -ssl_root_ca_path /path/to/cert/root.crt -ssl_cert_path /path/to/cert/client.crt -ssl_private_key_path /path/to/cert/client.key
```

At this point, you can enable mTLS in NebulaGraph.

Last update: March 7, 2024

Enable mTLS in NebulaGraph¶

Prerequisites¶

Encryption scenarios¶

mTLS with certificate hot-reloading¶

Examples¶

Must-have fields¶

spec.sslCerts¶

spec.sslCerts.insecureSkipVerify¶

spec.<meta|graph|storage>.config¶

initContainers, sidecarContainers, volumes, and volumeMounts¶

spec.<meta|graph|storage>.config¶

initContainers, sidecarContainers, volumes, and volumeMounts¶

Connect to the Graph service¶

mTLS without hot-reloading¶

Create a TLS-type Secret¶

Configure certifications¶

Encryption strategies¶

Example of enabling mTLS without hot-reloading¶

`spec.sslCerts`¶

`spec.sslCerts.insecureSkipVerify`¶

`spec.<meta|graph|storage>.config`¶

`initContainers`, `sidecarContainers`, `volumes`, and `volumeMounts`¶

`spec.<meta|graph|storage>.config`¶

`initContainers`, `sidecarContainers`, `volumes`, and `volumeMounts`¶