Enable Horizontal Pod Autoscaling (HPA) for Graph services¶

NebulaGraph Operator provides the NebulaAutoscaler object for you to implement Horizontal Pod Autoscaling (HPA) for the Graph service.

Prerequisites¶

A NebulaGraph cluster is created. For details, see Create a NebulaGraph cluster.

Limitations¶

Automatic scaling via HPA is supported only for the Graph service.

Steps¶

The following procedure shows how to enable HPA in a NebulaGraph cluster:

Install the metrics-server.

Developed based on the HorizontalPodAutoscaler in Kubernetes, the NebulaAutoscaler automatically scales the number of pods based on the metrics collected by the metrics-server.

Run the following command to install the latest metrics-server release:
```
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
```

Verify that the metrics-server is working.

The metrics-server implements the Metrics API, which provides information about resource usage for nodes and pods in the cluster. The following example calls the Metrics API to obtain resource usage data from the metrics-server. If resource usage data is successfully returned, it indicates the metrics-server is working.

Run the following command to query resource usage on a pod named nebula-graphd-1:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/nebula-graphd-1" | jq '.'

Sample response:

{
  "kind": "PodMetrics",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "name": "nebula-graphd-1",
    "namespace": "default",
    "creationTimestamp": "2023-09-27T13:39:54Z",
    "labels": {
      "app.kubernetes.io/cluster": "nebula",
      "app.kubernetes.io/component": "graphd",
      "app.kubernetes.io/managed-by": "nebula-operator",
      "app.kubernetes.io/name": "nebula-graph",
      "controller-revision-hash": "nebula-graphd-56cf5f8b66",
      "statefulset.kubernetes.io/pod-name": "nebula-graphd-1"
    }
  },
  "timestamp": "2023-09-27T13:39:48Z",
  "window": "15.015s",
  "containers": [
    {
      "name": "graphd",
      "usage": {
        "cpu": "323307n",
        "memory": "12644Ki"
      }
    }
  ]
}

Run the following command to query resource usage on a node named 192-168-8-35:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes/192-168-8-35" | jq '.'

Sample response:

{
  "kind": "NodeMetrics",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "name": "192-168-8-35",
    "creationTimestamp": "2023-09-27T14:00:13Z",
    "labels": {
      "beta.kubernetes.io/arch": "amd64",
      "beta.kubernetes.io/os": "linux",
      "kubernetes.io/arch": "amd64",
      "kubernetes.io/hostname": "192-168-8-35",
      "kubernetes.io/os": "linux",
      "nebula": "cloud",
      "node-role.kubernetes.io/control-plane": "",
      "node.kubernetes.io/exclude-from-external-load-balancers": ""
    }
  },
  "timestamp": "2023-09-27T14:00:00Z",
  "window": "20.045s",
  "usage": {
    "cpu": "164625163n",
    "memory": "8616740Ki"
  }
}

Create a NebulaAutoscaler object.

Use the following YAML sample to create a NebulaAutoscaler object that automatically adjusts the number of pods between 2 to 5 based on the average CPU utilization.

apiVersion: autoscaling.nebula-graph.io/v1alpha1
kind: NebulaAutoscaler
metadata:
  name: nebula-autoscaler
spec:
  nebulaClusterRef:
    name: nebula
  graphdPolicy:
    minReplicas: 2
    maxReplicas: 5
    metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
  pollingPeriod: 30s

The key parameters are as follows:

spec.nebulaClusterRef: The target cluster that the NebulaAutoscaler applies to.
spec.graphdPolicy: The auto scaling policy adopted by the NebulaAutoscaler. All the child fields are compatible with the fields used by the Kubernetes HorizontalPodAutoscaler. Check the API reference for details.
spec.pollingPeriod: The time interval between each resource usage checks by the NebulaAutoscaler.

The NebulaAutoscaler also supports the behavior parameter, which enables you to control the scaling procedure in great detail by configuring separate scale-up and scale-down behaviors.

Make sure you understand the API reference before you use the behavior parameter.

The following sample creates a NebulaAutoscaler object that behaves differently in scale-up and scale-down.

apiVersion: autoscaling.nebula-graph.io/v1alpha1
kind: NebulaAutoscaler
metadata:
  name: nebula-autoscaler
spec:
  nebulaClusterRef:
    name: nebula
  graphdPolicy:
    minReplicas: 2
    maxReplicas: 5
    metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 300
        policies:
        - type: Percent
          value: 100
          periodSeconds: 15
      scaleUp:
        stabilizationWindowSeconds: 0
        policies:
        - type: Percent
          value: 100
          periodSeconds: 15
        - type: Pods
          value: 4
          periodSeconds: 15
        selectPolicy: Max
  pollingPeriod: 30s

Check whether HPA is working.

After running kubectl apply to create the NebulaAutoscaler object, you can use the following commands to check whether HPA is working.

Run the kubectl get na command to check the NebulaAutoscaler status.

Sample response:

NAME                REFERENCE   MIN-REPLICAS   MAX-REPLICAS   CURRENT-REPLICAS   ACTIVE   ABLETOSCALE   LIMITED   READY   AGE
nebula-autoscaler   nebula      2              5              2                  True     True          True      True    19h

Run the kubectl get nc command to check the cluster status.

Sample response:

NAME     READY   GRAPHD-DESIRED   GRAPHD-READY   METAD-DESIRED   METAD-READY   STORAGED-DESIRED   STORAGED-READY   AGE
nebula   True    2                2              1               1             3                  3                20h

Last update: November 22, 2023