Skip to content

System design suggestions

QPS or low-latency first

  • NebulaGraph 2.6.2 is good at handling small requests with high concurrency. In such scenarios, the whole graph is huge, containing maybe trillions of vertices or edges, but the subgraphs accessed by each request are not large (containing millions of vertices or edges), and the latency of a single request is low. The concurrent number of such requests, i.e., the QPS, can be huge.
  • On the other hand, in interactive analysis scenarios, the request concurrency is usually not high, but the subgraphs accessed by each request are large, with thousands of millions of vertices or edges. To lower the latency of big requests in such scenarios, you can split big requests into multiple small requests in the application, and concurrently send them to multiple graphd processes. This can decrease the memory used by each graphd process as well. Besides, you can use Nebula Algorithm for such scenarios.

Horizontal or vertical scaling

NebulaGraph 2.6.2 supports horizontal scaling.

  • The horizontal scaling of the Storaged process:

    - Increasing the number of machines deployed with the Storaged process can increase the overall capability of the cluster linearly, including increasing the overall QPS and reducing latency.

    - However, the number of partitions is fixed when creating a graph space. The service capability of a single partition is determined by a single server. The operations depending on a single partition include fetching properties of a single vertex (FETCH), a breadth-first traversal from a single vertex (GO), etc.

  • The horizontal scaling of the Graphd process:

    - Each request from the client is handled by one and only one Graphd process, with no other Graphd processes participating in the processing of the request.

    - Therefore, increasing the number of machines deployed with the Graphd process can increase the overall QPS of the cluster, but cannot lower the latency of a single request.

  • Metad does not support horizontal scaling.

Vertical scaling usually has higher hardware costs, but relatively simple operations. NebulaGraph 2.6.2 can also be scaled vertically.

Data transmission and optimization

  • Read/write balance. NebulaGraph fits into OLTP scenarios with balanced read/write, i.e., concurrent write and read. It is not suitable for OLAP scenarios that usually need to write once and read many times.
  • Select different write methods. For large batches of data writing, use SST files. For small batches of data writing, use INSERT.
  • Run COMPACTION and BALANCE jobs to optimize data format and storage distribution at the right time.
  • NebulaGraph 2.6.2 does not support transactions and isolation in the relational database and is closer to NoSQL.

Query preheating and data preheating

Preheat on the application side:

  • The Grapd process does not support pre-compiling queries and generating corresponding query plans, nor can it cache previous query results.
  • The Storagd process does not support preheating data. Only the LSM-Tree and BloomFilter of RocksDB are loaded into memory at startup.
  • Once accessed, vertices and edges are cached respectively in two types of LRU cache of the Storage Service.
Last update: March 13, 2023