Storage Service configurations¶

NebulaGraph provides two initial configuration files for the Storage Service, nebula-storaged.conf.default and nebula-storaged.conf.production. Users can use them in different scenarios conveniently. The default file path is /usr/local/nebula/etc/.

Caution

It is not recommended to modify the value of local_config to false. If modified, the NebulaGraph service will first read the cached configurations, which may cause configuration inconsistencies between clusters and cause unknown risks.
It is not recommended to modify the configurations that are not introduced in this topic, unless you are familiar with the source code and fully understand the function of configurations.

How to use the configuration files¶

To use the initial configuration file, choose one of the above two files and delete the suffix .default or .production from the initial configuration file for the Meta Service to apply the configurations defined in it.

About parameter values¶

If a parameter is not set in the configuration file, NebulaGraph uses the default value. Not all parameters are predefined. And the predefined parameters in the two initial configuration files are different. This topic uses the parameters in nebula-metad.conf.default. For parameters that are not included in nebula-metad.conf.default, see nebula-storaged.conf.production.

Caution

Some parameter values in the configuration file can be dynamically modified during runtime. We label these parameters as Yes that supports runtime dynamic modification in this article. When the local_config value is set to true, the dynamically modified configuration is not persisted, and the configuration will be restored to the initial configuration after the service is restarted. For more information, see Modify configurations.

Note

The configurations of the Raft Listener and the Storage service are different. For details, see Deploy Raft listener.

For all parameters and their current values, see Configurations.

Basics configurations¶

Name	Predefined value	Description	Whether supports runtime dynamic modifications
`daemonize`	`true`	When set to `true`, the process is a daemon process.	No
`pid_file`	`pids/nebula-storaged.pid`	The file that records the process ID.	No
`timezone_name`	`UTC+00:00:00`	Specifies the NebulaGraph time zone. This parameter is not predefined in the initial configuration files, if you need to use this parameter, add it manually. For the format of the parameter value, see Specifying the Time Zone with TZ. For example, `--timezone_name=UTC+08:00` represents the GMT+8 time zone.	No
`local_config`	`true`	When set to `true`, the process gets configurations from the configuration files.	No

Note

While inserting property values of time types, NebulaGraph transforms time types (except TIMESTAMP) to the corresponding UTC according to the time zone specified by timezone_name. The time-type values returned by nGQL queries are all UTC.
timezone_name is only used to transform the data stored in NebulaGraph. Other time-related data of the NebulaGraph processes still uses the default time zone of the host, such as the log printing time.

Logging configurations¶

Name	Predefined value	Description	Whether supports runtime dynamic modifications
`log_dir`	`logs`	The directory that stores the Meta Service log. It is recommended to put logs on a different hard disk from the data.	No
`minloglevel`	`0`	Specifies the minimum level of the log. That is, log messages at or above this level. Optional values are `0` (INFO), `1` (WARNING), `2` (ERROR), `3` (FATAL). It is recommended to set it to `0` during debugging and `1` in a production environment. If it is set to `4`, NebulaGraph will not print any logs.	Yes
`v`	`0`	Specifies the detailed level of VLOG. That is, log all VLOG messages less or equal to the level. Optional values are `0`, `1`, `2`, `3`, `4`, `5`. The VLOG macro provided by glog allows users to define their own numeric logging levels and control verbose messages that are logged with the parameter `v`. For details, see Verbose Logging.	Yes
`logbufsecs`	`0`	Specifies the maximum time to buffer the logs. If there is a timeout, it will output the buffered log to the log file. `0` means real-time output. This configuration is measured in seconds.	No
`redirect_stdout`	`true`	When set to `true`, the process redirects the`stdout` and `stderr` to separate output files.	No
`stdout_log_file`	`graphd-stdout.log`	Specifies the filename for the `stdout` log.	No
`stderr_log_file`	`graphd-stderr.log`	Specifies the filename for the `stderr` log.	No
`stderrthreshold`	`3`	Specifies the `minloglevel` to be copied to the `stderr` log.	No
`timestamp_in_logfile_name`	`true`	Specifies if the log file name contains a timestamp. `true` indicates yes, `false` indicates no.	No

Networking configurations¶

Name	Predefined value	Description	Whether supports runtime dynamic modifications
`meta_server_addrs`	`127.0.0.1:9559`	Specifies the IP addresses and ports of all Meta Services. Multiple addresses are separated with commas.	No
`local_ip`	`127.0.0.1`	Specifies the local IP for the Storage Service. The local IP address is used to identify the nebula-storaged process. If it is a distributed cluster or requires remote access, modify it to the corresponding address.	No
`port`	`9779`	Specifies RPC daemon listening port of the Storage service. The external port for the Meta Service is predefined to `9779`. The internal port is predefined to `9777`, `9778`, and `9780`. Nebula Graph uses the internal port for multi-replica interactions. `9777`: The port used by the Drainer service, which is only exposed in the Enterprise Edition cluster. `9778`: The port used by the Admin service, which receives Meta commands for Storage. `9780`: The port used for Raft communication.	No
`ws_ip`	`0.0.0.0`	Specifies the IP address for the HTTP service.	No
`ws_http_port`	`19779`	Specifies the port for the HTTP service.	No
`heartbeat_interval_secs`	`10`	Specifies the default heartbeat interval. Make sure the `heartbeat_interval_secs` values for all services are the same, otherwise NebulaGraph CANNOT work normally. This configuration is measured in seconds.	Yes

Caution

The real IP address must be used in the configuration file. Otherwise, 127.0.0.1/0.0.0.0 cannot be parsed correctly in some cases.

Raft configurations¶

Name	Predefined value	Description	Whether supports runtime dynamic modifications
`raft_heartbeat_interval_secs`	`30`	Specifies the time to expire the Raft election. The configuration is measured in seconds.	Yes
`raft_rpc_timeout_ms`	`500`	Specifies the time to expire the Raft RPC. The configuration is measured in milliseconds.	Yes
`wal_ttl`	`14400`	Specifies the lifetime of the RAFT WAL. The configuration is measured in seconds.	Yes

Disk configurations¶

Name	Predefined value	Description	Whether supports runtime dynamic modifications
`data_path`	`data/storage`	Specifies the data storage path. Multiple paths are separated with commas. For NebulaGraph of the community edition, one RocksDB instance corresponds to one path. For NebulaGraph of the enterprise edition, one RocksDB instance corresponds to one partition.	No
`minimum_reserved_bytes`	`268435456`	Specifies the minimum remaining space of each data storage path. When the value is lower than this standard, the cluster data writing may fail. This configuration is measured in bytes.	No
`rocksdb_batch_size`	`4096`	Specifies the block cache for a batch operation. The configuration is measured in bytes.	No
`rocksdb_block_cache`	`4`	Specifies the block cache for BlockBasedTable. The configuration is measured in megabytes.	No
`disable_page_cache`	`false`	Enables or disables the operating system's page cache for NebulaGraph. By default, the parameter value is `false` and page cache is enabled. If the value is set to `true`, page cache is disabled and sufficient block cache space must be configured for NebulaGraph.	No
`engine_type`	`rocksdb`	Specifies the engine type.	No
`rocksdb_compression`	`lz4`	Specifies the compression algorithm for RocksDB. Optional values are `no`, `snappy`, `lz4`, `lz4hc`, `zlib`, `bzip2`, and `zstd`. This parameter modifies the compression algorithm for each level. If you want to set different compression algorithms for each level, use the parameter `rocksdb_compression_per_level`.	No
`rocksdb_compression_per_level`	\	Specifies the compression algorithm for each level. The priority is higher than `rocksdb_compression`. For example, `no:no:lz4:lz4:snappy:zstd:snappy`. You can also not set certain levels of compression algorithms, for example, `no:no:lz4:lz4::zstd`, level L4 and L6 use the compression algorithm of `rocksdb_compression`.	No
`enable_rocksdb_statistics`	`false`	When set to `false`, RocksDB statistics is disabled.	No
`rocksdb_stats_level`	`kExceptHistogramOrTimers`	Specifies the stats level for RocksDB. Optional values are `kExceptHistogramOrTimers`, `kExceptTimers`, `kExceptDetailedTimers`, `kExceptTimeForMutex`, and `kAll`.	No
`enable_rocksdb_prefix_filtering`	`true`	When set to `true`, the prefix bloom filter for RocksDB is enabled. Enabling prefix bloom filter makes the graph traversal faster but occupies more memory.	No
`enable_rocksdb_whole_key_filtering`	`false`	When set to `true`, the whole key bloom filter for RocksDB is enabled.
`rocksdb_filtering_prefix_length`	`12`	Specifies the prefix length for each key. Optional values are `12` and `16`. The configuration is measured in bytes.	No
`enable_partitioned_index_filter`	`false`	When set to `true`, it reduces the amount of memory used by the bloom filter. But in some random-seek situations, it may reduce the read performance. This parameter is not predefined in the initial configuration files, if you need to use this parameter, add it manually.	No

Misc configurations¶

Caution

The configuration snapshot in the following table is different from the snapshot in NebulaGraph. The snapshot here refers to the stock data on the leader when synchronizing Raft.

Name	Predefined value	Description	Whether supports runtime dynamic modifications
`query_concurrently`	`true`	Whether to turn on multi-threaded queries. Enabling it can improve the latency performance of individual queries, but it will reduce the overall throughput under high pressure.	Yes
`auto_remove_invalid_space`	`true`	After executing `DROP SPACE`, the specified graph space will be deleted. This parameter sets whether to delete all the data in the specified graph space at the same time. When the value is `true`, all the data in the specified graph space will be deleted at the same time.	Yes
`num_io_threads`	`16`	The number of network I/O threads used to send RPC requests and receive responses.	Yes
`num_max_connections`	`0`	Max active connections for all networking threads. 0 means no limit. Max connections for each networking thread = num_max_connections / num_netio_threads	No
`num_worker_threads`	`32`	The number of worker threads for one RPC-based Storage service.	Yes
`max_concurrent_subtasks`	`10`	The maximum number of concurrent subtasks to be executed by the task manager.	Yes
`snapshot_part_rate_limit`	`10485760`	The rate limit when the Raft leader synchronizes the stock data with other members of the Raft group. Unit: bytes/s.	Yes
`snapshot_batch_size`	`1048576`	The amount of data sent in each batch when the Raft leader synchronizes the stock data with other members of the Raft group. Unit: bytes.	Yes
`rebuild_index_part_rate_limit`	`4194304`	The rate limit when the Raft leader synchronizes the index data rate with other members of the Raft group during the index rebuilding process. Unit: bytes/s.	Yes
`rebuild_index_batch_size`	`1048576`	The amount of data sent in each batch when the Raft leader synchronizes the index data with other members of the Raft group during the index rebuilding process. Unit: bytes.	Yes

RocksDB options¶

Name	Predefined value	Description	Whether supports runtime dynamic modifications
`rocksdb_db_options`	`{}`	Specifies the RocksDB database options.	Yes
`rocksdb_column_family_options`	`{"write_buffer_size":"67108864",` `"max_write_buffer_number":"4",` `"max_bytes_for_level_base":"268435456"}`	Specifies the RocksDB column family options.	Yes
`rocksdb_block_based_table_options`	`{"block_size":"8192"}`	Specifies the RocksDB block based table options.	Yes

The format of the RocksDB option is {"<option_name>":"<option_value>"}. Multiple options are separated with commas.

Supported options of rocksdb_db_options and rocksdb_column_family_options are listed as follows.

rocksdb_db_options

max_total_wal_size
delete_obsolete_files_period_micros
max_background_jobs
stats_dump_period_sec
compaction_readahead_size
writable_file_max_buffer_size
bytes_per_sync
wal_bytes_per_sync
delayed_write_rate
avoid_flush_during_shutdown
max_open_files
stats_persist_period_sec
stats_history_buffer_size
strict_bytes_per_sync
enable_rocksdb_prefix_filtering
enable_rocksdb_whole_key_filtering
rocksdb_filtering_prefix_length
num_compaction_threads
rate_limit

rocksdb_column_family_options

write_buffer_size
max_write_buffer_number
level0_file_num_compaction_trigger
level0_slowdown_writes_trigger
level0_stop_writes_trigger
target_file_size_base
target_file_size_multiplier
max_bytes_for_level_base
max_bytes_for_level_multiplier
disable_auto_compactions

For more information, see RocksDB official documentation.

Black Box configurations¶

Enterpriseonly

The Nebula-BBox configurations are for the Enterprise Edition only.

Name	Predefined value	Description	Whether supports runtime dynamic modifications
`ng_black_box_switch`	`true`	Whether to enable the Nebula-BBox feature.	No
`ng_black_box_home`	`black_box`	The name of the directory to store Nebula-BBox file data.	No
`ng_black_box_dump_period_seconds`	`5`	The time interval for Nebula-BBox to collect metric data. Unit: Second.	No
`ng_black_box_file_lifetime_seconds`	`1800`	Storage time for Nebula-BBox files generated after collecting metric data. Unit: Second.	Yes

Memory Tracker configurations¶

For details about Memory Tracker, see Memory Tracker: Memory Management Practice in NebulaGraph Database

Name	Predefined value	Description	Whether supports runtime dynamic modifications
`memory_tracker_limit_ratio`	`0.8`	The value of this parameter can be set to `(0, 1]`, `2`, and `3`. `(0, 1]`: The percentage of free memory. When the free memory is lower than this value, NebulaGraph stops accepting queries. Calculated as follows: `Free memory / (Total memory - Reserved memory)` Note: For clusters with a mixed-used environment, the value of `memory_tracker_limit_ratio` should be set to a lower value. For example, when Graphd is expected to occupy only 50% of memory, the value can be set to less than `0.5`. `2`: Dynamic Self Adaptive mode. MemoryTracker dynamically adjusts the available memory based on the system's current available memory. Note: This feature is experimental. As memory usage cannot be monitored in real time in dynamic adaptive mode, an OOM error may still occur to handle large memory allocations. `3`: Disable MemoryTracker. MemoryTracker only logs memory usage and does not interfere with executions even if the limit is exceeded.	Yes
`memory_tracker_untracked_reserved_memory_mb`	`50`	The reserved memory that is not tracked by the Memory Tracker. Unit: MB.	Yes
`memory_tracker_detail_log`	`false`	Whether to enable the Memory Tracker log. When the value is `true`, the Memory Tracker log is generated.	Yes
`memory_tracker_detail_log_interval_ms`	`60000`	The time interval for generating the Memory Tracker log. Unit: Millisecond. `memory_tracker_detail_log` is `true` when this parameter takes effect.	Yes
`memory_purge_enabled`	`true`	Whether to enable the memory purge feature. When the value is `true`, the memory purge feature is enabled.	Yes
`memory_purge_interval_seconds`	`10`	The time interval for the memory purge feature to purge memory. Unit: Second. This parameter only takes effect if `memory_purge_enabled` is set to true.	Yes

For super-Large vertices¶

When the query starting from each vertex gets an edge, truncate it directly to avoid too many neighboring edges on the super-large vertex, because a single query occupies too much hard disk and memory. Or you can truncate a certain number of edges specified in the Max_edge_returned_per_vertex parameter. Excess edges will not be returned. This parameter applies to all spaces.

Property name	Default value	Description	Whether supports runtime dynamic modifications
max_edge_returned_per_vertex	`2147483647`	Specifies the maximum number of edges returned for each dense vertex. Excess edges are truncated and not returned. This parameter is not predefined in the initial configuration files, if you need to use this parameter, add it manually.	No

Storage configurations for large dataset¶

Warning

One graph space takes up at least about 300 MB of memory.

When you have a large dataset (in the RocksDB directory) and your memory is tight, we suggest that you set the enable_partitioned_index_filter parameter to true. The performance is affected because RocksDB indexes are cached.

Last update: July 21, 2023