Deploy Raft Listener for NebulaGraph Storage service¶

NebulaGraph CommunityNebulaGraph Enterprise

Full-text index data is written to the Elasticsearch cluster asynchronously. The Raft Listener (Listener for short) is a separate process that fetches data from the Storage Service and writes them into the Elasticsearch cluster.

Prerequisites¶

You have read and fully understood the restrictions for using full-text indexes.

You have deployed a NebulaGraph cluster.

You have deploy a Elasticsearch cluster.

You have prepared at least one extra Storage Server. To use the full-text search, you must run one or more Storage Server as the Raft Listener.

Precautions¶

The Storage Service that you want to run as the Listener must have the same or later release with all the other Nebula Graph services in the cluster.

For now, you can only add all Listeners to a graph space once and for all. Trying to add a new Listener to a graph space that already has a Listener will fail. To add all Listeners, set them in one statement.

Deployment process¶

Step 1: Install the Storage service¶

The Listener process and the storaged process use the same binary file. However, their configuration files and using ports are different. You can install NebulaGraph on all servers that need to deploy a Listener, but only the Storage service can be used. For details, see Install NebulaGraph by RPM or DEB Package.

Step 2: Prepare the configuration file for the Listener¶

You have to prepare a corresponding configuration file on the machine that you want to deploy a Listener. The file must be named as nebula-storaged-listener.conf and stored in the etc directory. A template is provided for your reference. Note that the file suffix .production should be removed.

Most configurations are the same as the configurations of Storage Service. This topic only introduces the differences.

Name	Default value	Description
`daemonize`	`true`	When set to `true`, the process is a daemon process.
`pid_file`	`pids/nebula-metad-listener.pid`	The file that records the process ID.
`meta_server_addrs`	-	IP addresses and ports of all Meta services. Multiple Meta services are separated by commas.
`local_ip`	-	The local IP address of the Listener service.
`port`	-	The listening port of the RPC daemon of the Listener service.
`heartbeat_interval_secs`	`10`	The heartbeat interval of the Meta service. The unit is second (s).
`listener_path`	`data/listener`	The WAL directory of the Listener. Only one directory is allowed.
`data_path`	`data`	For compatibility reasons, this parameter can be ignored. Fill in the default value `data`.
`part_man_type`	`memory`	The type of the part manager. Optional values are `memory` and `meta`.
`rocksdb_batch_size`	`4096`	The default reserved bytes for batch operations.
`rocksdb_block_cache`	`4`	The default block cache size of BlockBasedTable. The unit is Megabyte (MB).
`engine_type`	`rocksdb`	The type of the Storage engine, such as `rocksdb`, `memory`, etc.
`part_type`	`simple`	The type of the part, such as `simple`, `consensus`, etc.

Note

Use real IP addresses in the configuration file instead of domain names or loopback IP addresses such as 127.0.0.1.

Step 3: Start Listeners¶

Run the following command to start the Listener.

./bin/nebula-storaged --flagfile <listener_config_path>/nebula-storaged-listener.conf

${listener_config_path} is the path where you store the Listener configuration file.

Step 4: Add Listeners to NebulaGraph¶

Connect to NebulaGraph and run USE <space> to enter the graph space that you want to create full-text indexes for. Then run the following statement to add a Listener into NebulaGraph.

ADD LISTENER ELASTICSEARCH <listener_ip:port> [,<listener_ip:port>, ...]

Warning

You must use real IPs for a Listener.

Add all Listeners in one statement completely.

nebula> ADD LISTENER ELASTICSEARCH 192.168.8.5:9789,192.168.8.6:9789;

Show Listeners¶

Run the SHOW LISTENER statement to list all Listeners.

Example¶

nebula> SHOW LISTENER;
+--------+-----------------+-----------------------+----------+
| PartId | Type            | Host                  | Status   |
+--------+-----------------+-----------------------+----------+
| 1      | "ELASTICSEARCH" | "[192.168.8.5:46780]" | "ONLINE" |
| 2      | "ELASTICSEARCH" | "[192.168.8.5:46780]" | "ONLINE" |
| 3      | "ELASTICSEARCH" | "[192.168.8.5:46780]" | "ONLINE" |
+--------+-----------------+-----------------------+----------+

Remove Listeners¶

Run the REMOVE LISTENER ELASTICSEARCH statement to remove all Listeners in a graph space.

Example¶

nebula> REMOVE LISTENER ELASTICSEARCH;

Full-text index data is written to the Elasticsearch cluster asynchronously. The Raft Listener (Listener for short) is a separate process that fetches data from the Storage Service and writes them into the Elasticsearch cluster.

Prerequisites¶

You have read and fully understood the restrictions for using full-text indexes.

You have deployed a NebulaGraph cluster.

You have deployed a Elasticsearch cluster.

You have prepared one or multiple servers to run one or multiple raft listeners.

Precautions¶

The Storage Service that you want to run as the Listener must have the same or later release with all the other Nebula Graph services in the cluster.

For now, you can only add all Listeners to a graph space once and for all. Trying to add a new Listener to a graph space that already has a Listener will fail. To add all Listeners, set them in one statement.

Deployment process¶

Step 1: Install the Listener service¶

The Listener service uses the same binary as the storaged service. However, the configuration files are different and the processes use different ports. You can install NebulaGraph on all servers that need to deploy a Listener, but only the storaged service can be used. For details, see Install NebulaGraph by RPM or DEB Package.

Step 2: Prepare the configuration file for the Listener¶

In the etc directory, remove the suffix from nebula-storaged-listener.conf.default or nebula-storaged-listener.conf.production to nebula-storaged-listener.conf, and then modify the configuration content.

Most configurations are the same as the configurations of Storage Service. This topic only introduces the differences.

Name	Default value	Description
`daemonize`	`true`	When set to `true`, the process is a daemon process.
`pid_file`	`pids/nebula-metad-listener.pid`	The file that records the process ID.
`meta_server_addrs`	-	IP addresses and ports of all Meta services. Multiple Meta services are separated by commas.
`local_ip`	-	The local IP address of the Listener service. Use real IP addresses instead of domain names or loopback IP addresses such as `127.0.0.1`.
`port`	-	The listening port of the RPC daemon of the Listener service.
`heartbeat_interval_secs`	`10`	The heartbeat interval of the Meta service. The unit is second (s).
`listener_path`	`data/listener`	The WAL directory of the Listener. Only one directory is allowed.
`data_path`	`data`	For compatibility reasons, this parameter can be ignored. Fill in the default value `data`.
`part_man_type`	`memory`	The type of the part manager. Optional values are `memory` and `meta`.
`rocksdb_batch_size`	`4096`	The default reserved bytes for batch operations.
`rocksdb_block_cache`	`4`	The default block cache size of BlockBasedTable. The unit is Megabyte (MB).
`engine_type`	`rocksdb`	The type of the Storage engine, such as `rocksdb`, `memory`, etc.
`part_type`	`simple`	The type of the part, such as `simple`, `consensus`, etc.

Step 3: Start Listeners¶

To initiate the Listener, navigate to the installation path of the desired cluster and execute the following command:

./bin/nebula-storaged --flagfile etc/nebula-storaged-listener.conf

Step 4: Add Listeners to NebulaGraph¶

Connect to NebulaGraph and run USE <space> to enter the graph space that you want to create full-text indexes for. Then run the following statement to add a Listener into NebulaGraph.

ADD LISTENER ELASTICSEARCH <listener_ip:port> [,<listener_ip:port>, ...]

Warning

You must use real IPs for a Listener.

Add all Listeners in one statement completely.

nebula> ADD LISTENER ELASTICSEARCH 192.168.8.100:9789,192.168.8.101:9789;

Show Listeners¶

Run the SHOW LISTENER statement to list all Listeners.

Example¶

nebula> SHOW LISTENER;
+--------+-----------------+------------------------+-------------+
| PartId | Type            | Host                   | Host Status |
+--------+-----------------+------------------------+-------------+
| 1      | "ELASTICSEARCH" | ""192.168.8.100":9789" | "ONLINE"    |
| 2      | "ELASTICSEARCH" | ""192.168.8.100":9789" | "ONLINE"    |
| 3      | "ELASTICSEARCH" | ""192.168.8.100":9789" | "ONLINE"    |
+--------+-----------------+------------------------+-------------+

Remove Listeners¶

Run the REMOVE LISTENER ELASTICSEARCH statement to remove all Listeners in a graph space.

Example¶

nebula> REMOVE LISTENER ELASTICSEARCH;

Last update: July 5, 2023