Upgrade NebulaGraph to 3.4.1¶
This topic describes how to upgrade NebulaGraph from version 2.x and 3.x to 3.4.1, taking upgrading from version 2.6.1 to 3.4.1 as an example.
Applicable source versions¶
This topic applies to upgrading NebulaGraph from 2.5.0 and later 2.x, and 3.x versions to 3.4.1. It does not apply to historical versions earlier than 2.5.0, including the 1.x versions.
To upgrade NebulaGraph from historical versions to 3.4.1:
- Upgrade it to the latest 2.5 version according to the docs of that version.
- Follow this topic to upgrade it to 3.4.1.
To upgrade NebulaGraph from versions earlier than 2.0.0 (including the 1.x versions) to 3.4.1, you need to find the
date_time_zonespec.csv in the
share/resources directory of 3.4.1 files, and then copy it to the same directory in the NebulaGraph installation path.
- Rolling Upgrade is not supported. You must stop all the NebulaGraph services before the upgrade.
- There is no upgrade script. You have to manually upgrade each server in the cluster.
- This topic does not apply to scenarios where NebulaGraph is deployed with Docker, including Docker Swarm, Docker Compose, and K8s.
- You must upgrade the old NebulaGraph services on the same machines they are deployed. DO NOT change the IP addresses, configuration files of the machines, and DO NOT change the cluster topology.
- Known issues that could cause data loss are listed on GitHub known issues. The issues are all related to altering schema or default values.
- DO NOT use soft links to switch the data directories.
- You must have the sudo privileges to complete the steps in this topic.
After the upgrade, you will not be able to connect to NebulaGraph from old clients. You will need to upgrade all clients to a version compatible with NebulaGraph 3.4.1.
A few configuration parameters have been changed. For more information, see the release notes and configuration docs.
The nGQL syntax is partially incompatible:
- Disable the
YIELDclause to return custom variables.
YIELDclause is required in the
- It is required to specify a tag to query properties of a vertex in a
MATCHstatement. For example, from
- Disable the
Before upgrading a NebulaGraph cluster with full-text indexes deployed, you must manually delete the full-text indexes in Elasticsearch, and then run the
SIGN INcommand to log into ES and recreate the indexes after the upgrade is complete. To manually delete the full-text indexes in Elasticsearch, you can use the curl command
curl -XDELETE -u <es_username>:<es_password> '<es_access_ip>:<port>/<fullindex_name>', for example,
curl -XDELETE -u elastic:elastic 'http://192.168.8.xxx:9200/nebula_index_2534'. If no username and password are set for Elasticsearch, you can omit the
There may be other undiscovered influences. Before the upgrade, we recommend that you read the release notes and user manual carefully, and keep an eye on the posts on the forum and issues on Github.
Preparations before the upgrade¶
Download the package of NebulaGraph 3.4.1 according to your operating system and system architecture. You need the binary files during the upgrade. Find the package on the download page.
You can also get the new binaries from the source code or the RPM/DEB package.
Locate the data files based on the value of the
data_pathparameters in the Storage and Meta configurations, and backup the data files. The default paths are
The old data will not be automatically backed up during the upgrade. You must manually back up the data to avoid data loss.
- Backup the configuration files.
Collect the statistics of all graph spaces before the upgrade. After the upgrade, you can collect again and compare the results to make sure that no data is lost. To collect the statistics:
SUBMIT JOB STATS.
SHOW JOBSand record the result.
Stop all NebulaGraph services.
<nebula_install_path>/scripts/nebula.service stop all
nebula_install_pathindicates the installation path of NebulaGraph.
The storaged progress needs around 1 minute to flush data. You can run
nebula.service status allto check if all services are stopped. For more information about starting and stopping services, see Manage services.
If the services are not fully stopped in 20 minutes, stop upgrading and ask for help on the forum or Github.
Starting from version 3.0.0, it is possible to insert vertices without tags. If you need to keep vertices without tags, add
--graph_use_vertex_key=truein the configuration file (
nebula-graphd.conf) of all Graph services within the cluster; and add
--use_vertex_key=truein the configuration file (
nebula-storaged.conf) of all Storage services."
In the target path where you unpacked the package, use the binaries in the
bindirectory to replace the old binaries in the
bindirectory in the NebulaGraph installation path.
Update the binary of the corresponding service on each NebulaGraph server.
Modify the following parameters in all Graph configuration files to accommodate the value range of the new version. If the parameter values are within the specified range, skip this step.
- Set a value in [1,604800] for
session_idle_timeout_secs. The recommended value is 28800.
- Set a value in [1,604800] for
client_idle_timeout_secs. The recommended value is 28800.
The default values of these parameters in the 2.x versions are not within the range of the new version. If you do not change the default values, the upgrade will fail. For detailed parameter description, see Graph Service Configuration.
- Set a value in [1,604800] for
Start all Meta services.
Once started, the Meta services take several seconds to elect a leader.
To verify that Meta services are all started, you can start any Graph server, connect to it through NebulaGraph Console, and run
SHOW HOSTS metaand
SHOW META LEADER. If the status of Meta services are correctly returned, the services are successfully started.
Start all the Graph and Storage services.
Connect to the new version of NebulaGraph to verify that services are available and data are complete. For how to connect, see Connect to NebulaGraph.
Currently, there is no official way to check whether the upgrade is successful. You can run the following reference statements to test the upgrade:
nebula> SHOW HOSTS; nebula> SHOW HOSTS storage; nebula> SHOW SPACES; nebula> USE <space_name> nebula> SHOW PARTS; nebula> SUBMIT JOB STATS; nebula> SHOW STATS; nebula> MATCH (v) RETURN v LIMIT 5;
You can also test against new features in version 3.4.1.
Upgrade failure and rollback¶
If the upgrade fails, stop all NebulaGraph services of the new version, recover the old configuration files and binaries, and start the services of the old version.
All NebulaGraph clients in use must be switched to the old version.
Can I write through the client during the upgrade?¶
A: No. You must stop all NebulaGraph services during the upgrade.
Space 0 not found warning message during the upgrade process¶
Space 0 not found warning message appears during the upgrade process, you can ignore it. The space
0 is used to store meta information about the Storage service and does not contain user data, so it will not affect the upgrade.
How to upgrade if a machine has only the Graph Service, but not the Storage Service?¶
A: You only need to update the configuration files and binaries of the Graph Service.
How to resolve the error
A: Try again with the sudo privileges.
Is there any change in gflags?¶
A: Yes. For more information, see the release notes and configuration docs.
Is there a tool or solution for verifying data consistency after the upgrade?¶
A: No. But if you only want to check the number of vertices and edges, run
SUBMIT JOB STATS and
SHOW STATS after the upgrade, and compare the result with the result that you recorded before the upgrade.
How to solve the issue that Storage is
Leader count is
A: Run the following statement to add the Storage hosts into the cluster manually.
ADD HOSTS <ip>:<port>[, <ip>:<port> ...];
ADD HOSTS 192.168.10.100:9779, 192.168.10.101:9779, 192.168.10.102:9779;
If the issue persists, ask for help on the forum or GitHub.
Why the job type changed after the upgrade, but job ID remains the same?¶
SHOW JOBS depends on an internal ID to identify job types, but in NebulaGraph 2.5.0 the internal ID changed in this pull request, so this issue happens after upgrading from a version earlier than 2.5.0.