Skip to content

Upgrade NebulaGraph to v2.6.2

The legacy versions of NebulaGraph refer to the versions lower than NebulaGraph v2.0.0-GA. This topic describes how to upgrade NebulaGraph to v2.6.2.

Note

To upgrade NebulaGraph v2.0.0-GA or later versions to v2.6.2, see NebulaGraph v2.0.x to v2.6.2.

Limitations

  • Rolling Upgrade is not supported. You must stop the NebulaGraph services before the upgrade.
  • There is no upgrade script. You have to manually upgrade each server in the cluster.
  • This topic does not apply to scenarios where NebulaGraph is deployed with Docker, including Docker Swarm, Docker Compose, and K8s.
  • You must upgrade the old NebulaGraph services on the same machines they are deployed. DO NOT change the IP addresses, configuration files of the machines, and DO NOT change the cluster topology.
  • The hard disk space of each machine should be three times as much as the space taken by the original data directories.
  • Known issues that could cause data loss are listed on GitHub known issues. The issues are all related to altering schema or default values.
  • To connect to NebulaGraph 2.0.0, you must upgrade all the NebulaGraph clients. The communication protocols of the old versions and the latest versions are not compatible.
  • DO NOT use soft links to switch the data directories.
  • You must have the sudo privileges to complete the steps in this topic.

Installation paths

Old installation path

By default, old versions of NebulaGraph are installed in /usr/local/nebula/, hereinafter referred to as ${nebula-old}. The default configuration file path is ${nebula-old}/etc/.

  • Storaged data path is defined by the --data_path option in the ${nebula-old}/etc/nebula-storaged.conf file. The default path is data/storage.
  • Metad data path is defined by the --data_path option in the ${nebula-old}/etc/nebula-metad.conf file. The default path is data/meta.

Note

The actual paths in your environment may be different from those described in this topic. You can run the Linux command ps -ef | grep nebula to locate them.

New installation path

${nebula-new} represents the installation path of the new NebulaGraph version, such as /usr/local/nebula-new/.

# mkdir -p ${nebula-new}

Upgrade steps

  1. Stop all client connections. You can run the following commands on each Graph server to turn off the Graph Service and avoid dirty write.

    # ${nebula-old}/scripts/nebula.service stop graphd
[INFO] Stopping nebula-graphd...
[INFO] Done

  2. Run the following commands to stop all services of the old version NebulaGraph.

    # ${nebula-old}/scripts/nebula.service stop all
[INFO] Stopping nebula-metad...
[INFO] Done
[INFO] Stopping nebula-graphd...
[INFO] Done
[INFO] Stopping nebula-storaged...
[INFO] Done

    The storaged process needs about 1 minute to flush data. Wait 1 minute and then run ps -ef | grep nebula to check and make sure that all the NebulaGraph services are stopped.

    Note

    If the services are not fully stopped in 20 minutes, stop upgrading and go to the NebulaGraph community for help.

  3. Install the new version of NebulaGraph on each machine.

    1. Install the new binary file.

      • To install with RPM/DEB packages, download the installation package of the corresponding operating system from release page.

        # sudo rpm --force -i --prefix=${nebula-new}  ${nebula-package-name.rpm} # for centos/redhat
# sudo dpkg -i --instdir==${nebula-new} ${nebula-package-name.deb} # for ubuntu

        For detailed steps, see Install NebulaGraph with RPM or DEB package.

      • To install with the source code, follow the substeps. For detailed steps, see Install NebulaGraph by compiling the source code. Some key commands are as follows.

        • Clone the source code.

          # git clone --branch v2.6.2 https://github.com/vesoft-inc/nebula-graph.git

        • Configure CMake.

          # cmake -DCMAKE_INSTALL_PREFIX=${nebula-new} -DENABLE_TESTING=OFF -DCMAKE_BUILD_TYPE=Release ..

    2. Copy the configuration files from the old path to the new path.

      # cp -rf ${nebula-old}/etc ${nebula-new}/

  4. Follow the substeps to prepare the Meta servers (usually 3 of them in a cluster).

    • Locate the old Meta data path and copy the data files to the new path.

      Find the --data_path option in ${nebula-old}/etc/nebula-metad.conf. The default value is data/meta.

      • If the legacy versions has not changed the --data_path item, run the following command to copy the meta data to the new directory.

        # mkdir -p ${nebula-new}/data/meta/
# cp -r ${nebula-old}/data/meta/* ${nebula-new}/data/meta/
      • If the legacy versions change the default metad directory, copy it according to the actual directory.

    • Modify the new Meta configuration files.

      • Edit the new metad configuration file.

        # vim ${nebula-new}/nebula-metad.conf

      • [Optional]Add the following parameters in the Meta configuration files if you need them.

        --null_type=false: Disables the support for using NULL.The default value is true. When set to false, you must specify a default value when altering tags or edge types, otherwise, data reading fails.

        --string_index_limit=32: Specifies the index length for string values as 32. The default length is 64.

      Note

      You must make sure that this step is applied on every Meta server.

  5. Prepare the Storage configuration files on each Storage server.

    • [Optional]If the old Storage data path is not the default setting --data_path=data/storage, modify it.

      # vim ${nebula-new}/nebula-storaged.conf
      Change the value of --data_path as the new data path.

    • Create the new Storage data directories.
      # mkdir -p ${nebula-new}/data/storage/

    If the --data_path default value has been modified, create the Storage data directories according to the modification.

  6. Start the new Meta Service.

    • Run the following command on each Meta server.
      # ${nebula-new}/scripts/nebula.service start metad
[INFO] Starting nebula-metad...
[INFO] Done
    • Check if every nebula-metad process is started normally.
      # ps -ef |grep nebula-metad
    • Check if there is any error information in the Meta logs in ${nebula-new}/logs.

    Note

    If any nebula-metad process cannot start normally, stop upgrading, start the NebulaGraph services from the old directories, and take the error logs to the NebulaGraph community for help.

  7. Run the following commands to upgrade the Storage data format.

    # ${nebula-new}/bin/db_upgrader  \
--src_db_path=<old_storage_directory_path> \
--dst_db_path=<new_storage_directory_path>  \
--upgrade_meta_server=<meta_server_ip1>:<port1>[,<meta_server_ip2>:<port2>,...] \
--upgrade_version=<old_nebula_version> \

    The parameters are described as follows.

    • --src_db_path: Specifies the absolute path of the OLD Storage data directories. Separate multiple paths with commas, without spaces.
    • --dst_db_path: Specifies the absolute path of the NEW Storage data directories. Separate multiple paths with commas, without spaces. The paths must correspond to the paths set in --src_db_path one by one.
    • --upgrade_meta_server: Specifies the addresses of the new Meta servers that you started in step 6.
    • --upgrade_version: If the old NebulaGraph version is v1.2.0, set the parameter value to 1. If the old version is v2.0.0-RC1, set the value to 2. Do not set the value to other numbers.

    Danger

    Do not mix up the order of --src_db_path and --dst_db_path. Otherwise, the old data will be damaged during the upgrade.

    For example, upgrade from v1.2.x:

    # /usr/local/nebula_new/bin/db_upgrader \
--src_db_path=/usr/local/nebula/data/storage/data1/,/usr/local/nebula/data/storage/data2/ \
--dst_db_path=/usr/local/nebula_new/data/storage/data1/,/usr/local/nebula_new/data/storage/data2/\
--upgrade_meta_server=192.168.*.14:45500,192.168.*.15:45500,192.168.*.16:45500 \
--upgrade_version=1

    For example, upgrade from v2.0.0-RC1:

    # /usr/local/nebula_new/bin/db_upgrader \
--src_db_path=/usr/local/nebula/data/storage/ \
--dst_db_path=/usr/local/nebula_new/data/storage/ \
--upgrade_meta_server=192.168.*.14:9559,192.168.*.15:9559,192.168.*.16:9559 \
--upgrade_version=2

    Note

    • If anything goes wrong, Stop upgrading, stop all the Meta servers, and start the NebulaGraph services from the old directories.
    • Make sure that all the Storage servers have finished the upgrade.

  8. Start the new Storage Service on each Storage server.

    # ${nebula-new}/scripts/nebula.service start storaged
# ${nebula-new}/scripts/nebula.service status storaged

    Note

    If this step goes wrong on any server, Take the logs in ${nebula-new}/logs/ to the NebulaGraph community for help. Stop upgrading. Stop all the Meta servers and Storage servers. Start the NebulaGraph services from the old directories.

  9. Start the new Graph Service on each Graph server.

    # ${nebula-new}/scripts/nebula.service start graphd
# ${nebula-new}/scripts/nebula.service status graphd

    Note

    If this step goes wrong on any server, take the logs in ${nebula-new}/logs/ to the NebulaGraph community for help. Stop upgrading. Stop all the Meta servers, Storage servers, and Graph servers. Start the NebulaGraph services from the old directories.

  10. Connect to NebulaGraph with the new versions of Nebula Console. Verify if the NebulaGraph services are available and if the data can be accessed normally. Make sure that the command parameters, including the IP address and port of the Graph Service, are the same as the old one.

    nebula> SHOW HOSTS;
nebula> SHOW SPACES;
nebula> USE <space_name>
nebula> SHOW PARTS;
nebula> SUBMIT JOB STATS;
nebula> SHOW STATS;

    Note

    The old releases of Nebula Console may have compatibility issues.

  11. Upgrade other NebulaGraph clients.

    You must upgrade all other clients to corresponding NebulaGraph v2.6.2. The clients include but are not limited to Python, Java, go, C++, Flink-connector, Algorithm, Exchange, Spark-connector, and Nebula Bench. Find the v2.6.2 branch for each client.

    Note

    Communication protocols of v2.6.2 are not compatible with that of the old releases. To upgrade the clients, compile the v2.6.2 source code of the clients or download corresponding binaries.

    Tip for maintenance: The data path after the upgrade is ${nebula-new}/. Modify relative paths for hard disk monitor systems, log, or ELK, etc.

Upgrade failure and rollback

If the upgrade fails, stop all NebulaGraph services of the new version, and start the services of the old version.

All NebulaGraph clients in use must be switched to the old version.

Appendix 1: Test Environment

The test environment for this topic is as follows:

  • Machine specifications: 32 CPU cores, 62 GB memory, and SSD.
  • Data size: 100 GB of NebulaGraph 1.2.0 LDBC test data, with 1 graph space, 24 partitions, and 92 GB of data directory size.
  • Concurrent configuration: --max_concurrent=5, --max_concurrent_parts=24, and --write_batch_num=100.

The upgrade cost 21 minutes in all, including 13 minutes of compaction. The descriptions are as follows.

Parameter Default value
--max_concurrent 5
--max_concurrent_parts 10
--write_batch_num 100

Appendix 2: NebulaGraph V2.0.0 code address and commit ID

Code address Commit ID
graphd 91639db
storaged and metad 761f22b
common b2512aa

FAQ

Can I write through the client during the upgrade?

A: No. The state of the data written during this process is undefined.

Can I upgrade other old versions except for v1.2.x and v2.0.0-RC to v2.6.2?

A: Upgrading from other old versions is not tested. Theoretically, versions between v1.0.0 and v1.2.0 could adopt the upgrade approach for v1.2.x. v2.0.0-RC nightly versions cannot apply the solutions in this topic.

How to upgrade if a machine has only the Graph Service, but not the Storage Service?

A: Upgrade the Graph Service with the corresponding binary or rpm package.

How to resolve the error Permission denied?

A: Try again with the sudo privileges.

Is there any change in gflags?

A: Yes. For more information, see github issues.

What are the differences between deleting data then installing the new version and upgrading according to this topic?

A: The default configurations for v2.x and v1.x are different, including the ports used. The upgrade solution keeps the old configurations, and the delete-and-install solution uses the new configurations.

Is there a tool or solution for verifying data consistency after the upgrade?

A: No.

Last update: March 13, 2023