Skip to content

Job manager and the JOB statements

The long-term tasks running by the Storage Service are called jobs. For example, there are jobs for COMPACT, FLUSH, and STATS. These jobs can be time-consuming if the data size in the graph space is large. The job manager helps you run, show, stop, and recover the jobs.

SUBMIT JOB COMPACT

The SUBMIT JOB COMPACT statement triggers the long-term RocksDB compact operation.

nebula> SUBMIT JOB COMPACT;
+------------+
| New Job Id |
+------------+
| 40         |
+------------+

For more information about compact configuration, see Storage Service configuration.

SUBMIT JOB FLUSH

The SUBMIT JOB FLUSH statement writes the RocksDB memfile in memory to the hard disk.

nebula> SUBMIT JOB FLUSH;
+------------+
| New Job Id |
+------------+
| 96         |
+------------+

SUBMIT JOB STATS

The SUBMIT JOB STATS statement starts a job that makes the statistics of the current graph space. Once this job succeeds, you can use the SHOW STATS statement to list the statistics. For more information, see SHOW STATS.

Note

If the data stored in the graph space changes, in order to get the latest statistics, you have to run SUBMIT JOB STATS again.

nebula> SUBMIT JOB STATS;
+------------+
| New Job Id |
+------------+
| 97         |
+------------+

SHOW JOB

The Meta Service parses a SUBMIT JOB request into tasks and assigns them to the nebula-storaged processes. The SHOW JOB <job_id> statement shows the information about a specific job and all its tasks.

The job ID is created when you run the SUBMIT JOB statement.

nebula> SHOW JOB 96;
+----------------+---------------+------------+-------------------------+-------------------------+
| Job Id(TaskId) | Command(Dest) | Status     | Start Time              | Stop Time               |
+----------------+---------------+------------+-------------------------+-------------------------+
| 96             | "FLUSH"       | "FINISHED" | 2020-11-28T14:14:29.000 | 2020-11-28T14:14:29.000 |
+----------------+---------------+------------+-------------------------+-------------------------+
| 0              | "storaged2"   | "FINISHED" | 2020-11-28T14:14:29.000 | 2020-11-28T14:14:29.000 |
+----------------+---------------+-------------------------+------------+-------------------------+
| 1              | "storaged0"   | "FINISHED" | 2020-11-28T14:14:29.000 | 2020-11-28T14:14:29.000 |
+----------------+---------------+------------+-------------------------+-------------------------+
| 2              | "storaged1"   | "FINISHED" | 2020-11-28T14:14:29.000 | 2020-11-28T14:14:29.000 |
+----------------+---------------+------------+-------------------------+-------------------------+

The description of the return message is as follows.

Column Description
Job Id(TaskId) The first row shows the job ID, and the other rows show the task IDs.
Command(Dest) The first row shows the command executed, and the other rows show on which storaged processes the task is running.
Status Shows the status of the job or task. For more information about job status, see Job status.
Start Time Shows a timestamp indicating the time when the job or task enters the RUNNING phase.
Stop Time Shows a timestamp indicating the time when the job or task gets FINISHED, FAILED, or STOPPED.

Job status

The description of the job status is as follows.

Status Description
QUEUE The job or task is waiting in a queue. The Start Time is empty in this phase.
RUNNING The job or task is running. The Start Time shows the beginning of this phase.
FINISHED The job or task is successfully finished. The Stop Time shows the time when the job or task enters this phase.
FAILED The job or task failed. The Stop Time shows the time when the job or task enters this phase.
STOPPED The job or task is stopped without running. The Stop Time shows the time when the job or task enters this phase.
REMOVED The job or task is removed.

Status switching is described as follows.

Queue -- running -- finished -- removed
     \          \                /
      \          \ -- failed -- /
       \          \            /
        \ ---------- stopped -/

SHOW JOBS

The SHOW JOBS statement lists all the unexpired jobs. The default job expiration interval is one week. You can change it by modifying the job_expired_secs parameter of the Meta Service. For how to modify job_expired_secs, see Meta Service configuration.

nebula> SHOW JOBS;
+--------+----------------------+------------+-------------------------+-------------------------+
| Job Id | Command              | Status     | Start Time              | Stop Time               |
+--------+----------------------+------------+-------------------------+-------------------------+
| 97     | "STATS"              | "FINISHED" | 2020-11-28T14:48:52.000 | 2020-11-28T14:48:52.000 |
+--------+----------------------+------------+-------------------------+-------------------------+
| 96     | "FLUSH"              | "FINISHED" | 2020-11-28T14:14:29.000 | 2020-11-28T14:14:29.000 |
+--------+----------------------+------------+-------------------------+-------------------------+
| 95     | "STATS"              | "FINISHED" | 2020-11-28T13:02:11.000 | 2020-11-28T13:02:11.000 |
+--------+----------------------+------------+-------------------------+-------------------------+
| 86     | "REBUILD_EDGE_INDEX" | "FINISHED" | 2020-11-26T13:38:24.000 | 2020-11-26T13:38:24.000 |
+--------+----------------------+------------+-------------------------+-------------------------+

STOP JOB

The STOP JOB statement stops jobs that are not finished.

nebula> STOP JOB 22;
+---------------+
| Result        |
+---------------+
| "Job stopped" |
+---------------+

RECOVER JOB

The RECOVER JOB statement re-executes the failed jobs and returns the number of recovered jobs.

nebula> RECOVER JOB;
+-------------------+
| Recovered job num |
+-------------------+
| 5 job recovered   |
+-------------------+

FAQ

How to troubleshoot job problems

The SUBMIT JOB operations use the HTTP port. Please check if the HTTP ports on the machines where the Storage Service is running are working well. You can use the following command to debug.

curl "http://{storaged-ip}:19779/admin?space={space_name}&op=compact"

Last update: April 22, 2021