Jobs

The Jobs page is your operational hub for managing all data processing activities in your model. Here you can see everything that has run, is running, or is scheduled to run in the future. It's like a dashboard for all your data operations.

Page overview

┌───────────────────────────────────────────────────────────────┐
│ Jobs Page                                                     │
│ ┌─────────────┬────────────┐   ┌────────────────────────────┐ │
│ │ Last Jobs ▣ │ Scheduled  │   │ New Job ⊕ • Sort ▾         │ │
│ └─────────────┴────────────┘   └────────────────────────────┘ │
│ ┌───────────────────────────────────────────────────────────┐ │
│ │ Status | Name | Type | Rule | Created By | Info | Actions │ │
│ │   ●    | tap→sink dump                                    │ │
│ │   ○    | nightly discover                                 │ │
│ └───────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────┘

Jobs table columns

Status indicators

When a job counts as completed

A job is completed only when every entity it had to process is completed in the sink. It ends failed, never completed, when:

• an entity failed, or was still running when the job ended
• the job ended with an error (for example, the connection to the source or the sink was lost)
• the job stalled: no progress and no heartbeat for jobs.stallTimeoutSec seconds (900 by default, 0 turns the check off). The log shows job stalled: no progress for N seconds
• the job was canceled or interrupted (see below)

Continue streaming on fail (Rule panel of the New Job modal, on by default) only decides whether the other entities go on after one fails. With it on, the remaining entities still run, and the job still ends failed so you can restart it for the entities that failed. With it off, the job stops at the first failure.

The server settings named here are described in Jobs configuration.

Working with jobs

Starting new jobs

The New Job button (+) in the toolbar opens the New Job modal, which is the primary interface for creating new data processing jobs. The modal provides a comprehensive configuration interface with the following capabilities:

Key features of the New Job modal:

• From panel: Choose your source (tap or dataset) with environment and driver information
• To panel: Route data to sinks, taps, or create new datasets
• Rule panel: Apply optional transformation or anonymization rules
• Load options: Fine-tune batch sizes, write modes, and performance settings
• Schedule panel: Choose between immediate execution, one-time scheduling, or recurring pipelines

How to use:

1. Click the New Job (+) button in the top-right toolbar
2. Follow the step-by-step panels to configure your data flow
3. Choose to run immediately (Run Now), schedule for later (Run Later), or save as a reusable pipeline
4. Review the collapsed panel summaries to ensure all required fields are configured

For detailed information about all available options and configuration settings, see the complete New Job modal documentation.

Managing existing jobs

• Click any job name to view its details and logs
• Use the Actions menu (⋮) to perform context-appropriate actions like:

Cancel jobs that are queued or running

Sometimes you may need to stop a job that's currently running or waiting in the queue. This is useful when:

• You've started a job by mistake
• You realize you need to make changes to the configuration before proceeding
• The job is taking longer than expected and blocking other operations
• You've identified an issue that makes the job unnecessary

What happens when you cancel:

• A queued or scheduled job does not run.
• A running job stops reading and writing, then waits for its finishing steps to restore the sink (for example, to put back the constraints and triggers it disabled) before it ends. On large tables this can take a while. The wait ends when those steps are done, when they send no heartbeat for jobs.stallTimeoutSec seconds, or after jobs.finishTimeoutSec seconds (24 hours by default). The job then ends failed with the reason job canceled.
• If you cannot wait and the job is still shown as running, cancel it again: it ends right away, without waiting for those steps. They may still be running in the background, so check the sink before you restart the job.

Important: Canceling does not undo what the job already wrote. The entities it completed stay in the sink and the ones in progress are left partly loaded. Restart the job to finish them, or rerun it.

Restart jobs that failed

Restart is available for failed dump, pump and load jobs that are no longer running. It runs the same job again, with the same name and settings, and goes on from where the last run stopped:

• Entities that completed are skipped.
• The other entities resume after the last batch committed to the sink (their checkpoint), without truncating the sink, when the number of rows in the sink matches what the job recorded. The log shows resuming <entity> after record N.
• Entities that cannot resume start over from the beginning, and the log says why (cannot resume <entity>, it starts over: ...). This happens, for example, when the sink count does not match, or when the table was not read in a stable order. See Large databases for which tables can resume.
• An entity loaded in Append mode that cannot resume fails instead of starting over, because it would load again the rows the last run wrote. Empty it, or load it with Truncate.
• The job keeps its dictionary: it is not overwritten, so values masked in the earlier run keep the same masked value.
• A dump goes on writing to the dataset the first run opened, so the entities it finished stay there.
• A load or pump over its own tap with a rollback backup reuses the backup the first run made, which holds the tables as they were before the job.

A job whose worker is still alive cannot be restarted. If you just canceled it and it is still restoring the sink, the restart is refused with "The job is still running: cancel it and wait for it to stop before restarting it". Wait for it to end and try again.

Rerun jobs that completed

Rerun creates a new job that starts over completely from the beginning, reusing all the original job settings. Nothing is resumed from the earlier run: a dump writes to its dataset as it was configured (a new one, unless it overwrites or merges into an existing dataset), and a load over its own tap makes a new rollback backup. Why would you want to do this?

• Repeat the same operation: Run the exact same scan, discovery, or data processing again with the same parameters
• Refresh data: Get updated results based on the current state of your data sources
• Test consistency: Verify that the job produces the same results when run multiple times
• Apply to new data: If your data source has been updated, rerun the job to process the new information

This is an efficient way to repeat operations without having to reconfigure all the settings.

Like a restart, a rerun is refused while the worker of the earlier run is still alive, for example while a canceled job restores the sink: "The job is still running: cancel it and wait for it to stop before running it again".

Delete job history

Remove completed jobs from your job history list. This action:

• Cleans up your job list for better organization
• Removes old jobs that are no longer relevant
• Helps maintain a focused view of recent and active operations

Deleting job history only removes the record from this list - it doesn't affect any data that was processed or created by the job itself.

Download rule configuration as YAML

Export the rule configuration used in a job as a YAML file. This is helpful when:

• You want to share job configurations with team members
• You need to send configuration details to Gigantics support for debugging
• You want to replicate an issue or specific job configuration
• You need to audit or document the rules applied to your data
• You're migrating configurations between environments

Interrupted jobs

A running job belongs to the job worker that runs it. While the job runs, the worker renews a lease on it every jobs.heartbeatSec seconds (15 by default). If the worker stops, the job is ended as interrupted:

By default an interrupted job ends failed, and you can restart it. With jobs.autoResume turned on, interrupted dump, pump and load jobs are queued to resume from their checkpoints instead, as Restart would, up to jobs.maxAutoResumes times (3 by default). After that they end failed. The log shows job interrupted (<reason>), resuming. Other jobs, such as pipelines, discover and scan jobs, always end failed.

The check for dead workers runs in the job workers. If the only server stops, its jobs are ended when it starts again. See Jobs configuration for these settings.

Scheduled jobs

Jobs can be scheduled to run automatically in several ways:

Scheduled jobs appear in the Scheduled Jobs tab until they run, making it easy to see what's coming up.

Job details

Click a job name to see its summary, the progress of each entity and the job log.

Summary

The Summary section shows the type, rule, status and overall progress of the job, and these time rows:

While the job runs, End and Elapsed have a help icon with the tooltips "Estimated end time based on current progress" and "Estimated total time based on current progress". The estimate is shown only when the job has passed 1% of progress and its estimated total time is over 30 seconds. Before that, End shows - and Elapsed shows only the time so far.

The overall progress is the records read by all the entities of the job against the sum of their totals, and it stays at 99% at most until the job ends. The estimates assume that the rest of the job goes at the same pace: the estimated total time is the time so far multiplied by the total records and divided by the records read. They are counted in records, not in time per entity, so they are rough when the tables are very different in size or speed. A failed entity keeps its total in the progress, so a job with failed entities never reaches 100%.

After a Restart, Start and Elapsed are those of the last run, while the progress includes the records the earlier runs already read.

Entity progress

Each entity shows the records read against its total, with a progress bar that turns red when the entity fails. Inside the job, each entity is running, completed or failed. An entity is completed only once its last batch is committed to the sink, not when its last record is read.

The total of an entity can be an estimate from the database statistics, so the percentage is approximate. For filtered reads (a where or a custom query) of tables under 5 million rows, the job also runs an exact count: alongside the stream when the driver has a spare connection, before it otherwise. This cannot be changed in the rules of Gigantics: the exactCount option (auto, always, never) only works in the rules of a debaser.yml file used with the debaser engine directly. See Advanced read options.

Job log

The log opens with the latest 2,000 lines of the job. When there are more, a Load earlier lines button at the top of the log loads the 2,000 lines before them. Click it again to go further back. While the job runs, new lines keep arriving at the bottom.

Download log reads the whole log from the server, not only the lines loaded in the view, with the log levels selected.

What the log keeps depends on two server settings (see Jobs configuration):

• jobs.logRetentionDays: log lines older than this number of days are deleted, even those of a job still running. When it is not set, jobLogRetentionDays applies (90 by default); 0 keeps them forever.
• jobs.logTraceLevel: the trace lines saved to the log of a job whose pipeline has no log level. With the default 1, the SQL of each query (level 3) and the lines of each batch (level 4) are not saved.

Shard workers

When other job workers help a pump stream the ranges of its large tables (jobs.shardWorkers, see Large databases), each helper runs as a job of its own that is hidden from the job lists. Its lines appear in the log of the job it helps, prefixed [shard worker N], and that job's progress adds up what the helpers stream. If a helper dies, its part fails and the job ends failed, so you can restart it.

Where jobs originate

Jobs come from various actions in the platform:

• Discover — Scanning your data sources for sensitive information
• Rules — Data anonymization or transformation operations
• Datasets — Data export, copy, or merge operations
• Sinks — Loading processed data to destinations
• Pipelines — Automated sequences of jobs

Whenever you configure one of these operations, if it's scheduled to run in the future, it will appear in your Scheduled Jobs tab.