Jobs Configuration

The jobs section of the configuration file controls how the job workers run pump, load and dump jobs: how they detect a job that stopped making progress or whose worker died, whether those jobs resume on their own, how many pipelines a job streams side by side, and how much of the job log is kept.

Every property has a default, so the section can be left out or contain only the properties you want to change. These settings matter most for long jobs on large databases. See Working with Large Databases for the whole picture.

Configuration Properties

The jobs section in your config/default.yaml file, with its default values:

jobs:
  stallTimeoutSec: 900
  finishTimeoutSec: 86400
  heartbeatSec: 15
  leaseStaleSec: 300
  autoResume: false
  maxAutoResumes: 3
  concurrent: 1
  shardWorkers: 0
  copy: true
  logRetentionDays: 0
  logTraceLevel: 2

Property Details

stallTimeoutSec

• Type: Integer (minimum 0)
• Default: 900 (15 minutes)
• Description: A running job that reports no progress, no heartbeat and no log line for this many seconds is stopped and ends failed with the message job stalled: no progress for N seconds. 0 turns the check off.
• When to change it: Long operations that do not move records (counting a table, rebuilding constraints, taking a snapshot, saving the dictionary) send heartbeats, so the default rarely needs to change. Raise it if your source has queries that take longer than 15 minutes to return their first rows (for example a custom query with a large sort). Do not turn it off: a silent network drop can leave a job waiting forever.

finishTimeoutSec

• Type: Integer (minimum 0)
• Default: 86400 (24 hours)
• Description: When a job is stopped by an error, its run still restores the sink before the job ends (for example, it re-creates the constraints and re-enables the triggers it disabled). The job waits at most this many seconds for that work. It also stops waiting when that work sends no heartbeat for stallTimeoutSec seconds. 0 waits as long as it takes.
• When to change it: Rebuilding the constraints of very big tables can take hours. Raise it if restoring your sink can take longer than a day, so the job does not end (and cannot be restarted) while the sink is still being restored.

heartbeatSec

• Type: Integer (minimum 1)
• Default: 15
• Description: How often, in seconds, a running job renews its lease in MongoDB. See Leases and heartbeats.
• When to change it: Rarely. Lower values detect dead workers on other hosts sooner only if you lower leaseStaleSec too. leaseStaleSec must be at least 3 times this value.

leaseStaleSec

• Type: Integer (minimum 30, and at least 3 × heartbeatSec)
• Default: 300 (5 minutes)
• Description: A lease not renewed for this many seconds belongs to a dead worker: the stale job sweep ends the job, or resumes it with autoResume. A worker that cannot renew its own lease for this long stops its job.
• When to change it: Lower it to detect sooner the workers that die on another host (a worker on the same host is detected by its process id within a minute). Raise it if MongoDB can be unreachable for minutes at a time, so a slow MongoDB is not taken for a dead worker.

autoResume

• Type: Boolean
• Default: false
• Description: When true, a pump, load or dump job whose worker died or was stopped is queued to run again at once and resume from its checkpoints, instead of ending failed. It works like the Restart action of the job: finished entities are skipped and the others continue where they were left (see Resuming a failed or interrupted job).
• When to change it: Turn it on for long jobs on large databases, where a server restart, a deployment or a killed process should not throw away hours of work.

maxAutoResumes

• Type: Integer (minimum 0)
• Default: 3
• Description: How many times a job is resumed automatically. After that, the next interruption fails the job. The count is kept across the runs of the job; a Rerun starts it again.
• When to change it: Raise it for jobs that last days on infrastructure that restarts often.

concurrent

• Type: Integer (minimum 1)
• Default: 1
• Description: The least number of pipelines (reader, transform and writer) a job streams side by side, whatever the Concurrency of its options says: the job uses the larger of the two. The pipelines are shared by the entities streamed in parallel and by the ranges of a big table.
• When to change it: Raise it to give every job of the instance parallel reads without editing the options of each rule or job. Every pipeline holds connections to the source and the sink: keep it within what your databases accept.

shardWorkers

• Type: Integer (minimum 0)
• Default: 0 (off)
• Description: How many other job workers of the same host help a pump job stream the ranges of its big tables. See Shard helper workers.
• When to change it: Set it when a single job worker process (one Node.js process) is the bottleneck of big pumps. The host must run enough job workers: see start (-j).

copy

• Type: Boolean
• Default: true
• Description: Postgres sinks load every batch with COPY FROM STDIN instead of INSERT statements. See COPY writes for Postgres. It applies to every Postgres sink of the instance, except a sink with copy: false in its Custom Params, which always uses INSERT.
• When to change it: Set it to false only if COPY gives trouble with your Postgres setup (for example a proxy or pooler that does not support it).

logRetentionDays

• Type: Integer (minimum 0)
• Default: 0 (kept forever)
• Description: Days the lines of the job logs are kept in MongoDB. A TTL index removes older lines. Lines of a job that runs longer than this are removed too, while the job runs.
• When to change it: Set it when the job logs make the MongoDB database grow too much, as with many long jobs on large databases.

logTraceLevel

• Type: Integer (minimum 0)
• Default: 2
• Description: The most detailed trace lines saved to the job log. Level 3 adds the SQL of every query, and level 4 adds the lines of every batch. With the default 2 those are left out.
• When to change it: Raise it to 3 or 4 for a while to troubleshoot a job. On large tables these levels write lines per batch and per query, which can mean millions of lines.

How It Works

Leases and heartbeats

A job worker that starts a job takes a lease on it, a MongoDB document with the owner of the job (host and process id) and an epoch number. While the job runs the worker renews the lease every heartbeatSec seconds.

The epoch fences the job: every write of the run (job state, checkpoints) checks that it still has the lease epoch it started with. If another worker takes the job over, the old run sees its lease lost and stops without writing to the job again. Two workers never write the same job.

worker A   take lease (epoch 7)
           |--beat--|--beat--|--beat--|    X  process killed
                                             :
sweep                                        :  ...no beat for leaseStaleSec
                                             :  (or pid gone on the same host)
                                             v
                                       epoch 8: job failed, or queued to
                                       resume with autoResume
worker B                                         take lease (epoch 9), resume

Stale job sweep

Every job worker runs a sweep about once a minute. It looks at the jobs that are locked as running and ends the ones whose worker is gone:

• the lease owner is on the same host and its process no longer exists,
• the lease was not renewed for leaseStaleSec seconds,
• or the job was locked for leaseStaleSec seconds and no worker ever took its lease.

The sweep fences the old lease first, then fails the job with the reason (job interrupted: ...), or, with autoResume, queues it to resume while it has resumes left. Every worker sweeps, but only one acts on each job.

When a job worker is stopped (SIGTERM or SIGINT, as gig stop does), it interrupts its running jobs first: with autoResume they are handed over to resume, otherwise they end failed with job interrupted: worker stopped.

Stall detection

Apart from the lease, every run watches its own progress: records read and written, log lines and heartbeats. Long operations that move no records (counts, planning, snapshots, constraint rebuilds, dictionary saves) send heartbeats, which also show in the job log at most once a minute per phase. If nothing arrives for stallTimeoutSec seconds the job fails as stalled. A stalled job is not resumed automatically: use Restart to continue it from its checkpoints.

Shard helper workers

Each job worker process runs one job at a time. With shardWorkers greater than 0, a pump that reads big tables in parallel ranges publishes those ranges as units in MongoDB and queues up to shardWorkers helper jobs named pump-shard@<host>. Idle job workers of the same host pick them up and stream units side by side with the pump, which streams units too.

• The helpers run only on the host of the pump: they share the job dictionary file on its disk.
• They log into the log of the pump and are hidden from the job lists.
• The pump adds their progress to its own, and fails the units of a helper that dies.
• A big table is then split in up to Concurrency × (1 + shardWorkers) ranges, so every worker has work.

Helpers are used only when:

• the job is a pump into a datasource of a supported sink driver: PostgreSQL, MySQL, Oracle or MongoDB (SQL Server keeps session settings such as IDENTITY_INSERT in the process that prepared the load, DB2 keeps the constraints it dropped, and SQLite and files have a single writer);
• the load mode is not merge or update.

If no job worker of the host is free, the helpers wait in the queue and the pump streams the units itself.

COPY writes for Postgres

With copy: true a Postgres sink loads each batch with COPY FROM STDIN, which is much faster than INSERT for big loads. Merge and update loads COPY into a temporary staging table and run the upsert or update from it in the same transaction. A batch COPY cannot load is loaded by INSERT instead; after 3 such batches the table goes on with INSERT. Tables with composite-type columns always use INSERT.

Log retention

With logRetentionDays greater than 0, the job workers create (or change) a TTL index on the date of the job log lines when they start, and MongoDB removes the older lines in the background. Setting it back to 0 drops the index. Restart the server after changing it.

Examples

Default configuration

The section can be left out. This is equivalent:

jobs:
  stallTimeoutSec: 900
  autoResume: false
  concurrent: 1
  shardWorkers: 0

Large databases profile

A starting point for an instance that runs multi-hour pumps on databases of many terabytes, on a host with at least 5 job workers (gig start -j=6):

jobs:
  # long queries on the source before the first rows
  stallTimeoutSec: 1800
  # constraint rebuilds of very big tables
  finishTimeoutSec: 172800
  heartbeatSec: 15
  leaseStaleSec: 300
  # a restart of the server does not lose the work done
  autoResume: true
  maxAutoResumes: 5
  # at least 4 pipelines per job, big tables read in ranges
  concurrent: 4
  # 3 more workers of the host help each pump
  shardWorkers: 3
  copy: true
  # keep a month of job logs
  logRetentionDays: 30
  logTraceLevel: 2

Validation

The server checks the section when it starts and does not start with an invalid value, for example a negative number, or a leaseStaleSec below 30 or below 3 times heartbeatSec. The error names the property, such as:

leaseStaleSec must be 3 times heartbeatSec at least

Applying Configuration Changes

Restart the application after modifying the jobs configuration:

./gig stop
./gig start

Jobs running when the server stops are interrupted: with autoResume: true they resume when the job workers start again.

See Also

• Working with Large Databases
• Jobs
• Process Configuration
• Planning Disk Space