New job modal

The New job modal launches whenever you click the New job button on the Jobs page (or when other workflows defer to the job engine). It is a multi-panel form that adapts to your selections to produce either a one-off job or a reusable pipeline.

┌───────────────────────────────────────────────────────────────────────┐
│ New job                                                               │
│ ┌───────────────────────────────────────────────────────────────┐     │
│ │ From ▾ | To ▾ | Rule ▾ | Load options ▾ | Transform ▾ | …     │     │
│ └───────────────────────────────────────────────────────────────┘     │
│ Panels collapse once configured so you can scan the summary rows      │
└───────────────────────────────────────────────────────────────────────┘

When Manual execution or Repeat is selected in the Schedule panel, the footer reveals a required pipeline name field (Write pipeline name) and the primary button switches to Save as Pipeline. In all other cases the button reads Run Now or Run Later depending on the selected schedule.

Panel overview

The following sections break down every configurable field.

From panel

Internal behaviour:

• Choosing Tap keeps your job type aligned with the target (pump/dump).
• Selecting a dataset stores its ID for use in later steps (merge and overwrite operations require it).

To panel

Behind the scenes the system automatically adjusts job settings whenever you change your source or destination. This prevents incompatible combinations (for example, when overwriting your source database, certain write modes like Truncate or Drop are automatically disabled).

Rule panel

Selecting a rule unlocks the Transform options panel described below.

Transform options (conditional)

Load options panel

The Load options panel appears when your destination requires writing to a sink or database. For simple dataset copies, this panel is automatically hidden.

Stream content options

The Stream content setting determines what type of information is written to your database during job execution:

• Data and metadata (default): Loads both your actual data records and additional metadata about the processing, including schema information and job statistics.
• Data only: Loads only your actual data records without any additional metadata information.
• Metadata only: Loads only metadata information about the job processing without your actual data records.

Write batch size impact

The Write batch size affects how data is committed to your database:

• Smaller batch sizes: Provide more frequent commits, which can be helpful for monitoring progress but may result in slower overall processing due to increased transaction overhead.
• Larger batch sizes: Reduce transaction overhead and can improve overall throughput, but provide less frequent progress updates and may consume more memory during processing.

Concurrency behavior

The Concurrency setting of the Rule panel controls parallel processing:

• Each stream processes a different table, so several tables are processed at the same time.
• With Concurrency above 1, a large table read by primary key or row id can be split into key ranges that are read and written side by side, each range taking one of the streams. See Large databases for when a table is split.
• A deterministic job (with a seed) does not split tables into key ranges, because its random values follow the order of the records.
• The job runs with the larger of this value and the server setting jobs.concurrent.

Batch size and concurrency visualization

Below is a visualization showing how read and write batch sizes interact with concurrent workers:

Understanding backpressure

When the read batch size is significantly larger than the write batch size, backpressure can occur. This means that data is being read from the source faster than it can be written to the destination, causing a buildup of data in memory.

For example, if you set:

• Read batch size: 100,000 records
• Write batch size: 1,000 records
• Concurrency: 3

Each worker processes 100,000 records before moving to the next batch, but only writes 1,000 records at a time. This creates a buffer where processed data waits to be written, consuming memory. The system may slow down as it tries to manage this backlog, and in extreme cases, could run out of memory.

To optimize performance and memory usage:

• Keep read and write batch sizes relatively balanced
• If you need larger read batches, consider increasing write batch sizes proportionally
• Monitor your system's memory usage during job execution

Schedule panel

Schedule summaries when collapsed

When you've configured your schedule options, the panel will collapse and show a summary of your selection:

┌──────────────────┬───────────────────────────────────────────────┐
│ Run now          │ Summary shows: "Run: Now"                     │
├──────────────────┼───────────────────────────────────────────────┤
│ Schedule         │ "Run at:" + your selected date and time       │
├──────────────────┼───────────────────────────────────────────────┤
│ Manual execution │ Displays "Run pipeline manually"              │
├──────────────────┼───────────────────────────────────────────────┤
│ Repeat           │ Next run date/time + repeat interval          │
│                  │ (e.g. "Run at: 2024-07-24 18:00:00 / Every    │
│                  │ 1 day")                                       │
└──────────────────┴───────────────────────────────────────────────┘

Putting it together

1. Start with From and To. These two panels determine how your data will flow and automatically configure appropriate settings for the next steps.
2. Optionally select a Rule to apply masking or transformation logic to your data; doing so will enable additional configuration options.
3. Use Load options and Transform options to fine-tune performance settings like batch sizes, determinism, and dictionary handling.
4. Finish in the Schedule panel to decide when your job should run. If you're creating a reusable pipeline, provide a unique name in the footer before saving.
5. Review the collapsed panel summaries to ensure all required fields are filled. Each header will highlight missing required fields so misconfigurations are easy to spot.

Closing the modal clears all your selections so each new job starts with fresh default settings.