Datasets

General

When you run a rule, a dataset is generated with the result of the execution. This new dataset can be a subsets (for example using operations like limit or include/exclude) or full datasets.

The datasets can be downloaded to local folder (JSON or CSV) or can be loaded into one of the available sinks.

Load

To load a dataset into a sink, the sink must be added to the model. You can find more information about how to add a sink to the model in the Sinks page.

Once executed, a job will be created where you can see the progress of the load.

UI Options

The Datasets page provides several interaction options for each dataset in the table:

Share Functionality

The Share button allows you to create public URLs to share your datasets. When you click the Share button:

1. If you have only one API key in your project, it automatically creates a shareable endpoint
2. If you have multiple API keys, you'll be prompted to select which one to use

Share URL Structure

Once shared, datasets are accessible via URLs with the following structure:

[protocol]//[hostname]/api/[organization]/[project]/model/[model-seq]/dataset/[dataset-seq]

For example:

https://yourserver/api/acme-corp/sales-data/model/1/dataset/42

URL Parameters

You can customize the shared URL with additional parameters:

Example URLs with parameters:

• https://yourserver/api/acme-corp/sales-data/model/1/dataset/42?format=json-zip
• https://yourserver/api/acme-corp/sales-data/model/1/dataset/42?format=csv-zip
• https://yourserver/api/acme-corp/sales-data/model/1/dataset/42?format=sql

Sharing UI Details

When you click the Share button, the row expands to show:

1. API Endpoint: The full URL to access your dataset
2. Format Selector: Dropdown to choose the format (JSON ZIP, CSV ZIP, SQL)
3. Copy Button: Copy the URL with selected format to clipboard
4. Call Counter: Shows how many times the endpoint has been accessed
5. Delete Option: Remove the API endpoint if no longer needed

ASCII Representation of Dataset Table

┌──────────────────────────────────────────────────────────────────────────────────────────────┐
│ Dataset Name         Rule     Last Updated   Created By   Size       Actions                 │
├──────────────────────────────────────────────────────────────────────────────────────────────┤
│ customers_2023       filter   2 hours ago    John Doe     2.4 MB     Download Share Delete   │
│ products_filtered    select   1 day ago      Jane Smith   15.7 MB    Download Share Delete   │
└──────────────────────────────────────────────────────────────────────────────────────────────┘

When you click the Share button for a dataset, the row expands:

┌──────────────────────────────────────────────────────────────────────────────────────────────┐
│ Dataset Name         Rule     Last Updated   Created By   Size       Actions                 │
├──────────────────────────────────────────────────────────────────────────────────────────────┤
│ customers_2023       filter   2 hours ago    John Doe     2.4 MB     Download Share Delete   │
├──────────────────────────────────────────────────────────────────────────────────────────────┤
│ ┌── GET ────┐ https://yourserver/api/org/proj/model/1/dataset/42           [Copy]            │
│ │  # calls  │                                                              [Delete]          │
│ └───────────┘  ?format= [json-zip ▼]                                                         │
└──────────────────────────────────────────────────────────────────────────────────────────────┘

This expanded view shows the shareable URL, allows you to select different formats, and provides options to copy the URL or delete the endpoint.

Backup Datasets

A job that writes anonymized data back into the tap it reads from (destination Overwrite current tap) changes your source data in place. A backup dataset keeps the original rows the job overwrites, so you can restore them.

Creating a backup

In the To panel of the new job dialog, choose Overwrite current tap and check Generate a backup (This will create a backup dataset of the current tap.). The checkbox is only enabled with that destination.

The backup needs the Update or Merge load mode, which write back only the rows the job reads. With another load mode the job fails before touching the tap: The backup of the tap needs the update or merge load mode.

The job:

1. Creates the backup dataset before writing anything, named backup-<tap>-<model>-<n> in lowercase, where <n> is a sequence number for that tap and model. The dataset is linked to the job and to its rule.
2. Copies each row into it as it reads it from the tap, before the rule changes it.
3. Saves every batch in the backup before writing that batch to the tap.

If the backup dataset cannot be created, the job stops without touching the tap: Backup creation failed: ... Aborting job to prevent data loss. If a batch cannot be saved in the backup, the job stops before that batch is written to the tap.

The job log shows these steps as backup messages: Creating backup dataset, Backup dataset created: <name>, and one backed up <entity> (N rows read) per table. The job details show the backup under Backup (or No backup generated).

If the job is restarted, it reuses the backup it already made (Reusing existing backup dataset) instead of creating a new one and keeps what it already saved. A row whose primary key is already in the backup is not copied again, so the backup keeps its original values; rows without a primary key are copied again. Rerun starts a new job, which makes a new backup. See Jobs.

Backups page

Backup datasets are not listed on the Datasets page. They have their own Backups page in the model menu, after Datasets, with the subtitle Backup datasets that can be used to restore the original data in your data source. The menu item only appears when the model has at least one backup and you have the View sensitive data role, because backups contain real data.

┌───────────────────────────────────────────────────────────────────────────────────────────────┐
│ Backups                                                                                       │
│ Backup datasets that can be used to restore the original data in your data source.            │
├───────────────────────────────────────────────────────────────────────────────────────────────┤
│    Name                                  Rule       Last updated  Created by  Size  Schema    │
├───────────────────────────────────────────────────────────────────────────────────────────────┤
│ ●  [Rollback] backup-crm-customers-3     mask-pii   1 hour ago    John Doe    2 GB  v4 Latest │
│ ●  [Rollback] backup-crm-customers-2     mask-pii   3 days ago    John Doe    2 GB  v3        │
└───────────────────────────────────────────────────────────────────────────────────────────────┘

In the table, Rename, Delete and Copy need the Delete datasets role. Backups cannot be downloaded or shared from the interface: the page of a backup has no download buttons, and without the View sensitive data role it shows an access denied page. Its toolbar has Rename, Delete and Rollback (instead of Load). Rollback is enabled there under the same conditions as in the table.

Rolling back

Click Rollback on the Backups page or on the backup page. A confirmation shows what the rollback will do:

┌───────────────────────────────────────────────────────────────────┐
│ Are you sure you want to rollback this dataset?                   │
├───────────────────────────────────────────────────────────────────┤
│ This will restore the tap data from this backup dataset.          │
│ The current tap data will be updated.                             │
│                                                                   │
│ Rollback Options                                                  │
│   Destination   Overwrite current tap                             │
│   Load mode     Update                                            │
│   Dictionary    None                                              │
├───────────────────────────────────────────────────────────────────┤
│                                   [Cancel]  [Yes, rollback]       │
└───────────────────────────────────────────────────────────────────┘

Yes, rollback starts a load job from the backup into the model tap right away. It reuses the options of the job that made the backup, with two changes: the load mode is Update, so the rows in the tap are updated with the values in the backup, and no dictionary is used. The job appears with the type rollback in the job lists. Starting a rollback needs the Run load job and Overwrite current tap roles, and a tap that allows being overwritten, the same as a job with the Overwrite current tap destination. The server checks them too: without the roles the rollback is denied, and if the tap does not allow being overwritten it fails with This tap is not allowed to be overwritten.

The rollback is recorded in data lineage as a restore of the tap from the backup, with no rule and no transformation.

  Job with backup                                  Rollback
  ───────────────                                  ────────
  tap ──copy──▶ backup-crm-customers-3             backup-crm-customers-3
  tap ◀──anonymized rows── rule                          │ load, mode Update
                                                         ▼
                                                        tap (original values)

The rollback fails with Rollback dataset not found, Original job not found for this rollback dataset or Model does not have an associated tap when the backup, the job that made it, or the model tap no longer exist.