Data Lineage

Data lineage shows, for every column a job has written, where the data came from, which rule changed it and where it ended up. Use it to answer questions such as "which function was applied to customers.email in the dataset we sent to QA?" or "which columns of the production tap were overwritten with anonymized data?".

Gigantics captures lineage automatically. There is nothing to configure: run a job and its lineage appears.

Where to find it

There are two views of the same data:

The project page is titled Data Lineage with the subtitle Track the path data follows from source to destination. The model page shows Data lineage for model <name>.

What is captured

Lineage is recorded when a job completes. If any entity fails, the job fails and nothing is recorded, even when the job was launched with Continue Streaming On Fail (see New job): that option lets the other entities go on, but the job still ends as failed. The lineage of the destination stays as it was before the job. Once the job is resumed or run again and completes, its lineage is recorded. A problem while recording lineage is only written to the server log: it never makes the job fail.

So there are three destination types:

For each destination, Gigantics stores one record per column with:

• the source tap, the entity and the column
• the rule and the transformation applied to the column, with its options
• the column type from the schema
• the path the value followed (see Data path)
• the date of the last update

Which columns are recorded depends on the rule:

• If the rule has anonymize or synthesize operations, only the columns those operations configure are recorded. Disabled operations are ignored.
• If there is no rule, or the rule has no anonymize or synthesize fields, every column of every entity in the latest schema is recorded, with no transformation.

Columns of the partitions of an entity group get the transformations configured on the main entity of the group.

A rollback restores the original data, so it is recorded as a restore: every column of the entities it restored is recorded with no rule and no transformation, even if the backup was made by a job with a rule. Its path is Tap ──▶ Entity ──▶ Dataset ──▶ Original Tap, where the dataset is the backup. As those columns are no longer transformed, Transformed only hides them.

The lineage page

┌────────────────────────────────────────────────────────────────────────────────────┐
│ Data Lineage                                                                       │
│ Track the path data follows from source to destination                             │
├────────────────────────────────────────────────────────────────────────────────────┤
│ [Search entity or field   ] [⇕] [Destination type ▼] (●) Transformed only          │
│                                                     Destinations: 3   Fields: 42   │
├──────────────────────────────────────┬─────────────────────────────────────────────┤
│ ▾ ▣ qa-customers         (dataset)   │ Field             email                     │
│   ▾ customers                        │ Entity            customers                 │
│       Aa email          [fake]       │ Destination       qa-customers              │
│       Aa phone          [fake]       │ Destination Type  dataset                   │
│       #  card_number    [delete]     │ Source Tap        production                │
│   ▸ orders                           │ Rule              mask-pii                  │
│ ▸ ⛁ staging-pg            (sink)     │ Transformation    fake  (i)                 │
│ ▸ ⛁ production   (original tap)      │ Last Updated      2026-10-20 09:14          │
│                                      │                                             │
│                                      │ Data Path                                   │
│                                      │ [Tap] → [Entity] → [Rule] → [Dataset]       │
└──────────────────────────────────────┴─────────────────────────────────────────────┘

Toolbar

Tree

The left side is a tree of destinations, then entities, then fields. The icon of each destination tells its type. The columns of a destination are loaded when you expand it; until then the node shows how many columns it has.

Each field shows an icon for its type and, when it was transformed, an orange tag with the transformation.

Field details

Select a field to see its details on the right. Until you select one, the panel shows Browse lineage.

The transformations you will see are:

The info button next to the transformation opens Transformation Details, with the field, the action and the options of the function. Only the options that are set are listed:

Data path

Under the details, Data Path draws the nodes the value went through, in order:

 Tap ──▶ Entity ──▶ Rule ──▶ Dataset ──▶ Sink / Original Tap

The rule node appears only if the job had a rule, and the dataset node only if a dataset was involved (a dump, a load from a dataset or a rollback). When the dataset is the destination, it appears once, at the end. Each node has its own colour:

Some typical paths:

Dump                 Tap ──▶ Entity ──▶ Rule ──▶ Dataset
Pump to a sink       Tap ──▶ Entity ──▶ Rule ──▶ Sink
Load of a dataset    Tap ──▶ Entity ──▶ Rule ──▶ Dataset ──▶ Sink
Overwrite the tap    Tap ──▶ Entity ──▶ Rule ──▶ Original Tap
Rollback             Tap ──▶ Entity ──▶ Dataset ──▶ Original Tap

Empty page

Until a job records lineage, the page shows No lineage data and Run a job to start capturing data lineage. The view returns up to 5000 columns per query.

Lineage of jobs run before the upgrade

When Gigantics is upgraded to a version with data lineage, a migration runs on start. For every model, it uses the latest schema to fill in lineage for:

• the existing datasets (backup datasets are skipped)
• the sinks and taps written by the last completed job

Administrators can also run an optional script that recalculates the lineage of all datasets and fixes older records (transformation details, entity group partitions, saved function names, column types, paths, tap names and dates):

mongo <database> scripts/backfill-lineage.js

Run it from the Gigantics installation folder. It is not needed on a new installation.

Datasets recorded without a name, such as the ones of some older jobs, are shown with the current name of the dataset in the tree and in the report. Lineage recorded by older dumps may show the dataset twice in the data path; it is fixed the next time a job writes that dataset.

Lineage report

Lineage can be exported as a PDF report, which can be signed like a discovery report. See Audit.

What's next?

• Generate a data lineage report
• Group partitioned tables