Rules

What is a Rule

A rule is a set of operations that outputs anonymized or synthesized data into new datasets, the same or different datasources (sinks).

A rule is comparable to a program or an algorithm.

Rules run in jobs.

How to create a new rule

Open the Rules area of the tap model and click Create. The New rule window has two parts:

• General: the Name of the rule (required, "Rule name is required") and a Description of its purpose.
• Default Options: the options the rule uses when it runs, described below.

Import Rule creates a rule from a file exported with Export rule.

Options

The Default Options of a rule are:

Load Options:

Update and Merge need a primary key in the entity; without one the entity fails with an error like "Cannot update/merge database without any defined primary keys".

Dictionary modes:

With a dictionary you can also check:

• Cache dictionary (on by default): keeps a local cache of the dictionary for the project.
• Store new transformations in the dictionary (on by default): saves the new values so later runs reuse them.
• Overwrite existing dictionary (off by default): replaces the values already stored.

These checkboxes are disabled with No dictionary. For more information check the article on how to use dictionaries.

Ignore already anonymized values

Ignore already anonymized values protects data that was already anonymized when a rule runs again over the same tap. It is not a rule option: you choose it in each job that overwrites the tap, and only when a dictionary is used (the checkbox is disabled with No dictionary). It appears in:

It is off by default and it is never taken from the rule. In any job that doesn't overwrite the tap it is always off, whatever value was saved.

When it is on, every value is first looked up in the dictionary:

 value read from the tap
   │
   ├── stored as an original value ───► replaced with its stored
   │                                     anonymized value
   │
   ├── stored as an anonymized value ──► kept as it is
   │                                     (already anonymized)
   │
   └── not in the dictionary ──────────► anonymized as usual

If all the non-empty fields of a row that use the dictionary are already anonymized, the row is skipped and not written, so it stays as it is in the tap. The job log shows a line per entity like customers: 120 values ignored, 40 rows skipped (already anonymized).

Actions

The rules list shows a Run button for each rule and an Actions menu with:

• Export rule: downloads the rule as a .json file, which you can bring into another model with Import Rule.
• Edit rule: edits the rule options described in the above section Options.
• View operations: goes to the operations page of the rule.
• View jobs: shows the jobs of the rule.
• Delete: deletes the rule from the model, after asking "Are you sure you want to delete this rule?".

Run opens the New job panel with the rule already selected, From set to the tap and To set to New Dataset. You can change the destination to a sink, an existing dataset or Overwrite current tap.

The panel starts with the options of the rule, and you can change them for this run only:

• Concurrency, Continue Streaming On Fail, Retry On Fail, Read batch size and Write batch size.
• Target entities (the load mode, when the output goes to a sink) starts at the load mode of the rule. If the sink doesn't support it, the first mode it supports is used instead. Overwrite current tap always uses Update, and the job asks for confirmation ("Warning: DATA LOSS WILL OCCUR"). It is disabled when the tap doesn't allow overwrites.
• Transform options: Determinism, Seed, Check foreign keys and Dictionary, plus Ignore already anonymized values when overwriting the tap.

Operations list

After creating the rule, click on the name to enter into operations page. Add operation opens a grid with the operations you can add, divided into three areas. Each operation can be added once per rule; the operations already in the rule are shown as selected.

Query

Query Operations change how the data is read from the tap, as if it were a SQL query:

Transform

Transform Operations change the data or the order in which it is processed:

These transformations output anonymized or synthesized data into new datasets, the same or different datasources (sinks).

Output

Output Operations change the data written to the destination: