Schema

In this document you will find information about:

• Your database Schema
• Working with Schema Versions
• Viewing the database DDL view
• Grouping partitions with Entity groups

Schema

To get the tap schema, create a new discover job and select only scan option. This job will scan your tap and return the schema.

Schema navigation

You will be able to navigate through the structure of your tap, looking up the data of your entities (fields, size, DDL...) and the fields (properties and relations).

You cannot modify the data or see real data, it is only a representation of the schema

Keys and relations

The scan reads the primary keys and the foreign keys of the database, and you can complete them in the schema:

• The PK column of the entity fields marks or unmarks a field as part of the primary key.
• The Relations tab of an entity and of a field adds a relation (foreign key) to another entity, or removes it.

Rules use them to keep the order and the referential integrity of the data they write. These controls are only shown to users whose role in the project has the Edit schema permission, the same permission the server requires to save them. Without it the keys and the relations are read only.

Versions

Scan identifies all entities and their fields as well as the size and type of database scanned. If during this process, changes are detected, Gigantics will generate a new version of the schema.

Inside Schema page, you can compare versions to identify changes in its structure.

DDL

Shows the DDL of the tap schema. You can export it to a file or just copy it to clipboard

Entity groups

Some databases split one logical table into many physical ones: a table per month (orders_202401, orders_202402...), per customer or per region, or the partitions of a partitioned table. Each one is a separate entity in the schema, but they all have the same columns and should be treated the same way.

An entity group links those entities to one main entity:

• The main entity is the one you configure: its labels and its rule configuration are applied to the rest of the group.
• The partitions are the entities whose name matches the group pattern, except the main entity.

 Group "orders partitions"      pattern ^orders_\d{6}$

            orders  (main entity)
               │   labels and rule configuration
     ┌─────────┼──────────────┐
     ▼         ▼              ▼
orders_202401  orders_202402  orders_202403   (partitions)

Entity groups belong to the model.

Managing entity groups

In the model Discovery page, the Entity Groups button of the entities toolbar opens the Entity Groups dialog. The button is only shown to users whose role in the project has the Edit schema permission, the same permission the server requires to add, edit and delete groups:

┌──────────────────────────────────────────────────────────────────────────┐
│ Entity Groups                                                        [X] │
├──────────────────────────────────────────────────────────────────────────┤
│ Name               Main Entity   Pattern          Matches  Partitioned   │
│ orders partitions  orders        ^orders_\d{6}$   [12]     [Yes]   ✎ ✕   │
│ logs by region     logs          ^logs_[a-z]{2}$  [5]              ✎ ✕   │
├──────────────────────────────────────────────────────────────────────────┤
│ [+ Add Entity Group]                                                     │
├──────────────────────────────────────────────────────────────────────────┤
│                                                       [Close and refresh]│
└──────────────────────────────────────────────────────────────────────────┘

Use the edit button to change a group and the delete button to remove it (Delete this entity group?). Without groups, the table shows No entity groups defined.

Add Entity Group opens this form:

While you type, Matching Entities (N) lists the partitions the pattern finds in the schema. Click Create (or Update) to save, or Cancel.

The pattern is a JavaScript regular expression that is searched anywhere in the entity name. Use ^ and $ to match the whole name: orders_ alone also matches old_orders_2019 or customer_orders.

The group is not saved when:

When you have made changes, the button at the bottom of the dialog is Close and refresh, and closing it reloads the page so the new groups are applied everywhere.

Groups in discovery

In the discovery entity list:

• Partitions have a purple P tag. The popover of an entity shows Partition of <main entity> on a partition, and Group: <name> on the main entity.
• Partitions inherit the labels of the main entity, also in the discovery report. Selecting a partition shows View Labels, which takes you to the main entity.
• The page of a partition shows This entity is a partition in group "...". Labels are inherited from the main entity "..." and cannot be edited here. Edit the labels on the main entity.

When you run a discover, or create a pipeline with a discover step, the entity selection also offers Use entity groups, with the Include / Exclude choice. The selected groups are expanded into their main entity and partitions in the same way as in the include/exclude operation of a rule, including the partitioned table behaviour.

Groups in rules

Configure the main entity once and the rule applies the same configuration to its partitions:

• In include/exclude, Use entity groups selects whole groups. See Entity groups in include/exclude.
• In anonymize, synthesize, select, limit and query, the configuration of the main entity is copied to its partitions when the job runs. A configuration set directly on a partition is replaced by the one of the main entity. In a query, the name of the main entity is replaced by the name of each partition: see Entity groups in Read Query.
• Partitions are hidden in the anonymize entity table, and the main entity shows a Group: <name> (N) tag. The same tag appears in the entity lists of the other operations.

Which groups are applied depends on the include/exclude operation of the rule:

Partitioned tables

Check Partitioned table when the main entity is the parent of a partitioned table, for example a PostgreSQL partitioned table, whose rows are actually stored in its partitions. Reading the parent and the partitions would process the same rows twice.

When the group is selected with Use entity groups in include/exclude in Include mode, a partitioned table group adds only its partitions to the job: the main entity is configured in the rule but is not read or written. A normal group adds the main entity and its partitions.

In Exclude mode, both kinds of group exclude the main entity and its partitions, so neither the parent nor its partitions are processed.

Include, normal group        orders + orders_202401 + orders_202402 ...
Include, partitioned table   orders_202401 + orders_202402 ...    (orders is skipped)
Exclude, any group           everything except orders and orders_2024xx

Groups in data lineage

In data lineage, the columns of a partition show the transformations configured on the main entity.

What's next?

• Start discovering your data