Include/Exclude

The Include/Exclude operation allows you to select which entities to include in or exclude from your new dataset. This operation is applied at the beginning of the pipeline to determine the initial set of entities to process.

Overview

The Include/Exclude operation provides two ways to filter entities:

• Include mode: Only the selected entities are processed
• Exclude mode: All entities except the selected ones are processed

Additionally, you can use regular expressions to match entity names when you have a naming pattern, or select whole entity groups.

Configuration

Selection Mode

Choose between two modes for entity filtering:

Include: Only the entities you select will be included in your dataset. This is useful when you only need a few specific entities from a large dataset.

Exclude: All entities except those you select will be included in your dataset. This is useful when you want to omit just a few entities from a dataset where most entities are relevant.

The Include / Exclude choice appears with every method (Use selector, Use regular expression and Use entity groups), and it is saved with the operation: reopening an operation saved in exclude mode shows Exclude selected.

Entity Selection

Up to three methods are available for selecting entities: Use selector, Use regular expression and, when the model has entity groups, Use entity groups. Only the method shown when you save the operation is kept.

Selector: Choose entities from a list of available entities in your dataset. You can select multiple entities by checking them in the list. With Exclude the list on the right is titled Excluded entities, and the rule processes every other entity.

Regex: Define a regular expression pattern to match entity names. This is particularly useful when your entities follow naming conventions. For example, if you have entities named "customer_2023", "customer_2024", "order_2023", "order_2024", you could use the regex "customer_.*" to include all customer entities.

Entity Groups

Use entity groups only appears when the model has at least one entity group. It shows the Include / Exclude choice and one checkbox per group:

┌──────────────────────────────────────────────────────────────────┐
│ [Use selector] [Use regular expression] [Use entity groups]      │
├──────────────────────────────────────────────────────────────────┤
│ (o) Include  ( ) Exclude                                         │
│ [x] orders partitions   Main: orders   [12 entities]             │
│ [ ] logs by region      Main: logs     [6 entities]              │
└──────────────────────────────────────────────────────────────────┘

Each row shows the group name in bold, its main entity (Main: <entity>) and a tag with the number of entities the group includes in or excludes from the job:

With Exclude, every entity that is not in a selected group is processed. The partitions are the entities of the latest schema that match the group pattern, so partitions created after the group (for example the table of a new month) are picked up once the schema is scanned again.

If the selected groups no longer exist (they were deleted or renamed), an operation in include mode selects no entity at all instead of the whole schema, and the server log shows a warning. Edit the operation and select the groups again.

The method also decides which groups share the configuration of their main entity in the other operations of the rule (anonymize, synthesize, select, limit and query):

Examples

Include Specific Entities

To include only customer and order entities in your dataset:

1. Select "Include" mode
2. Choose "Selector" method
3. Check "customer" and "order" entities from the list

Exclude Specific Entities

To exclude the log and audit entities from your dataset:

1. Choose "Use regular expression"
2. Select "Exclude" mode
3. Enter regex pattern: "^(log|audit)$"

Include by Pattern

To include all entities that start with "sales_":

1. Choose "Use regular expression"
2. Select "Include" mode
3. Enter regex pattern: "^sales_.*"

Include an Entity Group

To process the monthly partitions of orders with the configuration of orders:

1. Create the group in the model Discovery page (Entity Groups button), with orders as main entity and the pattern ^orders_\d{6}$
2. In the rule, choose "Use entity groups"
3. Select "Include" mode
4. Check the group

Exclude an Entity Group

To process everything except a partitioned table and its partitions, for example audit_log partitioned by month:

1. Create the group with audit_log as main entity, the pattern ^audit_log_\d{6}$ and Partitioned table checked
2. In the rule, choose "Use entity groups"
3. Select "Exclude" mode
4. Check the group: neither audit_log nor its partitions are processed

This operation allows you to efficiently control the scope of your data processing pipeline by including only relevant entities or excluding irrelevant ones.