Where

The Where operation allows you to filter your dataset by creating queries that include only the records matching specific conditions. This operation is applied before any transformation operations in the pipeline.

In the operations grid it appears in Query Operations as Where ("Filter records by value").

Overview

The Where operation enables you to:

• Filter records based on field values
• Keep or discard records whose field is empty (NULL)
• Create complex conditions using And/Or logic
• Apply different filters to different entity types
• Work with various data types (strings, numbers, dates, booleans)

Configuration

Entity-Level Filtering

The Where operation is configured at the entity level, meaning you can apply different filters to each entity in your dataset. For each entity, you can:

1. Add multiple filtering rules
2. Group rules together with And/Or operators
3. Nest groups to create complex filtering logic

Entities without a Where configuration are read completely.

Pick the entities in the Search entities box (at least one is required: "You must select at least one entity"). Each selected entity is listed with a pen icon (Edit entity configuration) that opens its filter, and a bin icon (Remove entity configuration). An entity without a filter shows a warning ("Entity where query is not set, please click to edit this entity"), and the operation can't be saved until it has one ("There are unset entities. Please check your configuration").

Rule Configuration

Each rule is one row with three components:

• Field: The field you want to filter on. Only fields of type string, number (integer or decimal, such as float, double or decimal columns), date and boolean are listed. If the entity has none, the panel shows "No fields found for this entity".
• Operator: The comparison operator to use. The list depends on the field type and on the tap driver (see the table below).
• Value: The value to compare against. The input depends on the field type: a text box for strings, a number box for numbers (decimal numbers included) and a date and time picker for dates. Boolean fields and the is null / is not null operators have no value box.

When you change the field of a rule, the operator goes back to the first operator of the new field type.

Operators

These are the operators the UI offers, by field type:

What each operator does:

How values are sent

• Strings are sent as typed. With like you can also use the database wildcards inside the value (%, _).
• Numbers are sent as numbers.
• Dates are picked with date and time and are sent as an ISO 8601 date-time in UTC, for example 2026-10-21T13:45:00.000Z.
• Booleans are sent as true / false.
• In SQL databases the values are sent as query parameters, not pasted into the SQL text, so quotes in a value don't break the query. The exception is Ensure Referential Integrity, described below.
• A string value can't start with __: or ~~ (these prefixes mean raw SQL in the query builder). Such a rule makes the rule fail with the error Where value for field '<field>' cannot start with '<prefix>'.

Supported drivers

The same Where configuration works on every tap, but each driver applies it in a different place:

When the filter runs in Gigantics (CSV, File Storage and Redis), every record is read and then checked:

• is and is not compare the values as text.
• less than and more than compare as numbers when both sides are numbers, otherwise as text. Dates are compared as text with the ISO 8601 value, so they only compare correctly when the stored dates use the same format.
• like ignores upper and lower case, and both % and * work as wildcards.
• starts with and ends with are case sensitive.
• is null matches a missing field or a null value. An empty string is not null.

Grouping Logic

Every entity starts with one group (Group 1). In a group you can add rules with Rule + and nested groups with Group +. An empty group shows "Empty group, Click here to add a new rule" (Click here adds a rule).

Each group has an And / Or selector:

• And groups: All rules in the group must be true
• Or groups: At least one rule in the group must be true

Nested groups are numbered after their parent (Group 1.1, Group 1.2, ...) and can be removed with the - button next to the selector. Empty groups are ignored when the rule runs.

Ensure Referential Integrity

Check Ensure Referential Integrity to keep the subset consistent: a child row is extracted only if the parent row it references is extracted too. When it is enabled, the Where filter of a parent table is also applied when its children are filtered, so children only reference parent rows that pass the filter (and the Limit, if there is one).

In this mode the conditions of the child tables, and of the parent tables they are checked against, are written as text into the generated SQL, so:

• Text is quoted with single quotes (a quote in the value is doubled). On MySQL, Google Cloud SQL and Databricks backslashes are escaped too.
• Booleans are written as 1 / 0 on Oracle, MSSQL and DB2, and as TRUE / FALSE on the rest.
• Column names are quoted with "..." on Oracle and with backticks on Databricks.
• regex is not supported.

Examples

Simple Filter

To filter customer records where the status is "active":

1. Select the customer entity
2. Add a rule with:
   • Field: status
   • Operator: is
   • Value: active

Records Without a Value

To keep only the customers that have an email address:

1. Select the customer entity
2. Add a rule with:
   • Field: email
   • Operator: is not null

No value is needed.

Complex Filter with Groups

To filter order records where either:

• Order amount is greater than 1000 OR
• Order date is after a given date AND order status is "pending"

1. Select the order entity
2. Set Group 1 to Or
3. Add a rule to the group:
   • Field: amount
   • Operator: more than
   • Value: 1000
4. Add a nested group (Group 1.1) and set it to And
5. Add two rules to Group 1.1:
   • Field: date, Operator: after, Value: 2026-09-21 00:00:00
   • Field: status, Operator: is, Value: pending

Group 1                         [Or]
├── amount   more than   1000
└── Group 1.1                   [And]
    ├── date     after   2026-09-21 00:00:00
    └── status   is      pending

SQL equivalent:
WHERE amount > 1000
   OR (date > '2026-09-21T00:00:00.000Z' AND status = 'pending')

This creates a flexible filtering system that can handle complex business logic requirements.