Where

The Where operation allows you to filter your dataset by creating queries that include only the records matching specific conditions. This operation is applied before any transformation operations in the pipeline.

Overview

The Where operation enables you to:

• Filter records based on field values
• Create complex conditions using AND/OR logic
• Apply different filters to different entity types
• Work with various data types (strings, numbers, dates)

Configuration

Entity-Level Filtering

The Where operation is configured at the entity level, meaning you can apply different filters to each entity in your dataset. For each entity, you can:

1. Add multiple filtering rules
2. Group rules together with AND/OR operators
3. Nest groups to create complex filtering logic

Rule Configuration

Each rule consists of three components:

• Field: The field you want to filter on
• Operator: The comparison operator to use
• Value: The value to compare against

Supported operators vary by field type:

• String fields: =, !=, contains, starts with, ends with
• Number fields: =, !=, >, >=, <, <=
• Date fields: =, !=, >, >=, <, <=

Grouping Logic

You can organize your rules into groups with two types of logic:

• AND groups: All rules in the group must be true
• OR groups: At least one rule in the group must be true

Groups can be nested to create complex filtering conditions. For example, you might have an OR group containing two AND groups, allowing you to filter for records that match either of two different sets of conditions.

Examples

Simple Filter

To filter customer records where the status is "active":

1. Select the customer entity
2. Add a rule with:
   • Field: status
   • Operator: =
   • Value: active

Complex Filter with Groups

To filter order records where either:

• Order amount is greater than 1000 OR
• Order date is within the last 30 days AND order status is "pending"

1. Select the order entity
2. Create an OR group
3. Add a rule to the OR group:
   • Field: amount
   • Operator: >
   • Value: 1000
4. Add an AND group as another child of the OR group
5. Add two rules to the AND group:
   • Field: date, Operator: >, Value: (30 days ago)
   • Field: status, Operator: =, Value: pending

This creates a flexible filtering system that can handle complex business logic requirements.