Limit

The Limit operation allows you to restrict the number of records in your dataset output. You can limit by absolute number of rows or by percentage, and apply the same limit to every entity or a different one to each entity.

Overview

The Limit operation provides flexible ways to reduce dataset size:

• Limit by a fixed number of rows
• Limit by a percentage of available records
• Apply the same limit to every entity, or set one per entity
• Choose which records to keep (first, last, or random)

Configuration Options

Scope

The scope determines whether one limit is used for every entity or each entity has its own:

All entities: Applies the same limit to each entity separately; it is not a total for the whole dataset. For example, if you have 1000 customer records and 1000 order records, a limit of 500 returns up to 500 customers and up to 500 orders (1000 records at most). An entity configured with its own limit uses that one instead.

By entity: Applies the limit separately to each entity type. For example, if you have customer and order entities, a limit of 500 would return up to 500 customer records AND up to 500 order records (1000 total records maximum).

Limit Type

You can specify how the limit should be applied:

By number of rows: Specify an exact number of records to include. For example, limit to exactly 1000 records.

By percentage: Specify a percentage of the total available records. For example, limit to 20% of all records. When using percentages, you can also set minimum and maximum row constraints to ensure you get a reasonable number of records even when the percentage of a small dataset might be too few or too many records.

Row Position

Determines which records are selected when applying the limit:

First records: Selects records from the beginning of the dataset (useful for getting the most recent records when data is sorted chronologically).

Last records: Selects records from the end of the dataset (useful for getting the oldest records when data is sorted chronologically).

Random records: Selects records randomly from the dataset (useful for sampling data).

Percentage Constraints

When using percentage-based limits, you can set additional constraints:

Min rows: Ensures that even if the percentage of total records is small, you'll get at least this many rows. For example, if you set 5% but want at least 1000 rows, this setting ensures you'll get 1000 rows even if 5% of your dataset is less than that.

Max rows: Ensures that even if the percentage of total records is large, you won't get more than this many rows. For example, if you set 50% but only want a maximum of 50000 rows, this setting caps your output at 50000 rows even if 50% of your dataset would be more.

Referential Integrity

Check Ensure Referential Integrity to keep the subset consistent: a child row (for example an order) is extracted only if the parent row it references (the customer) is extracted too. Child rows whose foreign key is NULL are kept. The checkbox is only shown on SQL taps: Postgres, MySQL, Google Cloud SQL, Oracle, MSSQL, DB2, SQLite and Databricks. When it is enabled:

• The limit is applied to each parent table, including the All entities limit.
• A percentage is turned into a number of rows using the row count estimated for each table, and then the min and max rows are applied.
• The Where filter of a parent table is also applied when its children are filtered, so children only reference parent rows that pass the filter and the limit.
• First and Last order the rows by primary key. Random uses a repeatable order based on a hash of the primary key, so every run picks the same sample (on databases without a hash function it falls back to primary key order).
• On Oracle the tables are qualified with the Owner of the tap, in upper case (for example "SALES"."CUSTOMERS").

Examples

Limiting by Number of Rows

To get up to 1000 records from each entity:

1. Set Scope to "All entities"
2. Set Limit Type to "By number of rows"
3. Enter "1000" in the value field
4. Choose which records to keep (First, Last, or Random)

Limiting by Percentage

To get 20% of the records of each entity:

1. Set Scope to "All entities"
2. Set Limit Type to "By percentage"
3. Enter "20" in the value field
4. Choose which records to keep (First, Last, or Random)

Per-Entity Limiting

To get up to 500 records from each entity:

1. Set Scope to "By entity"
2. Configure each entity with:
   • Limit Type: "By number of rows"
   • Value: 500
   • Position: "First" (or your preferred selection)

This approach is particularly useful when working with related entities where you want to maintain balanced representation across all types.