Rate
The Rate operation limits the reading speed of your data source, either for the entire dataset or specifically by entity. This can be useful for controlling resource usage or for simulating slower data processing scenarios.
Overview
The Rate operation enables you to:
- Control how quickly data is read from sources
- Set rate limits for all entities collectively or individually
- Specify processing speed in records per second
- Manage system resource consumption during data processing
Configuration
Scope
Choose how to apply the rate limit:
All entities: Applies the same rate limit to all entities in your dataset. This is useful when you want uniform processing speed across all data.
By entity: Allows you to set different rate limits for different entity types. This is useful when some entities require slower processing than others.
Rate Value
Specify the maximum number of records to process per second. For example:
- Setting a value of 100 means process at most 100 records per second
- Setting a value of 10 means process at most 10 records per second
Lower values result in slower processing but may reduce load on your system or data sources.
Examples
Uniform Rate Limiting
To process all entities at 50 records per second:
- Select "All entities" scope
- Enter "50" as the rate value
Per-Entity Rate Limiting
To process customers at 100 records per second and orders at 25 records per second:
- Select "By entity" scope
- Configure customer entity with rate value of 100
- Configure order entity with rate value of 25
This operation helps you control the pace of data processing to avoid overwhelming your system or data sources.