Pipelines
In Gigantics we call Pipeline to a template or blueprint that allows the user to execute a job periodically and/or using a public link.
Pipelines API
Pipelines can be executed by using Run button or by using a URL. To enable
this type of remote execution it is necessary to Manage the APIs visible in the
actions menu of the pipeline.
From the management window, the user can create or revoke the keys. To use them, just copy the API key and invoke it from the url:
https://<server>:<port>/api/pipeline?api_key=<api_key>The user who runs the rule must have permissions to edit models in the project.
Examples
1- Use a pipeline to periodically scan the database for changes in its schema.
2- Provision environments with masked data: Using a transformation rule and dumping the results into a sink periodically.
3- Load a dataset in your test database to run unit tests in your CI/CD pipeline.
Create a New Pipeline
For detailed step-by-step instructions on creating new pipelines, including all configuration options, settings, and workflow details, see the New Pipeline documentation.
The New Pipeline guide covers:
- Three-step creation wizard (General Settings, Configuration, Scheduling)
- All job types and their specific options
- Concurrency, load, transform, and logging configurations
- Manual vs scheduled execution options
- Example pipeline configurations