Gigantics 2.0.0 Release Notes
Release date: October 30, 2026
We're pleased to present our new major release of Gigantics 2.0.0 with some important features and bugfixes:
Improvements and issues resolved
[ENH] #3365 iSeries driver OS-based
The DB2 for iSeries (DB2i) driver has been rebuilt as a single driver with a selectable "Backend Mode" when creating or editing a tap or sink: JS400 (new default, pure JavaScript, no IBM client required), SSH, ODBC or IBM-DB. SSH mode connects directly to the IBM i operating system, reading data in batches and writing through SFTP, with new SSH Port, SSH Private Key, SSH Passphrase and Remote Temp Path fields. The DB2 z/OS driver also gains a backend selector (IBM-DB or the new JS390 mode), and BIGINT values plus batch insert errors are now handled and reported correctly on both platforms.
[ENH] #3510 Docs are now in external web repository
The product documentation is no longer kept inside the application repository. The old in-repo docs folder was removed, and the documentation is now published as its own site at https://docs.gigantics.io. The Documentation entry in the user menu now opens that site instead of the old gigantics.io/docs page.
[ENH] #3622 Local in component api queries
Internal change to how the interface loads data. Screens can now run their API queries locally inside each component, through new shared query hooks, instead of always going through the global store. This keeps data fetching simpler and avoids keeping data in the store when it isn't needed. No visible change in behavior.
[ENH] #3623 UI components enhancements
Improved the shared interface components used across the application. Modals can now be stacked, so opening a dialog from another dialog no longer closes or replaces the first one, and buttons that start an asynchronous action show a loading state until it finishes. It also fixes the "create organization" dialog appearing duplicated on first login.
[ENH] #3673 Test button in Read Query panel
The Read Query editor in rules now has a "Test Query" button that runs the query against the actual tap, fetching up to 5 records, and shows the record count on success or the database error on failure. Variables of the form $(varName) are resolved before the test runs. Saving the query now runs the test first and only saves if it succeeds.
[ENH] #3674 Remove unimplemented operations
The Add operation panel of a rule now only shows operations that are actually available. Operations that were listed but not yet implemented (Generate, Declare and Sort) are hidden, and sections that would end up empty are no longer displayed.
[ENH] #3746 Gigantics mono repo
The Gigantics source code is now a single repository. The data engine (debaser), which used to be a separate submodule, and the discovery and fake data model project (cactus) are now part of the main repository. The build pipeline now trains and bundles the discovery brain and copies it into the engine before building the application, so every release is built from one consistent set of sources. No change for users.
[ENH] #3780 Better error logging from UI
When a server request fails, the interface now shows an error banner with the operation and status, and a Details view with endpoint, time, server, version, message and a trimmed stack trace. The information can be copied to the clipboard or downloaded as a JSON report to attach to support requests. Unexpected interface crashes are caught and reported instead of leaving a blank screen, and the server log now records each error with its identifier and full stack.
[ENH] #3784 Allow deleting master license
Root users can now remove a master license from Global Settings > Licensing with a new "Remove" button, after a confirmation dialog. Any taps assigned to that license are released, and tap limits apply again unless another license is added.
[ENH] #3792 Data lineage
New Data Lineage view, available both at project level and inside each model, that shows the path every destination column has followed: source tap, entity, rule, dataset and destination sink or tap. For each field you can see its type and the transformations applied (anonymize/synthesize functions, including saved functions, locale and custom code), with search and a "transformed only" filter; entity groups are expanded into their partitions. Lineage is recorded automatically by dump, load and pump jobs, existing jobs are migrated on upgrade, and a new Data Lineage report type can be generated as a PDF from the Audit section.
[ENH] #3810 Table name mask groups in discovery and rules
New Entity Groups let you define a group with a main entity and a regular-expression name pattern (e.g. ^orders_\d{6}$) that matches partitioned or sharded tables, with a live preview of matching entities; groups can be created before matching tables exist. Partition tables inherit labels, PII and sensitivity from the main entity in discovery (label edits are redirected to the main entity), and the heatmap, tree and audit report PDF show the group. Rule operations (Include/Exclude, Anonymize, Synthesize, Select, Limit and Query) can be configured once on the group and are automatically applied to every matching partition at execution time, including replacing the main table name in custom queries.
[ENH] #3813 Improve performance in large number of entities models
Improved how the PostgreSQL driver handles models with many tables spread across schemas. All queries (counts, reads, drops and constraint handling) now use quoted, fully qualified schema.table names, and the target schema is created automatically when it does not exist. This avoids failures with tables in non-default schemas or with special characters in their names, and fixes constraints being disabled on the wrong table.
[ENH] #3814 AI agents tools and docs
Added guidelines and tooling for AI coding agents that work on the Gigantics codebase. A rewritten agent guide sets out which parts of the code to read first and which generated files must not be edited by hand. It also lists the patterns to follow for forms and data fetching, and the build, lint and type-check steps that must pass before a change is complete. A new command regenerates the GraphQL schema on its own. Development-only change, no effect on the product.
[ENH] #3818 Resume in progress tables from the record where the job failed
When a failed or interrupted pump, dump or load job is restarted, tables that were partially processed now resume from the last committed position instead of being reprocessed from the beginning. The resume point is rounded down to the write batch size that was in use when the job ran (even if the batch size is changed before restarting), so at most one batch is reprocessed and no uncommitted rows are skipped.
[ENH] #3827 Global driver retry strategy
New "Retry On Fail" option in rule settings (also available when launching jobs, pipelines and dataset loads): when a write batch fails, it is split in half and retried repeatedly until the offending record is isolated, so all valid records are still written. Supported by the PostgreSQL, MySQL, SQL Server, Oracle, Databricks and Salesforce drivers. Error messages now report how many records were committed before the failure and log the failing record.
[ENH] #3828 Use dictionary to detect previous anonymization
A new dictionary option, "Ignore already anonymized values" (enabled by default when a dictionary mode is set), lets a rule recognise values that were already anonymized in a previous run and leave them untouched; rows whose anonymized fields are all already anonymized are skipped, and the job log reports how many values and rows were ignored. Dictionary entries now record the job that created them, with a new Job column, search, and a "Clear by job" action to remove entries from a bad run. A new "Manage Scopes" panel lets you view, filter and merge dictionary scopes. Job transfer rates are also computed more accurately.
[ENH] #3829 New DataBricks driver
Databricks is now available as a driver for taps and sinks. You connect with Host, HTTP Path, Catalog, Schema and a personal access token, and Gigantics can extract the schema and DDL, read data, and write it with insert, update and merge modes, as well as create, truncate and drop tables. Because Databricks tables often have no declared primary keys, you can now click the Primary Key column in the schema entity fields table to mark or unmark a field as a primary key by hand.
[ENH] #3831 Import taps and sinks
Taps and sinks can now be imported and exported as YAML files. The Import from file button creates one or several connections from a single- or multi-entry YAML file (including environment references), each row has a new Export action, and the toolbar has an Export all button to download every tap or sink. Secrets (passwords, tokens, keys and secret-looking Custom Params) are never written to the exported file: they are replaced with a mask, and after importing you type them again in each tap or sink.
[ENH] #3833 Files driver
New File Storage driver (under "Files & storage") that treats files in a folder as entities, so they can be scanned, discovered, anonymized and written to a sink like any database table. Supported formats: CSV, JSON/JSONL, XML, Excel (each sheet becomes an entity and multi-sheet workbooks are rebuilt in the sink), YAML, Parquet and EDI; files can be read from Local Disk, SSH/SFTP, Google Drive or SharePoint, with an optional file pattern. File sinks offer file naming options: add a timestamp (default), keep the original name, or a custom pattern using {entity}, {date}, {timestamp}, {ext} and {format}.
[ENH] #3838 Redis driver
New Redis driver, available under Big data in the driver selector and included in the open source license tier. You connect with Host, Port (6379 by default), an optional Database index and optional User/Password, or a password manager; SSH tunnels are supported. Keys are grouped into entities by their prefix (one entity per Redis type when a prefix mixes types), and all Redis types are supported: string, hash, list, set, sorted set and stream. You can preview data, read with Where filters, custom queries and limits, and write back keeping the native Redis types.
[ENH] #3844 Backup datasets should be generated in the same process
When backup is enabled for a pump or load job, the backup dataset is now written in the same pass as the data sent to the sink, instead of in a separate dump before the job. Each record is written to the sink and to the backup at the same time, so the source is read only once and the job finishes sooner. Backup activity appears under a new "backup" level in the job log, and rollback jobs are now shown as their own job type and can be filtered in the jobs lists.
[ENH] #3845 Add GiganticsWeb to the mono repo
The Gigantics website and the documentation site, previously kept in a separate GiganticsWeb repository, are now part of the main repository as two web applications. The public website and the documentation are now versioned and released together with the product, so the documentation for each release stays in step with the code.
[ENH] #3847 Manage partitioned tables: part I
Entity groups can now be marked as a partitioned table with a new "Partitioned table" checkbox in the entity group dialog, and the group list shows a Partitioned tag. For these groups the parent table is excluded from job execution and only its partitions are processed, and entity counts in rules and operations show only the number of partitions.
[ENH] #3860 upgrade node to v22.22.2
The Gigantics binaries are now built on Node.js 22 (v22.22.2). You still don't need to install Node.js: the bundled runtime is included in the download, along with the native database driver components (SQLite, Oracle, ODBC and IBM DB2), which are now packaged inside the distribution. The Node.js version in use is shown in Global Settings > About.
[ENH] #3869 Add en_AU and en_NZ
Added Australian (en_AU) and New Zealand (en_NZ) locales to discovery and data generation. Gigantics can now detect and generate local postcodes, landline and mobile phone numbers, vehicle plates, states/regions, addresses and person names (including Maori names for New Zealand), as well as Australian TFN, Medicare, ABN and BSB codes and the New Zealand IRD number, with check-digit validation where applicable. Discovery is also more robust on deeply nested documents.
[ENH] #3876 Subsetting should keep referencial integrity
Limit, Where, Query and Synthesize operations now have an "Ensure Referential Integrity" option. When it is enabled, Gigantics keeps foreign key relationships valid in the resulting subset, so child rows only reference parent rows that are also included, and synthesized child records reuse primary keys that exist in their parent tables. It works with Oracle, PostgreSQL, MySQL, SQL Server, SQLite, DB2i, DB2z and Databricks. You can also add or remove foreign key relations by hand from the Relations tab of an entity or field in the schema. This release also fixes progress bars going over 100% in synthesize append jobs and a crash when synthesizing JSON/complex columns.
[ENH] #3918 Add LATAM locales to cactus
PII discovery and data generation now support the Argentina (es-AR), Peru (es-PE), Ecuador (es-EC) and Chile (es-CL) locales for names, addresses, cities, states, zip codes, phone numbers, vehicle plates and national identifiers (cédula, Ecuadorian cédula, Chilean RUT) with check-digit validation. Chile also gets new labels for passport, bank account (CuentaRUT), driver licence and health data (clinical record, health insurance provider and affiliation number). Spanish name detection was also improved, and the field options panel in anonymize and synthesize rules no longer overlaps the fields grid.
[ENH] #3997 Upgrade to TypeScript 6.0
Upgraded TypeScript to version 6.0 across the whole codebase: server, interface, data engine, discovery models and website. Type-checking was added to the data engine and discovery model builds. Internal change with no change in behavior.
[ENH] #4085 Web redesign - Marketing
The gigantics.io marketing website has been redesigned, in English and Spanish. It has a new home page and new Platform pages (Discover, Anonymize, Provision and Govern). It also adds Connectors, Guides and Licensing pages, plus use case pages for compliance, non-production environments, secure data sharing, and data for analytics and AI. The About, Blog, Contact, Demo and ROI calculator pages were rebuilt. The old Features, Resources, Pricing and Data Anonymization Software pages now redirect permanently to Platform, Guides, Licensing and Platform > Anonymize.
[ENH] #4090 Support IS NULL and IS NOT NULL operators in WHERE clauses
The Where operation now offers two new operators, is null and is not null, for string, number, decimal, date and boolean fields, to keep or discard records whose field has no value. When one of them is selected, the value box is hidden. SQL taps run them as IS NULL / IS NOT NULL, Salesforce as = null / != null in SOQL, MongoDB in its $match stage, and CSV, File Storage and Redis check each record after reading it (a missing field or a null value counts as null; an empty string doesn't).
[ENH] #4093 Add a new format (last name, first name) to the fake name generation function
The fake data function for the person/name label has a new type, full (last, first), which generates the last name followed by the first name, separated by a space (for example Martin Laura), for columns that store names in that order. It is available in the Use fake data options of Anonymize and Synthesize, in the Edit options for all fields panel, and in custom code as $fn.fake(..., { label: 'person/name', type: 'last_first' }). In the type selector the existing options now read first, last and full (first, last).
[ENH] #4098 feat(cactus): detect compact dates YYYYMMDD and DDMMYYYY
Discovery (cactus) now recognizes dates written as eight digits without separators, in YYYYMMDD (19501203) and DDMMYYYY (13061950) formats, and labels those columns as dates instead of as generic numbers or identifiers. A value only counts as a compact date if it is a real calendar date between 1900 and 2099, so other 8-digit codes are not mistaken for dates.
[ENH] #4119 Online discover for the current table in Edit labels screen
The Edit labels view of an entity now has a Discover entity button that runs discovery right away on that entity only, without launching a discover job. Its dialog offers Merge with previous discovery (unchecking it asks for confirmation, because it resets both detected and user-edited labels of the entity), Row limit (up to 5,000 rows) with First / Last / Random records, and Labels probability. The labels are updated in the latest discovery, without creating a new discovery version. The button requires the Run discover permission and is disabled while a discover job is running and on partition tables of an entity group. This also fixes the label-removal events recorded by discovery, which were built from the added labels.
[FIX] #3615 Fix invite users input
Pressing Enter in multi-line text fields, such as the email list when inviting users, now adds a line break instead of submitting or closing the form, so you can enter several addresses one per line.
[FIX] #3672 SQL Server error can be an object
When a SQL Server entity failed in a job, the error was returned as an object and the job progress view broke with a "String cannot represent value" error. The error is now converted to readable text, so the job progress and the entity error message display correctly.
[FIX] #3679 Data sampler not showing data
The data sampler showed an empty grid with no columns for PostgreSQL taps. It now passes the correct schema when reading column information, so sample data is displayed again.
[FIX] #3681 Apply to all fields of tag in all pages
In the anonymize operation's label view, "Apply action to all" only applied the selected function to the fields on the page currently shown. It now applies it to every field with that label across all pages, and the dialog shows the real total number of fields affected.
[FIX] #3682 Move labeled fields to top not working
The "move labeled fields to top" option in the anonymize and synthesize operations now puts every labeled field at the top in a stable order. Previously some labeled fields stayed lower down, and in anonymize, fields that shared the same sort position could disappear from the table view even though they still appeared in the label view.
[FIX] #3706 Disabling all triggers per batch is causing SCH-M locks
In SQL Server sinks using UPDATE mode, triggers and constraints were disabled and re-enabled on every write batch, taking repeated schema modification (SCH-M) locks that blocked concurrent reads from the tap and caused timeouts. Triggers and constraints are now disabled once at the start of the write and re-enabled once at the end (for both INSERT and UPDATE modes).
[FIX] #3710 SQL Server driver does not honor ORDINAL_POSITION from the database schema
When writing to SQL Server with bulk insert, the column order did not follow the table's ORDINAL_POSITION, so values could end up in the wrong columns (e.g. coordinates in text fields or addresses in date fields), especially in wide tables after anonymization. Columns are now written in the order defined by the database schema.
[FIX] #3714 schema cached fetch
Fixed the model schema not loading or refreshing correctly in some cases, because the schema request was cached incorrectly when no specific schema version was selected. The current schema of the model is now loaded and updated as expected.
[FIX] #3731 Refactor variables to project-level scope
Variables can now be defined per project as well as globally. A new Variables page in the project configuration lets you create, edit, search and delete project variables. Where a project variable has the same name as a global one, the project value is used. The variables popover in rules picks up the current project automatically, and any logged-in user can now read variables.
[FIX] #3739 Global settings credentials leak
Security fix: the global settings, which include authentication provider configuration and secrets, could be queried by users without the required privileges, including unauthenticated users on the sign-in page. Global settings are now restricted to the root user, and the sign-in and preferences screens use a new public query that only exposes the authentication type and which providers (Google, GitHub, Slack, SAML) are enabled.
[FIX] #3781 Master license is not picked if tap already assigned a license
When running jobs, a tap with its own assigned license ignored the master license. The master license now always takes priority, and the tap's assigned license is only used when no master license exists, matching what the tap screen shows.
[FIX] #3785 Don't load orphan taps
Taps belonging to projects that have been deleted are no longer listed in the instance-wide taps list in the licensing settings. Previously these orphan taps could appear in the list or cause it to fail to load.
[FIX] #3786 Project cascade delete
Deleting a project now also removes everything that belongs to it: models and their rules, jobs, taps, sinks, environments, variables, labels, custom functions, datalists, password managers, dictionaries, API keys and endpoints, events, and the project avatar. The project is also removed from pending invitations. Deleting an organization now runs the same cleanup for each of its projects and also removes its invitations and events, so no orphaned data is left behind.
[FIX] #3795 PostgreSQL sampler doesn't view data in UI correctly
Data previews of PostgreSQL tables outside the default schema did not show the right data in the UI, because the schema in the table name was ignored and the public schema was assumed. The sampler now detects the schema from qualified table names (schema.table, including quoted identifiers), so previews and field lists show the correct table.
[FIX] #3805 Importing labels from another entity does not import PII nor sensitivity
When importing labels from another entity in discovery, only the labels were copied. The PII flag and sensitivity of each field are now imported as well.
[FIX] #3806 PostgreSQL db ops fail if table name has capital letters
PostgreSQL operations failed on tables whose names contain capital letters, because the name was converted to lowercase when counting rows, estimating table size or reading the table list. Table names now keep their exact case, so these tables are discovered, counted and processed correctly.
[FIX] #3809 Duplicated discover labels
Running discovery again on a model no longer duplicates the labels already assigned to fields. Discovery now starts from the schema alone, without the previous discovery results.
[FIX] #3812 fix: use fully quoted qualified names in postgressql ops
PostgreSQL operations now always use fully qualified, quoted schema and table names, so tables in non-default schemas or with mixed-case or special names are read and written correctly. Disabling constraints now targets the right schema and table, and if global schema metadata is missing the target schema is created per table as a fallback.
[FIX] #3815 fix schema generated File type
The type returned by the file upload API, used for user and project avatars and for datalist files, is now always published as File in the GraphQL schema. Previously its name depended on how the server was built, so the generated schema could expose it under a different name and break API clients and type generation that expect File.
[FIX] #3816 don't show debug banner on user generated errors
The technical error banner (with debug details) is no longer shown for expected errors such as validation errors, missing permissions, expired sessions or items not found. These keep showing their usual message, and the debug banner is reserved for real internal errors.
[FIX] #3817 Entity groups show all entities in anonymize operation
In the anonymize and synthesize operations, entity groups (partitions) now show only the entities that were included, instead of every entity in the model. Also, when editing an existing Select operation, entities from Include/Exclude groups that had not been saved before now appear, with their key and PII fields filled in from discovery.
[FIX] #3819 Update the number of rows before processing entities in jobs
Jobs now always run an exact row count for each table before processing it, instead of relying on the database catalog estimate when no custom query or Where filter is used. Estimates could be stale, which caused wrong progress figures and, on restart, entities being considered finished too early. Applies to the PostgreSQL, MySQL, SQL Server, Oracle, DB2 for i and DB2 for z/OS drivers.
[FIX] #3820 Labels view in anonymize operation with groups
The labels view in discovery included fields from partition entities of entity groups, unlike the entity view; now only fields from main and ungrouped entities are listed. In addition, importing labels from another entity now lets you choose between Merge (adds only missing labels and settings) and Replace (copies everything from the source entity), instead of always overwriting PII and sensitivity.
[FIX] #3821 Import labels from another entity
Importing labels from another entity had unpredictable results because the PII flag and severity were always overwritten from the source entity. You can now choose between Merge, which only adds the labels and settings that are missing, and Replace, which replaces the target's labels with those of the source.
[FIX] #3822 Improve options panel in anonymize/synthesize improvements
The field options and custom code panels in the anonymize and synthesize operations are easier to use. If you switch to another field with unsaved changes, a dialog now lets you cancel, discard or save them. While the options panel is open, the Type, Is PII and Nullable columns are hidden to leave more room for the fields table. Layout problems were also fixed: the table no longer overflows past the Back/Save footer and operation pages use the full page width.
[FIX] #3823 PostgresSQL potential deadlocks
Reduced the risk of deadlocks and hung jobs when writing to PostgreSQL: rows are now written in primary key order so locks are always acquired in the same order, and transactions use lock and statement timeouts. A new snapshot read option lets a rule read the source table through a temporary copy when the same table is read, transformed and written back. Also fixed: inserts now specify column names, the "continue on fail" behaviour now also works when an update fails, and table details are logged on errors to ease troubleshooting.
[FIX] #3830 Unsaved data modal shown after save
In the Anonymize and Synthesize operations of a rule, the unsaved changes dialog could appear when switching between fields even though nothing had been modified. The dialog now only appears when there are real pending changes.
[FIX] #3836 Fix licenses panel
The licensing panel has been reworked for clarity. The license list shows the tier types and their usage, dates are shown as YYYY-MM-DD instead of relative times, invalid licenses can now be removed, and a "Show expired and cancelled" switch (off by default) is available directly in the toolbar. When assigning a license to a tap, only active licenses are offered, with their expiration date, and the summary cards list all drivers for each tier.
[FIX] #3837 Columns in insert statements need to be quoted in PostgreSQL driver
The PostgreSQL driver now quotes column names in INSERT statements. Writing to tables with columns named after reserved words (for example "references") no longer fails with a syntax error.
[FIX] #3839 Jobs with very few records should not show @0/s in rate
Entities with fewer than 1,000 records always showed a rate of 0 records/s in job progress. The real rate is now calculated when each entity finishes, and a migration estimates the rate of previously completed jobs from their timestamps.
[FIX] #3848 ETA in jobs should show the real Estimated Time of Arrival
Job times now show real values instead of a rough ETA. In the jobs list, ETA is renamed End and ETT Elapsed (with a tooltip for estimated values), When becomes Start, and finished or failed jobs show their actual end time and duration. The estimates for running jobs are worked out from the rows read so far. A job with no activity for the time set in stallTimeoutSec (900 seconds by default; for example after a silent network drop) now fails with a "job stalled" error instead of hanging indefinitely.
[FIX] #3850 Mongo pubsub exceeds BSONObj size
Jobs that produce a lot of log output no longer fail with MongoDB's "BSONObj size" (16 MB) error. Live job log messages are now sent in smaller batches and no longer include the full rule definition.
[FIX] #3851 Dropped streaming logs
Some log lines were lost when following a running job's log in real time. Log delivery has been reworked so all messages are sent to the job log stream without being dropped.
[FIX] #3853 Progress in scan jobs stays in 0%
Scan jobs stayed at 0% progress, particularly with PostgreSQL and Databricks taps. Progress is now based on the number of entities actually scanned (respecting the configured filters, also in SQL Server), is capped below 100% until the job really finishes, and the Entities tab shows each scanned entity.
[FIX] #3857 Backup dataset should be reused on job restart
When a job with backup is restarted, it now continues writing to the existing backup dataset instead of creating a new one. The backup is committed batch by batch, so data saved before a cancellation is kept, and the restart resumes from the rows already committed (offsets are now also supported for MongoDB, Redis and Salesforce). Backups from cancelled or failed jobs can be browsed and used for rollback, rollback now restores all saved rows, and backup datasets can no longer be downloaded because they contain original data.
[FIX] #3858 PostgresSQL doesn't capture field length
The PostgreSQL driver now reads the full column type, including its length (for example varchar(100) or numeric(10,2)), so field lengths appear correctly in the schema. Type detection was also improved: array columns, int2/int4/int8, smallserial and uuid are now recognized.
[FIX] #3870 drivers getType type matching
Column types that include a length, precision or other modifiers (for example varchar(255), timestamp(6) or number(10,2)) were not always recognised and could be mapped to the wrong field type. Type detection now matches on the type name prefix in the Databricks, DB2 for i, DB2 for z/OS, SQL Server, MySQL and Oracle drivers, and MySQL-only types were removed from the other drivers' mappings.
[FIX] #3936 Owner of an organization cannot manage users
In organizations where every member is an owner, the member list disabled all controls, so no owner could remove or modify other members. Now the controls are only disabled on your own row and on the rows of other owners: owners can manage the rest of the members, can still select other owners to assign them to projects, and can't be removed by another owner (an owner leaves the organization with the Leave option, as long as another owner remains).
[FIX] #4079 Truncate does not qualify tables
On Oracle sinks defined with an OWNER, truncating or dropping target tables failed with ORA-00942 because the table name was not qualified with the owner. Truncate and drop now use the owner-qualified name. In addition, if preparing the sink fails, constraints and triggers are always re-enabled, and the load job is reported as failed instead of as successful.
[FIX] #4081 Ignore already anonymized values option slows the job dramatically
The "ignore already anonymized values" dictionary option no longer slows jobs down dramatically. Dictionary lookups are now much faster (internal tests went from about 20 to about 4,000 records per second). The option is now offered only when the job overwrites the tap, it is off by default, and it is no longer inherited from rule default options. Rules, pipelines and pending jobs that had it saved under the old default are reset automatically.
[FIX] #4086 Oracle: rules with a Where filter fail with NJS-097: no bind placeholder named ":PARAM0"
On Oracle taps, any rule or data preview with a Where filter failed with "NJS-097: no bind placeholder named :PARAM0", and with several conditions values could be bound to the wrong condition. Bind parameters are now numbered consistently, so Where filters work correctly in both job execution and previews.
[FIX] #4087 Large databases improvements
Jobs on very large databases are now more reliable and faster. Interrupted jobs resume from the last committed batch instead of starting over (with optional automatic resume of jobs whose worker died), stalled jobs are detected and failed instead of hanging, and a job with failed entities is never reported as successful; PostgreSQL sinks load data with COPY, and large tables can be read in parallel key ranges, optionally spread across helper workers. New server settings under jobs (stallTimeoutSec, heartbeatSec, leaseStaleSec, autoResume, maxAutoResumes, concurrent, shardWorkers, copy) and a new exactCount option for debaser.yml rules control this behavior. It also fixes several correctness issues: sha/hmac/encrypt functions, shuffle duplicating values, seeds not applied to all generators, referential-integrity subsetting limits, constraints left disabled on non-job tables in PostgreSQL/SQL Server, and rollback backups that stored anonymized rows instead of the original data.
[FIX] #4091 Fix Where operation across drivers and harden value handling
The Where operation behaved differently from one driver to another. Empty groups produced invalid SQL (WHERE ()), the boolean True / False operators didn't filter, nested groups failed on Databricks, CSV and File Storage taps ignored the filter, and Redis silently accepted operators it couldn't apply. Now empty groups are ignored, True / False compare against true / false, Databricks handles nested groups, CSV and File Storage apply the filter both in jobs and in data previews, and Redis reports an unsupported operator instead of ignoring it. With Ensure Referential Integrity, parent and child tables keep their Where conditions. Values can no longer start with __: or ~~, which the query builder would have treated as raw SQL. Changing a rule's field resets the operator to one that fits the new field type, and the operator labels are translated.
[FIX] #4094 Use HMAC to encrypt dictionary keys
Dictionary keys are now computed as an HMAC-SHA256 of the original value with a secret, instead of a plain MD5 hash, so they can't be reversed or recomputed without the secret. The secret is read from the GIG_DICTIONARY_SECRET environment variable or the new dictionarySecret setting. If neither is set, a dictionary.secret file is generated next to the configuration file on the first run; it must be backed up and kept, because jobs fail if the secret of an existing HMAC dictionary changes. Existing dictionaries keep their MD5 keys until they are cleared. Importing a file whose kind of key differs from the dictionary's entries is rejected. The dictionary dialogs warn about jobs that are still using the dictionary, and clearing it while jobs run no longer brings the removed entries back. A migration merges duplicated per-project dictionary metadata.
[FIX] #4095 Data leaks in job logs
Job logs no longer contain record values. Errors from drivers and databases, job failure reasons and the console mirror now hide the values, primary keys included, at every log level. In custom code, log describes records and objects only by their columns and the type and length of their values. Strings and numbers you log yourself are still written as they are. The job log is now HTML-escaped and sanitized, which also fixes HTML injection in the log viewer and the SQL highlighter. The default job log level goes back to 1. Job logs are deleted with their job and expire after the days set in the new jobLogRetentionDays setting (90 by default, 0 keeps them until the job is deleted; jobs.logRetentionDays overrides it). The rollback backup of a pump that overwrites its tap now saves the original rows as they are read, each batch before it is written. It requires the Update or Merge load mode, a restart doesn't copy already saved rows again, and it's no longer offered when loading from a dataset.
[FIX] #4097 Oracle: qualify tables in referential integrity queries with the owner
On Oracle taps with an Owner set, subsetting with Ensure Referential Integrity could fail with ORA-00942 (table or view does not exist), because the queries that filter child tables against their parents used table names without the owner. Those queries now qualify every table with the tap's Owner (in uppercase, as the Oracle driver does), so referential-integrity subsets work when the tables belong to a schema other than the connecting user's.
[FIX] #4102 The Run now button is disabled while the job is launching, so a second click can't start a duplicate job.
The Run Now button of the new job panel (also Run Later and Save as Pipeline) now shows a loading state and can't be clicked again until the job, load, copy or pipeline has been submitted, so a double click no longer launches a duplicate job.
[FIX] #4103 Where filter issues
On Oracle taps, Where filters on DATE or TIMESTAMP columns failed with ORA-01861, because the date was sent as ISO text that Oracle can't convert implicitly. Dates are now sent to Oracle as real dates (and as TIMESTAMP literals in referential integrity queries), so date filters work.
[FIX] #4104 DB2: where filters fail due to $n placeholders, invalid date literals and random-limit SQL
On DB2 for i and DB2 for z taps, any Where filter with a value failed, because the query used $1-style placeholders that DB2 rejects (the SSH connection lost the values entirely). Date filters were sent in an ISO format DB2 doesn't accept, and combining a Where with a random read produced invalid SQL. Filters now use DB2 placeholders, dates are converted to DB2 DATE / TIME / TIMESTAMP format according to the column type, ends with works on fixed-length CHAR columns, and random reads with a Where filter work.
[FIX] #4105 Mongo: date where filters never match
On MongoDB taps, Where filters on date fields never matched (and is not matched every document), because the date was compared as ISO text against values stored as BSON dates. The value is now converted to a date before it is compared, in jobs and in data previews, so date filters return the right documents.
[FIX] #4106 MSSQL: numeric where values bound as Int (decimals truncated, bigint overflow)
On SQL Server taps, every number in a Where filter was sent as a 32-bit integer. Decimals were truncated (price > 9.99 ran as price > 9) and values outside the 32-bit range failed. Decimals are now sent as decimals and large integers as BIGINT, so numeric filters compare the exact value.
[FIX] #4107 MySQL: where column names not quoted; ISO dates compared with ignored Z
On MySQL taps, Where filters failed on columns whose names are reserved words or contain spaces, because the column names weren't quoted. Date filters ignored the UTC offset of the picked date, which shifted the comparison by the time zone difference. Column names are now quoted and dates are compared at the right moment, including TIME columns. Data previews with a Where filter also send the values in the right order.
[FIX] #4108 Salesforce: date/time fields receive datetime literals; escaping issues
On Salesforce taps, Where filters on Date and Time fields failed because they always received a date-time literal. Text values that looked like a date were left unquoted, and backslashes in text weren't escaped. Date fields now receive a date, Time fields a time, values are quoted according to the field type, and backslashes are escaped, so these filters produce valid SOQL.
[FIX] #4109 SQLite: date where filters compared as text
On SQLite taps, Where filters on dates compared the stored text against the ISO value, so they returned wrong rows whenever the stored format was different or the date was saved as an epoch number. Dates are now compared as dates whatever the storage format, in local time. TIMESTAMP and lowercase date column types are recognized as dates, and BOOLEAN columns as booleans.
[FIX] #4110 Postgres: LIKE on non-text native types fails; timezone shift on naive timestamps
On PostgreSQL taps, like, starts with and ends with failed on columns that aren't text, such as uuid, inet or enum, and decimal values compared against integer columns failed. Date filters on timestamp columns without a time zone were shifted by the UTC offset. These columns are now compared as text or as numeric where needed, and dates on columns without a time zone are compared in local time, in both jobs and data previews.
[FIX] #4111 Databricks: bigint params typed INTEGER; boolean columns typed as string
On Databricks taps, numbers outside the 32-bit range in a Where filter were sent as INTEGER and failed, and BOOLEAN columns were read as text, so they didn't get the True / False operators. Large integers are now sent as BIGINT, BOOLEAN columns are booleans, and date filters follow the column type (DATE, TIMESTAMP_NTZ or TIMESTAMP). On servers without query parameter support, values are inserted as escaped literals.
[FIX] #4112 Oracle: CLOB equality and CHAR padding in where filters
On Oracle taps, is and is not on CLOB / NCLOB columns failed with ORA-00932, and filters on CHAR / NCHAR columns missed rows because of the trailing blank padding. An empty text value never matched, because Oracle stores empty strings as NULL. CLOB comparisons now work, CHAR values are compared without the padding, and an empty text value matches NULL (or not NULL for is not).
[FIX] #4113 CSV / File storage / Redis - In-memory where evaluator: wrong date, null and LIKE semantics
On CSV, File Storage and Redis taps, where the filter is applied by Gigantics record by record, dates were compared as text. Empty values matched less than / before, empty CSV cells didn't match is null, _ wasn't a like wildcard, * was treated as a wildcard, and starts with / ends with were case sensitive. Now dates are compared as dates (ISO and other common formats), empty values never match ordering operators, empty CSV cells count as null, and like uses % and _ (with * as plain text) and works on multi-line values. starts with and ends with now ignore case, and numbers are compared as numbers.
[FIX] #4114 RI where SQL: unquoted identifiers, dialect date literals, and not gated to SQL drivers
With Ensure Referential Integrity, Where conditions written into the generated SQL used dates in a format some databases (DB2, SQLite, Oracle and others) don't accept. Empty values on number or date fields were compared with '', and % or _ typed in a starts with / ends with value acted as wildcards. The option was also offered, and applied, on taps that aren't SQL databases. Dates are now written in each database's format according to the column type, empty values on typed fields give a clear error, and wildcards in the value are matched literally. The checkbox is only shown on SQL taps, including Google Cloud SQL, where it was missing.
[FIX] #4115 Where editor: default values shown but not stored, causing crash and empty comparisons
The Where editor showed a default value (0, today's date) that was never saved, so the rule ran with an empty value. That made the job fail or compare against an empty string, and changing a rule's field cleared its value. Defaults are now stored, both when a rule is opened and when its field changes. The default date is today at midnight, picked dates have no milliseconds, and fields that can't be filtered appear disabled with their type. The server also rejects an empty value on a number or date field with a clear error.
[FIX] #4131 Data sampling on large tables reads most of the table and can time out
View random rows read a large part of the table, or all of it, to show 5 rows: on a table of hundreds of millions of rows it took minutes and could end with an internal server error. SQL Server, PostgreSQL and Oracle now sample about 10,000 rows instead of 100,000 to 200,000 (and Oracle no longer returns fewer rows than asked for on tables of over 10 million rows). MySQL, DB2 for i and DB2 for z, which have no table sampling, now seek random points of a single integer primary key through its index instead of ordering the whole table at random, and fall back to the previous query on tables without such a key. See Previewing the data of a large table.
Ready to upgrade?
Just follow the standard procedure for installing the new version. Check the Installation guide for details.
Acknowledgements
Contact us to make suggestions and report bugs.
Thanks to everyone who helped us with their feedback.