Database Drivers Overview

This document provides an overview of all database drivers currently supported by the Gigantic application for data anonymization processes.

Supported Database Drivers

Gigantic supports a wide range of database systems as both sources (taps) and destinations (sinks) for data anonymization:

Relational Databases

• MySQL - Connects to MySQL database servers
• PostgreSQL - Connects to PostgreSQL database servers
• Oracle - Connects to Oracle database servers, including support for Multitenant architectures
• Microsoft SQL Server - Connects to MSSQL database servers
• DB2 for z/OS - Connects to IBM DB2 for z/OS mainframe databases
• DB2 for i (AS/400) - Connects to IBM DB2 for i (AS/400) systems
• DB2x - Connects to IBM DB2 for i (AS/400) systems with SFTP-based write operations
• SQLite - Connects to local SQLite database files

NoSQL Databases

• MongoDB - Connects to MongoDB document databases
• Redis - Connects to Redis key-value stores; keys are grouped into entities by their prefix

Cloud Platforms

• Salesforce - Connects to Salesforce CRM using JWT OAuth authentication and Bulk API 2.0
• Databricks - Connects to Databricks SQL Warehouses and clusters (Unity Catalog catalogs and schemas) with a personal access token

File-Based Systems

• File Storage - Reads and writes CSV, JSON, YAML, XML, Excel, Parquet and EDI files on the local disk, over SSH/SFTP, in Google Drive or in SharePoint
• CSV - Reads from and writes to CSV files in a specified directory. It is no longer offered when creating a tap or sink: use File Storage

Connection Methods

Each driver supports different connection methods:

1. Direct Connections - Most relational databases support direct TCP/IP connections with standard authentication
2. File-Based Access - SQLite, CSV and File Storage drivers work with files. File Storage can also reach files over SFTP, in Google Drive and in SharePoint
3. SSH Tunneling - Most database drivers support SSH tunneling for secure connections through bastion hosts
4. Specialized Approaches - DB2x uses a unique template-based shell script execution system for write operations

Authentication Options

Drivers support various authentication methods:

• Username/Password - Traditional database authentication
• Windows Authentication - For MSSQL (Integrated Security)
• JWT OAuth - For Salesforce (JWT Bearer Flow with RSA private keys)
• Password Managers - Integration with external password management systems
• Access Tokens - For Databricks (personal access token in the Password field)
• No Authentication - For file-based systems like SQLite, CSV and File Storage

Write Retries (Retry On Fail)

All drivers write data in batches. When Retry On Fail is enabled and a batch fails, the driver does not stop at once: it splits the batch in half and writes each half again, first the left half and then the right one. Halves that fail are split again, until the batch is written or a single record fails.

batch of 8 records fails
        │
        ├── records 1-4 ── ok, written
        └── records 5-8 ── fails
               ├── records 5-6 ── ok, written
               └── records 7-8 ── fails
                      ├── record 7 ── ok, written
                      └── record 8 ── fails: the error is reported with this record

• The records written before the error are counted, and the error message says how many records of the batch were committed: Error <action> <batch size> into <entity> after committing <n> record(s): <error> (for example Error inserting 8 into customers after committing 7 record(s): ...)
• The failing record is reported with the error. What happens next depends on Continue Streaming On Fail
• Every write error is retried in this way, with no delay between attempts. It does not retry connections or reads
• Retry On Fail is disabled by default. It is next to Concurrency and Continue Streaming On Fail in the rule Default Options, in the run and launch job panels, in the pipeline concurrency options, and in the load dataset, copy dataset and load sink panels
• It applies to the PostgreSQL, MySQL, Microsoft SQL Server, Oracle, Salesforce and Databricks drivers. The other drivers ignore it

Technical Implementation

Each driver is implemented as a Node.js class that follows a common interface, with specific implementations for:

• Connection management
• Schema extraction
• Metadata retrieval
• Data reading
• Data writing

For detailed information about each driver's parameters, authentication options, and technical implementation, please refer to the individual driver documentation pages.

Driver Dependencies

For information about Node.js library dependencies for each driver, see the Driver Dependencies document.