File Storage Driver Documentation

This documentation explains the File Storage driver implementation and how connection parameters map to UI fields in the Gigantic application. The driver reads and writes data files (CSV, JSON, YAML, XML, Excel, Parquet and EDI) stored in a folder. The folder can be on the local disk, on a remote server over SSH/SFTP, in Google Drive or in SharePoint.

File Storage is listed under Files & storage in the driver selector. The Local Disk and SSH / SFTP providers are available in the open source license tier; the Google Drive and SharePoint providers require a Cloud license tier.

             ┌──────────────┐
             │ File Storage │   one tap or sink = one folder
             └──────┬───────┘
    ┌───────────────┼───────────────┬──────────────────┐
    v               v               v                  v
Local Disk     SSH / SFTP     Google Drive        SharePoint
(path)         (remote path)  (folder ID)         (folder item ID)
    │               │               │                  │
    └───────────────┴───────┬───────┴──────────────────┘
                            v
        file format by extension: .csv .json .yaml/.yml .xml
                                  .xlsx/.xls .parquet .edi
                            v
                 one file (or sheet) = one entity

File Storage vs CSV Driver

Connection Parameters

The fields of the Connection section depend on the Provider.

Google Drive

With Service Account the OAuth2 fields are hidden and not required. The folder must be shared with the service account.

SharePoint

The driver gets an app-only token (client credentials, scope https://graph.microsoft.com/.default) and uses Microsoft Graph, so the app needs Graph permissions to read and write files in the site.

A siteId in Custom Params overrides the Site ID field.

SSH / SFTP

The Type of the Authentication section chooses how the driver logs in:

If there is no key and no password the connection fails with SSH authentication required. All the files of the remote Path are used: to use only some of them, set filePattern in Custom Params.

File Naming (sinks only)

Pattern variables:

Examples for the entity Filiale read from Filiale.csv:

A Custom pattern with an empty pattern falls back to Add timestamp.

Technical Details

Files and entities

• Each file in the folder is an entity. The entity name is the file name without the extension (customers.csv becomes customers)
• If two files have the same name with different extensions (customers.csv and customers.json), both entities keep the extension in the name
• Only the top level of the folder is read. Subfolders are skipped: use one tap per folder
• Files with an unsupported extension, with no fields, or that cannot be parsed are skipped
• The entity description is <FORMAT> file: <file name>
• The File pattern supports * (any characters except /) and ** (any characters)
• Where conditions, limit and offset are applied by Gigantics while reading the file (see Filters applied by Gigantics)

Formats

The format is chosen by the file extension.

Excel notes:

• Old binary Excel 97-2003 .xls files are not supported: they are skipped with the error old binary Excel 97-2003 (.xls) files are not supported, save the file as .xlsx. A .xls file that is an .xlsx workbook inside is read
• A workbook with several sheets gives one entity per sheet, named <file>.<sheet>: the sheets Customers and Orders of Report.xlsx are the entities Report.Customers and Report.Orders. A workbook with one sheet is the entity <file>

Parquet type mapping when reading. The logical (converted) type of the column comes first, so text columns are strings:

Parquet type mapping when writing:

A column with a value that does not fit its type (for example a number column with the value n/a) is written as UTF8.

Providers

Output format

Each entity is written in the format of its source file, with the same extension (Excel entities are always written as .xlsx). Entities that do not come from a file (for example the tables of a database tap) are written as CSV.

The Test button checks that the Path exists, also for a sink: a sink whose Path is not created yet fails the test, but creates the Path when it runs.

Write modes

The File Storage driver writes one file per entity and run. The write mode decides what happens to the files of earlier runs:

With Add timestamp and Append, each run adds a new file. A file with the same name as the new one is always replaced, in every mode.

Write batches are 1000 records unless the job or rule sets another write batch size. Retry On Fail has no effect on this driver.

Authentication Options

Changing the Provider sets the authentication back to the first option of the provider.

SSH Tunnel Support

The form shows the Enable SSH? option, but the File Storage driver does not use SSH tunnels. To reach files on a remote server, use the SSH / SFTP provider.

API Endpoints Used

File Storage connections are used in:

• Tap creation (data source discovery from files)
• Sink creation (data destination for anonymized data as files)
• Pipeline execution (data extraction from and loading to files)

Node.js Driver Dependencies

Custom Params

Custom Params (YAML) are merged into the driver configuration, so they can set or override any of these keys:

# All providers
basePath: /data/input          # Overrides Path / Folder ID
filePattern: "*.json"          # Overrides File pattern
providerType: local            # local, ssh, gdrive or sharepoint

# Sinks
fileNaming: custom             # timestamp, original or custom
fileNamingPattern: "{entity}_{date}{ext}"

# SSH / SFTP
sshPassword: secret            # Password login (or the Password authentication)
sshKeyContent: |               # Private key content instead of a key file
  -----BEGIN OPENSSH PRIVATE KEY-----
  ...
  -----END OPENSSH PRIVATE KEY-----
sshPassphrase: secret          # Passphrase of the private key

# Google Drive
serviceAccountKey: |           # Service account JSON key (or the Service Account authentication)
  { "type": "service_account", "client_email": "...", "private_key": "..." }

# SharePoint
siteId: contoso.sharepoint.com,guid1,guid2   # Overrides the Site ID field

Example Configurations

SFTP with a password from a variable

Provider SSH / SFTP, Path /home/data/export, SSH Host sftp.example.com, Username gig, authentication Key file. The password comes from the Gigantics variable SFTP_PASSWORD, so it is set in Custom Params instead of the Password field:

sshPassword: $SFTP_PASSWORD

SFTP, only the CSV files

Provider SSH / SFTP, any authentication:

filePattern: "*.csv"

Google Drive with a service account

Provider Google Drive, Folder ID of a folder shared with the service account, authentication Service Account, and the JSON key in Service account key:

{ "type": "service_account", "client_email": "gig@project.iam.gserviceaccount.com", "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n" }