Redis Driver Documentation

This documentation explains the Redis driver implementation and how connection parameters map to UI fields in the Gigantics application. Redis is a key-value store, so the driver groups keys into entities by their prefix and turns each key into one record.

Redis is listed under Big data in the driver selector and is available in the open source license tier.

Connection Parameters

UI Field	Technical Implementation	Description	Required	Default Value
Host	`socket.host` in the `redis` client	Hostname or IP address of the Redis server	Yes	localhost
Port	`socket.port` in the `redis` client	Port the Redis server listens on	Yes	6379 (from driverPorts)
Database	`database` in the `redis` client (sent as `db`)	Numeric Redis database index. Leave empty for database 0	No	0
User	`username` in the `redis` client	ACL user (Redis 6 or later)	No	None
Password	`password` in the `redis` client	Password of the user, or the `requirepass` password	No	None

The connection is checked with a PING command.

Technical Details

The Redis driver uses the redis Node.js library (node-redis v5).

Keys and entities

The driver scans all keys with SCAN and groups them into entities:

The entity name is the part of the key before the first :. The keys customer:1 and customer:2 belong to the entity customer
A key without : is an entity by itself (for example leaderboard)
If a prefix has keys of more than one Redis type, there is one entity per type, named <prefix>@<type>. The hashes user:1 and user:2 and the string user:count are the entities user@hash and user@string
The number of keys is shown as the entity row count, and the description is Redis <type> (<n> keys)

Redis keys                     Entities
---------------------------    -----------------------------------
customer:1        (hash)  ─┐
customer:2        (hash)  ─┴─> customer     Redis hash (2 keys)
session:ab12      (string)───> session      Redis string (1 keys)
leaderboard       (zset)  ───> leaderboard  Redis zset (1 keys)
user:1            (hash)  ───> user@hash    Redis hash (1 keys)
user:count        (string)───> user@string  Redis string (1 keys)

Fields

Field	Type	Description
`_key`	string	Full Redis key. It is the primary key of the entity
hash fields	string	For hashes, one field per hash field found in the keys
`_value`	string or object	For strings, lists, sets, sorted sets and streams, the value of the key
`_ttl`	number	Seconds until the key expires. Empty when the key has no expiry

Reading process

Reads the key with the entity name, if it exists, and every key that matches <entity>:*. If the entity name contains : or *, it is used as the match pattern. An entity <prefix>@<type> reads only the keys of its prefix of that type
Each record has _key, _type (the Redis type) and _ttl
The value depends on the type:

Redis Type	Record
string	`_value` with the string
hash	One field per hash field
list	`_value` with a JSON array of the items
set	`_value` with a JSON array of the members
zset	`_value` with a JSON array of `{ value, score }`
stream	`_value` with a JSON array of `{ id, message }`, one per entry, where `message` has the fields of the entry

Where conditions are evaluated on each record in Gigantics, not in Redis (see Filters applied by Gigantics). A Read Query must be a JSON where object, for example {"status": "active"}. Limit and offset are supported.

Writing process

For each record:

The key is _key. If the record has no _key (for example a row of a SQL tap), the key is <entity>:<primary key>, with the values of the primary key columns joined by :. If the entity has no primary key, the key is <entity>:<n>, where n is the record number
The type is _type. If the record has no _type, it is the type of the Redis entity it comes from. A record of another source is written as a string if it has a _value field, and as a hash of all its fields if it does not
string: SET with _value. Other fields are not written
hash: the key is deleted and recreated with HSET with every field except _key, _ttl and _type, in one MULTI, so fields of an older value do not stay
list, set and zset: the key is deleted and recreated (RPUSH, SADD, ZADD) from the JSON array in _value
stream: the key is deleted and recreated with one XADD per entry of the JSON array in _value, with the same entry IDs. Consumer groups are not copied

Redis stores strings: dates are written in ISO 8601 format, objects and arrays as JSON, and empty values as an empty string.

A row of a SQL tap is written as a hash and not as a JSON string so that each column stays a field: the Redis entity has the same fields as the table, and each one can be masked or read back on its own.

If _ttl is a number of 0 or more, the key expires after that many seconds, as in the tap (SET with EX for strings, EXPIRE for the other types). Fractions of a second are rounded up, and a _ttl of 0 (a key with less than a second left in the tap) expires after 1 second. Keys with an empty or negative _ttl have no expiry.

Write modes:

Write mode	Behaviour
Drop	Deletes the keys of the entity before writing: the key `<entity>` and the keys that match `<entity>:*`, found with `SCAN` and deleted with `UNLINK` in batches of 1000. For an entity `<prefix>@<type>`, only the keys of that type. The other keys of the database are not touched
Truncate	The same as Drop
Append, Update, Merge	Writes the keys, overwriting keys that already exist

Authentication Options

Auth Type	UI Mapping	Description
None	No credentials	For Redis servers without a password
User / Password	User and Password fields	Sends the user and password. Leave User empty to use only the `requirepass` password
Password Manager	Password manager fields	Fetches the user and password from an external password manager

SSH Tunnel Support

The Redis driver connects through the host and port, so it supports SSH tunnelling. For detailed information on configuring SSH tunnels, please refer to the SSH Tunnelling documentation.

API Endpoints Used

Redis connections are used in:

Tap creation (data source discovery)
Sink creation (data destination for anonymized data)
Pipeline execution (data extraction and loading)

Node.js Driver Dependency

Library: redis
Purpose: Redis client for Node.js

Custom Params

Custom Params (YAML) are merged into the driver configuration:

url: redis://user:password@redis.example.com:6379/2  # Connection URL, replaces Host and Port
batchSize: 1000    # Read/write batch size when the job or rule sets none (1000)

When url is set, Host and Port are ignored. Use the rediss:// scheme for TLS connections.

Example Configurations

TLS connection

url: rediss://redis.example.com:6380

On this page