Custom functions

It is possible to create customized functions using JavaScript. These functions can be created from two places:

• From Project: The functions created from this page can be used in any model that belongs to the project.

• From the Anonymize or Synthesize operations: Functions created from here only apply only to the selected column. These functions are not saved globally and thus they cannot be reused from other places in the application.

Creating a new function

The page for creating a new function is divided into 3 parts:

• Console: the area where the user codes the function.
• Preview: The user can see the result of the code being run by using the Input fields and clicking the Run button. It is also possible to add more records to the preview.
• Functions: A list of preset functions. When a function is clicked, a hint code of the function is pasted in the console.

Functions

chance (seed?)

Return a Chance instance

• seed: inital value to instantiate Chance with.

Examples

chance().integer({ min: 100, max: 1000 });
chance(3131231).integer({ min: 100, max: 1000 });
chance(Math.random() * Date.now()).integer({ min: 100, max: 1000 });
chance().address();
chance().paragraph({ sentences: 1 };

Check the [Chance package documentation ↗] (https://chancejs.com/) for more info

dateFormat (val, format)

Format date with a mask

• val: date value that can be a string, a number or a Date object.
• format: specifies the format in which the date will be shown.

return dateFormat($val, 'yyyy-mm-dd')

Examples

dateFormat(Date.now(), 'yy')
dateFormat('99/10/31', 'yy')
dateFormat('shortTime', 'yy')
dateFormat(new Date($val), 'yyyy-mm-dd')

db

Connects to a database using Gigantics drivers. List of methods available:

connect (driver, options)

Return a connection to a database.

• driver: driver string. It can be mysql, postgres, mssql, oracle, mongodb or sqlite3
• options: list of options to establish the connection.
  • host: endpoint address.
  • port: port number.
  • user: database user.
  • password: database password for the specified user.
  • db: database name.

Examples

const rows = await once(async () => {
  const d = await db.connect('mssql', {
    host: 'my.db.server',
    user: 'adventure',
    password: 's3CrEt',
    db: 'adventure'
  })
  const { recordset: res } = await d.query('SELECT * from Person.Person')
  return res[0]
})

decrypt (val, secret, options)

Decrypt data using aes-256-gcm algorithm

• val: string to be decrypted.
• secret: key to use to decrypt the string. It must be the same secret used with encrypt.
• options: only used for values encrypted with the older format (values that do not start with v2:).
  • saltLength: length of the salt parameter. Defaults to 5.

decrypt reads both the current v2: values and the values encrypted by earlier versions of Gigantics, so data encrypted before an upgrade can still be decrypted.

return decrypt($val, 'MyS3cr3tkey')

deepReplace (oldStr, newStr, options, record)

Replace the same data everywhere in the current record

• oldStr: old string value.
• newStr: new strig to replace with.
• options:
  • smartCase: if true word casing will be ignored when matching the string to be replaced.
• record: a record whose properties will be checked for data replacement.

Examples

deepReplace('prev text', 'new text', {}, $rec)
deepReplace('old text', 'new text', { smartCase: true })
deepReplace('old text', 'new text', { smartCase: true }, $rec)
deepReplace(
  'old text',
  'new text',
  { smartCase: true },
  { CUSTOMER: 'Acme', CUSTOMER_KEY: 'ACME', ID: 3433 }
)

encrypt (val, secret, options)

Encrypt data with aes-256-gcm

• val: value to be encrypted. Non-string values are converted to text. null values are not accepted.
• secret: key to use to encrypt the string.
• options: accepted for backwards compatibility and ignored.

The encryption key is derived from the secret once and reused for every value, so encrypting large tables is fast. Each value gets its own random initialization vector (IV). The output is a versioned string that starts with v2: followed by the IV, the authentication tag and the encrypted data in hex.

return encrypt($val, 'MyS3cr3tkey')

faker

Return a Faker instance

Examples

faker.address.city()
faker.lorem.lines()
faker.setLocale('es_MX')
faker.datatype.number({ min: 1, max: 100 })

Check the Faker package documentation for a complete list of available functions

file

Return a file object with the following methods available:

open (path, flags, ...args)

Open and possibly create a file.

• path: A path to a file. If a URL is provided, it must use the file: protocol.
• flags: A string indicating the type of operations to be performed on the returned file descriptor. Its values can be r, r+, rs+, w, wx, w+, wx+, a, ax, a+ or ax+
• args:
  • mode: A file mode. If a string is passed, it is parsed as an octal integer. If not supplied, it defaults to 0o666.

write (file, data, options)

Asynchronously write data to a file, replacing the file if it already exists. It is unsafe to call fsPromises.writeFile() multiple times on the same file without waiting for the Promise to be resolved (or rejected).

• file: A path to a file.
• data: The data to write. If something other than a Buffer or Uint8Array is provided, the value is coerced to a string.
• options: Either the encoding for the file, or an object optionally specifying the encoding, file mode, and flag. If encoding is not supplied, the default of utf8 is used. If mode is not supplied, the default of 0o666 is used. If mode is a string, it is parsed as an octal integer. If flag is not supplied, the default of w is used.

read (file, data, options)

Asynchronously read the entire contents of a file.

• file: A path to a file.
• options: An object that may contain an optional flag. If a flag is not provided, it defaults to 'r'.

append (file, data, options)

Asynchronously append data to a file, creating the file if it does not exist.

• file: A path to a file. If a URL is provided, it must use the file: protocol. URL support is experimental. If a FileHandle is provided, the underlying file will not be closed automatically.
• data: The data to write. If something other than a Buffer or Uint8Array is provided, the value is coerced to a string.
• options: Either the encoding for the file, or an object optionally specifying the encoding, file mode, and flag. If encoding is not supplied, the default of utf8 is used. If mode is not supplied, the default of 0o666 is used. If mode is a string, it is parsed as an octal integer. If flag is not supplied, the default of a is used.

readdir (path, ...args)

Asynchronously read a directory.

• path: A path to a file. If a URL is provided, it must use the file: protocol.
• options: The encoding (or an object specifying the encoding), used as the encoding of the result. If not provided, 'utf8' is used.

rename (oldPath, newPath)

Asynchronously change the name or location of a file or directory.

• oldPath: The path of the file aimed to be modified.
• newPath: New name or path for the file.

copyFile (src, dest, ...args)

Asynchronously copie src to dest. By default, dest is overwritten if it already exists. Node.js makes no guarantees about the atomicity of the copy operation. If an error occurs after the destination file has been opened for writing, Node.js will attempt to remove the destination.

• src: A path to the source file.
• dest: A path to the destination file.
• args:
  • flags: An optional integer that specifies the behavior of the copy operation. The only supported flag is fs.constants.COPYFILE_EXCL, which causes the copy operation to fail if dest already exists.

truncate (path, ...args)

Asynchronously truncate a file to a specified length.

• path: A path to a file. If a URL is provided, it must use the file: protocol.
• args:
  • len: If not specified, defaults to 0.

mkdir (path, ...args)

Asynchronously create a directory.

• path: A path to a file. If a URL is provided, it must use the file: protocol.
• args:
  • options: Either the file mode, or an object optionally specifying the file mode and whether parent folders should be created. If a string is passed, it is parsed as an octal integer. If not specified, defaults to 0o777.

mkdtemp (prefix, ...args)

Asynchronously create a unique temporary directory. Generates six random characters to be appended behind a required prefix to create a unique temporary directory.

• prefix: string that will preceed the generated temporary directory.
• args:
  • options: The encoding (or an object specifying the encoding), used as the encoding of the result. If not provided, 'utf8' is used.

base (path?)

Return base path

• path: File path to append to the base path.

home (path?)

Return home path

• path: File path to append to the home path.

Examples

file.append('/tmp/file.txt', '\nhello')

genLabel (label, locale, opts)

Return generated data from a label

• label: Gigantics label name.
• locale: Locale string.
• opts:
  • prefix: String to add before the fake data.
  • suffix: String to add after the fake data.
  • case: It can be upper, lower, title, kebab, snake.
  • type: Variant to generate, for labels that have types. For person/name it can be first, last, full (first name and last name) or last_first (last name and first name).
  • suffixEnabled: Boolean that indicates whether suffix use is enabled.
  • prefixEnabled: Boolean that indicates whether prefix use is enabled.

Examples

return genLabel('person/name', 'es-ES', {})

return genLabel('person/name', 'es-ES', { type: 'last_first' })

genLike (val, opts?)

Output random data that looks like the input

• val: String that will be used as reference for data generation.
• opts:
  • alphabet: String containing the characters that will be used in the resulting data.
  • re: Regular expression.
  • size: Size of the resulting string.
  • remove: String containing the characters to be removed.
  • add: String that will be added to the data.

genLike(23839293)
genLike('X121221Z')

genRe (re, opts?)

Return random data that matches the input regexp

• re: Regular expression.
• opts:
  • alphabet: String containing the characters that will be used in the resulting data.
  • re: Regular expression.
  • size: Size of the resulting string.
  • remove: String containing the characters to be removed.
  • add: String that will be added to the data.

Examples

genRe(/[A-D]{1}[1-3]{1}\d{4}/)
genRe(/\d{4}/) /* generate 4 digits numbers */
genRe('w{3}') /* 3 printable chars */
genRe(
  /[A-D]{1}[1-3]{1}\d{4}/
) /* letter from A to D, number from 1 to 3 then 4 digits */

hmac (val, secret, algo, outFmt)

Return a keyed hash (HMAC) of the input. The same value and secret always give the same output, so a value hashed in one table matches the same value hashed in another table, without a dictionary.

• val: Value to be hashed. Non-string values are converted to text. null and undefined are returned unchanged.
• secret: Secret key. It is required: the function fails if it is empty.
• algo: Algorithm to be applied. Defaults to SHA-256. It can take one of the following values: SHA-1, SHA-224, SHA3-224, SHA-256, SHA3-256, SHA-384, SHA3-384, SHA-512 or SHA3-512
• outFmt: Output format. Defaults to HEX. It can take one of the following values: HEX, B64 or BYTES

hmac($val, 'my-secret')
hmac($val, 'my-secret', 'SHA-512')
hmac($val, 'my-secret', 'SHA-256', 'B64')

When to use it:

• Instead of md5 or sha: these hash the value alone, so anyone can hash a list of known values (emails, national IDs) and compare the results to recover the originals. hmac mixes in a secret key, so without the key the hashes cannot be reproduced. Keep the secret out of the rule if possible, for example in a secret variable.
• Instead of the dictionary: use hmac when a hash is an acceptable replacement value and you need consistent values across tables, jobs and environments without storing a dictionary. Use the dictionary when the replacement must look like real data (a name, an email) or must keep the original format.
• Instead of encrypt: encrypt is reversible but gives a different output every time, so it cannot be used for keys. hmac is not reversible but always gives the same output, so it can be used for primary and foreign keys.

log

Send messages to the job logger. List of methods available

info (msg, ...args)

debug (msg, ...args)

trace (lev, msg, ...args)

warn (msg, ...args)

error (msg, ...args)

fail (msg, ...args)

Examples

log.info('my message')

md5 (str, encoding)

Return an md5 hashed value

• str: String to be hashed.
• encoding: If not specified, its value will be hex.

md5('Some text')
md5($val, 'base64')

mem

Reuse transformations values across entities and fields. List of methods available:

do (scope, oldVal, fn, options?)

Store a memory entry by applying a certain transformation if the value is not set already. If it was previously stored, it returns the memoized value.

• scope: The context or scope the value is or will be stored under.
• oldVal: Key value to look for in memory.
• fn: Transformation to be stored.
• options:
  • once: If true, the value won't be stored or updated when it previously existed. Its value is false by default.

set (scope, oldVal, newVal, options)

Store a value in memory under a certain scope.

• scope: The context or scope the value will be stored under.
• oldVal: Key value to look up the memory entry.
• newVal: The new value to be stored.
• options:
  • once: If true, the value won't be stored or updated if it exists.

get (scope, oldVal)

Retrieve a value from memory.

• scope: The context or scope the value is stored under.
• oldVal: Key value to look up the memory entry.

Examples

await mem.do('Person.Person', $val, () => $val + Math.random())
await mem.do('CUSTOMERS', $val, () => 'a new value')
await mem.set('person/name', $val, 'a new value')
const newVal = await mem.get('person/name', $val)

Help

• Use the mem package if you need to sincronize data across different columns/fields and entity tables.
• The mem package stores and retrieves data from project-wide dictionary, and sets values using a combination of a scope and a hashed or encrypted field value.
• We recommend the use of the mem.do method. It will only set a value if the value is not found, and return either the memoized value, if it exists, or the function return value. When a table is read in parallel and two readers compute a value for the same input at the same time, the first value saved wins and mem.do returns it to both.
• mem.get(scope, $val) - returns the memoized value, or undefined if not found.
• mem.set(scope, $val, newVal) - sets a new value into a scope
• The keys are computed like the rest of the project dictionary (HMAC-SHA256 with the dictionary secret, or MD5 in dictionaries created by previous versions), so the values stored with mem are shared with the dictionary of the rules.

once (cb)

Run a function once, caches and returns the result.

• cb: Function to be run

const myVar = once(() => 1000 * 1000)

randomDate (start?, end?)

Return a random date, between two given dates.

• start: Start date as number, string or Date object.
• end: End date as number, string or Date object.

randomDate('1980-01-01', '2010-12-31')

randomNumber (min, max, type, opts)

Return random number of the specified type between two values

• min: lower value the generated number can take
• max: higher value the generated number can take
• type: Can be integer or floating
• opts:
  • fixed: Specifies the number of decimals when the type is floating

randomNumber(0, 100, 'integer' || 'floating')
randomNumber(100)
randomNumber(100, 1_000_000)
randomNumber(100, 1_000_000, 'integer')
randomNumber(-100, 1_000_000, 'floating', { fixed: 2 })

sha (val, algo, inFmt, outFmt)

Return a hashed value from input

• val: Value to be hashed. Numbers are hashed by their text (1 as '1'); null is returned unchanged.
• algo: Algorithm to be applied. Defaults to SHA-512. It can take one of the following values: SHA-1, SHA-224, SHA3-224, SHA-256, SHA3-256, SHA-384, SHA3-384, SHA-512 or SHA3-512
• inFmt: Input format. Defaults to TEXT. It can take one of the following values: HEX, TEXT, B64, BYTES, ARRAYBUFFER, or UINT8ARRAY
• outFmt: Output format. Defaults to HEX. It can take one of the following values B64, HEX, BYTES, ARRAYBUFFER, or UINT8ARRAY

The hash depends only on the value: the same input always gives the same output, in any row, table or job. Earlier versions carried state from one row to the next, so the same value could give different hashes; values hashed with sha by those versions do not match the ones produced now.

sha($val, 'SHA-512', 'TEXT', 'HEX')
sha('my data')
sha('mydata', 'SHA3-256')
sha($val, 'SHA-512', 'TEXT', 'BYTES')
sha('bXlkYXRh', 'SHA3-256', 'B64', 'B64')

smartReplace (str, before, after)

Replace a string keeping the casing

• str: Source string.
• before: String to match.
• after: String to use as a replacement.

smartReplace('some string', 'search', 'replacement')
smartReplace('SmartPeople', 'people', 'phone')

strReverse (str)

Reverse a given string.

• str: String to reverse

strReverse($val)
strReverse('hello')

Examples

We can create a function to convert a temperature column from Farenheit to Celsius:

return (5 / 9) * ($val - 32)

Where $val represents the value of the field in the table.

Input -> Output:

We can also use JavaScript functions:

function toCelsius(fahrenheit) {
  return (5 / 9) * (fahrenheit - 32)
}

return toCelsius($val).toFixed(2)

Input -> Output: