Synthesize

The Synthesize operation allows you to generate new synthetic data records based on your existing dataset or using custom functions. This operation is particularly useful for creating larger datasets for testing purposes while maintaining realistic data characteristics.

In the operations grid it appears in Transform Operations as Synthesize ("Populate entities with fake data").

Overview

The Synthesize operation enables you to:

• Generate new data records that maintain realistic characteristics
• Create datasets of specific sizes based on your requirements
• Apply different synthesis techniques to different fields
• Use fake data generation, functions, saved functions or custom code
• Choose between appending synthesized data to existing data or replacing it entirely

Configuration

Auto synth

The Auto synth section sets the parameters for all entities.

Expected output data size

Select the size of your new synthesized dataset based on the source entity size:

• Same size as source entity (default): Generate exactly the same number of rows for each entity as exists in the source data.
• Proportional: Generate a different size using a multiplier of the source size, from 0 to 50 (for example 0.5 for half the rows, 1.5 for 150%, 10 for ten times as many). It can be set with the number box or with the slider (marks at 1%, 10%, 50%, 100%, 10x and 50x). You can also check Min rows: and Max rows: to keep each entity between a minimum and a maximum number of rows.

Behaviour

Set how the synthesized data interacts with your existing data:

• Append to source data (default): Keeps the existing rows and adds the synthesized rows to the end of each entity.
• Replace source data: Removes existing rows completely and inserts only the newly synthesized rows.

By entity

The By entity section lists the entities, with a search box and sorting by Name. For each entity you can see the Current records and the Expected output records, and:

• Process this entity: a switch to include or skip the entity.
• Customize: opens the fields of the entity to choose how each field is synthesized.

Field-Level Synthesis

Each field shows its Field, Type, Label, whether it Is PII, is Nullable or is a Primary key, and the Action used to generate it:

Columns that don't allow NULL get a default instead of an empty value: 0 for numbers, false for booleans, the current date for dates and an empty text for the rest.

Fields of type enum offer Enum List (random values of the enum), Set as null and Do not synthesize.

The pen icon (Edit options) opens the Edit function panel with the Field info (Name, Type, Label), the Configuration of the action and the Cancel / Save buttons. For custom functions the code icon (Edit code) opens the code editor.

Fake data options

For Use fake data the panel shows the options of the label first, with the option name as the field label (for example locale). For the person/name label the type option selects which part of the name is generated:

The type starts at the type detected for the field by the discovery. Use full (last, first) for columns that store names like Smith John.

Then come the common text options:

• Text format: None, UPPERCASE, lowercase, Title Case, Snake_case (spaces to underscores) or Kebab-case (spaces to hyphens).
• Prefix / Suffix: check the box and type a text to add before or after every generated value.

With the mixed label you add several labels, each with a percentage (Input percentage). The pen next to each label (Edit Label) opens the same options for that label, including type for person/name.

Number options

• Type: Sequential (default) or Random numbers.
• Start from: first value of the sequence (default 0).
• Limits: for random numbers, check the boxes to set a minimum and a maximum value.

Ensure Referential Integrity

The Ensure Referential Integrity checkbox at the bottom keeps the synthesized data consistent with the foreign keys:

• When only some entities are synthesized, their parent entities are added automatically.
• The entities are processed parents first, so child rows can reference the generated parent rows.

It is the same option as in Where, Limit and Read Query: if it is checked in any of these operations, it is on for the whole rule. It only applies to SQL taps: on other taps the option has no effect.

Examples

Basic Synthesis with Auto synth

To generate a dataset that is 150% the size of your source data:

1. Add the Synthesize operation to your rule
2. Select Proportional for the expected output data size
3. Set the multiplier to 1.5
4. Choose Replace source data
5. Keep Process this entity on for the entities to generate

Field-Level Synthesis Customization

To customize synthesis for specific fields in a customer entity:

1. Click Customize on the customer entity
2. For the "name" field, choose Use fake data with the person/name label, and set type to full (first, last)
3. For the "email" field, choose Use fake data with an email label
4. For the "id" field, choose Functions / Number with Sequential to generate unique IDs
5. For the "age" field, choose Functions / Number with Random numbers and limits 18 and 80
6. For the "phone" field, choose Custom function to keep the format

Using Custom Functions

To apply a custom synthesis algorithm:

1. Select a field and choose Custom function
2. Write JavaScript code that generates appropriate synthetic values
3. Use built-in helpers like chance(), faker, or genLike() for realistic data generation

Example:

// Generate realistic postal codes using the genLike function
return genLike('A1A 1A1') // Creates values like 'K9P 7N3' or 'M2J 4V8'

This operation provides powerful data generation capabilities that can help you create realistic test datasets while ensuring privacy compliance when combined with anonymization operations.