PII Discovery
PII (Personally Identifiable Information) Discovery is the first step in identifying sensitive data in your databases. Gigantics provides multiple options to customize how you discover PII in your data.
PII Discovery Options
Select Entities
You can choose which database entities (tables, collections, etc.) to include in the discovery process:
- Include specific entities: Select only certain tables or collections for analysis
- Exclude specific entities: Scan all entities except those you specify
- Scan all entities: Analyze every table and collection in your database
This selection is important because:
- It allows you to focus discovery efforts on databases with sensitive data
- Reduces processing time by excluding system tables or non-sensitive data
- Enables compliance scanning of only regulated data sets
Regular Expressions
Use regular expressions to define patterns for entities you want to discover:
- Pattern matching: Automatically include entities that match specified patterns
- Flexible filtering: Create complex rules to select tables based on naming conventions
- Bulk selection: Efficiently select multiple entities without manual checkboxes
Examples of useful regex patterns:
user.*- Match all tables starting with "user".*_pii$- Match all tables ending with "_pii"[a-z]+_[0-9]+- Match tables with letterunderscorenumber patterns
Discovery Scope Configuration
You can configure the scope of your discovery with these parameters:
Scan Depth
- Sample size: Define how many rows to scan for accurate detection
- Column coverage: Specify whether to analyze all columns or focus on specific types
Processing Options
- Rate limit: Control the speed of discovery to avoid impacting database performance
- Concurrency: Set how many entities to scan in parallel
- Schedule: Run discovery immediately or schedule for off-peak hours
Running PII Discovery
To start a PII Discovery job:
- Navigate to the Model section
- Select "Discovery" from the sidebar
- Click "New Discovery Job"
- Choose "PII Discovery" option
- Configure your entity selection method
- Set processing parameters
- Review and start the job
Discovering a Single Entity
To rediscover only one entity without launching a discovery job, open the entity in the discovery's Edit labels view and click "Discover entity". Discovery runs right away on that entity, and its labels are updated in the latest discovery without creating a new discovery version.
The dialog has these options:
- Merge with previous discovery: keeps the entity's current labels and adds the new ones. If you uncheck it, Gigantics asks for confirmation, because the entity's previous labels are reset, both the auto-detected ones and the ones edited by users.
- Row limit: number of rows to analyze, up to 5,000, taken from the First records, Last records or Random records of the entity. To analyze more rows, run a discovery job.
- Labels probability: labels are assigned automatically when their probability is higher than this value.
The button is only shown to users with the "Run discover job" permission. It is disabled while a discovery job is running and on the partition tables of an entity group.
Following the Discovery Job
While the job runs, the Discovery page shows a progress banner at the top:
+---------------------------------------------------------------+
| (42%) Job running |
| Job discover-1234 is running |
+---------------------------------------------------------------+| Status of the job | Banner title | Banner text |
|---|---|---|
| Running | Job running | Job <name> is running |
| Scheduled | Job scheduled | Job <name> |
| Queued | Job queued | Job <name> |
| Completed | Job success | Job <name> finished successfully |
| Failed | Job error | Job <name> finished with errors |
The job name is a link to the job detail. When the job is no longer running, the banner shows a Reload the page button to load the new results.
How Progress Is Counted
- While Gigantics reads the schema, progress counts the tables (entities) read out of the total found in the data source.
- The progress stays below 100% until the read is really finished, so the bar never shows 100% too early.
- At the end, the totals are adjusted to the entities that were actually scanned, after your include/exclude selection.
- If the data source returns an error, the discovery job stops at the first error and is marked as failed. The progress is not set to 100%.
Job Summary
The job detail page (the link in the banner) shows these rows in the Summary section:
| Row | Finished job | Running job |
|---|---|---|
| Start | Date and time when the job started (YYYY-MM-DD HH:mm:ss) | Same |
| End | Time when the job finished (HH:mm:ss) | Estimated end time and time left, for example 14:35 (12m:4s remaining) |
| Elapsed | Total duration, for example 1h:5m | Time so far and estimated total, for example 3m:10s (15m:14s total) |
| Transfer rate | Records per second, for example 1,250 rec / s | Same, updated while the job runs |
While the job runs, End and Elapsed have a help icon with the tooltips "Estimated end time based on current progress" and "Estimated total time based on current progress".
The estimate is only shown when the job has passed 1% of progress and its estimated total time is over 30 seconds. Before that, End shows - and Elapsed shows only the time so far.
After the discovery completes, you'll be able to Review Labels and adjust classifications as needed.