MongoDB Driver Documentation

This documentation explains the MongoDB driver implementation and how connection parameters map to UI fields in the Gigantic application.

Connection Parameters

Technical Details

The MongoDB driver implementation uses the mongodb Node.js library with MongoClient for connections.

• Document-based database using collections instead of tables
• Connection string format: mongodb://host:port or custom URL via config.url
• Schema extraction performed by sampling documents from collections
• Foreign key relationships detected by looking up ObjectId references across collections
• Uses document sampling approach for schema inference
• Supports connection options like useUnifiedTopology: true

When SSH tunneling is enabled, connections are first established through the tunnel before connecting to the MongoDB server.

Authentication Options

MongoDB supports standard username/password authentication and also connection string based authentication.

SSH Tunnel Support

The MongoDB driver supports SSH tunneling for secure access to databases behind firewalls or in private networks. For detailed information on configuring SSH tunnels, please refer to the SSH Tunnelling documentation.

API Endpoints Used

MongoDB connections are primarily used in:

• Tap creation (data source discovery)
• Sink creation (data destination for anonymized data)
• Pipeline execution (data extraction and loading)

Large Tables and Resume

This section lists only what is specific to MongoDB. How resume, parallel ranges and row estimates work in every driver is explained in Large databases.

Reading

• Collections are read in chunks ordered by _id, so a job that stops resumes after the last _id it saved. Each chunk is an aggregation run with allowDiskUse.
• A collection whose _id values are of several BSON types (for example ObjectIds and strings) is read with one cursor and starts over if the job stops. The job log says it has _ids of several types.
• A where whose first stage is a pipeline stage other than $match also reads the collection with one cursor, because that stage may change the documents or their _id.

Parallel Ranges and Row Estimates

• Ranges are cut from a $sample of the _id values. Collections with _id values of several types are not split.
• The row estimate is estimatedDocumentCount(), read from the collection metadata.

Loading

• Documents are written with ordered insertMany. The truncate option removes the documents with deleteMany; the drop option drops the collection.
• To resume a collection, the sink counts its documents with countDocuments. The MongoDB sink does not count the documents of one key range, so when a collection was read in ranges and the count does not tell which batches in flight were written, the collection starts over.

Custom Params

The MongoDB driver supports additional custom parameters that can be specified in YAML format to fine-tune connection behavior, pooling, SSL settings, and other advanced options.

Connection Parameters

# SSL/TLS Configuration
tls: true
tlsCAFile: /path/to/ca.pem
tlsCertificateKeyFile: /path/to/client-cert.pem
tlsCertificateKeyFilePassword: certificate_password
tlsAllowInvalidCertificates: false
tlsAllowInvalidHostnames: false
tlsInsecure: false

# Authentication Options
authMechanism: "SCRAM-SHA-256"  # or "SCRAM-SHA-1", "MONGODB-X509", "GSSAPI", "PLAIN"
authSource: "admin"
authMechanismProperties:
  SERVICE_NAME: "mongodb"
  CANONICALIZE_HOST_NAME: false

# Timeout Settings
connectTimeoutMS: 30000
socketTimeoutMS: 360000
serverSelectionTimeoutMS: 30000
heartbeatFrequencyMS: 10000
maxIdleTimeMS: 30000

# Read Preferences
readPreference: "primary"  # or "primaryPreferred", "secondary", "secondaryPreferred", "nearest"
maxStalenessSeconds: 90

# Compression
compressors: ["snappy", "zlib"]
zlibCompressionLevel: 6

Pool Configuration

# Pool sizing options
maxPoolSize: 100
minPoolSize: 10
maxConnecting: 2

# Pool timeout options
maxIdleTimeMS: 30000
waitQueueTimeoutMS: 0

Example Configurations

Basic Configuration

maxPoolSize: 50
serverSelectionTimeoutMS: 5000

SSL Configuration

tls: true
tlsCAFile: ${file:/path/to/ca-cert.pem}
tlsCertificateKeyFile: ${file:/path/to/client-cert.pem}
tlsCertificateKeyFilePassword: cert_password
authMechanism: "MONGODB-X509"

Replica Set Configuration

host: rs1.example.com,rs2.example.com,rs3.example.com
port: 27017
database: replicated_db
readPreference: "secondaryPreferred"
maxStalenessSeconds: 120
maxPoolSize: 25
minPoolSize: 5
compressors: ["snappy"]