Overview

SymmetricDS Pro 3.16 release includes 12 features, 18 improvements, and 5 bug fixes.

Security Fixes

Issue Summary Severity

6234

Web console logins with OAuth 2.0 and OpenID Connect (Pro)

Medium

6692

Prevent logging of secrets in symadmin and wrapper

Medium

Performance Fixes

Issue Summary Severity

2338

Bulk Extract for MSSQL (Pro)

Medium

6538

Use source staging for incoming batch when engines are hosted together

Medium

6544

Bulk loader for Sybase ASE (Pro)

Medium

6551

Bulk export for SQL Server and Sybase ASE (Pro)

Medium

6585

Postgresql binary mode for COPY statement for bulk loads (Pro)

Medium

6616

Save column references as numeric values for faster look-up in AbstractDatabaseWriter.getRowData()

Medium

6687

Ready queues for more efficient push/pulls

Medium

6688

Dynamic reload queue to scale initial load (Pro)

Medium

6734

Multi-threaded routing by channel enabled by default

Medium

What’s New

UI Improvements

The Configure tab now features a Canvas, replacing the Design tab with a new approach to configuring replication. Common replication scenarios are now simplified to setup using a visual canvas of connected nodes and the tables being replicated. Transforms are now created by drawing lines between tables and columns, even showing implied mappings as dashed lines. Changes are applied with a deployment wizard that collects final settings, creates engines, and prepares replication. The familiar original screens are still available with advanced settings.

Scaling Initial Load

Scaling the number of threads used for an initial load is now as easy as changing a parameter setting, one for extract threads (initial.load.extract.thread.per.server.count) and another for push/pull threads (initial.load.queue.sync.thread.count). The "reload" channel now defaults to a "reload" queue, which tells the system to use the parameter settings for the number of threads. All batches for a table are assigned to the same thread so there is no contention for the table. Users should request loads that either create tables or clear tables and defer constraints, so the system knows that foreign keys will not be present and it can use multiple threads.

Bulk Extract and Load Improvements

Bulk extract using native tools is now supported for SQL-Server and SAP (Sybase) ASE using the bcp bulk copy utility. Bulk loading using the bcp bulk copy utility is now supported on SAP ASE in addition to SQL-Server. The setup program and node wizard now perform automatic installation and configuration of native bulk loaders for SQL-Server and Oracle. The PostgreSQL bulk loader was improved to create unlogged tables and use binary mode copy, resulting in a 30% performance improvement.

File Sync with Azure Blob and S3

File sync now supports Azure Blob Storage and S3 buckets as either source or target of files. Paths to files that start with "azure://" or "s3://" will prompt the user for connection information to the cloud provider.

OAuth2 and OpenID Connect

Web console users can now authenticate with single sign-on (SSO) using the OAuth2 and OpenID Connect (OIDC) security standards with a third party identity provider. A role mapping feature allows the user’s role from OIDC to be mapped into a web console role.

Exasol Database Support

Migrate data and replicate changes to Exasol database. Exasol is an in-memory relational database built for high performance analytics. Even though it was designed to run in memory, it persists data to disk following ACID rules and follows SQL query standards. Support for Exasol makes it a compelling data migration target for building a data warehouse.

Channel Improvements

Internal operational SYM tables, such as node, node_security, and incoming/outgoing_error, now use a "system" channel so they act independently and are easier to see as batches for operations versus configuration changes (on the "config" channel). All SYM tables are using channels with a "system" queue so they sync on their own thread even when other channels have a backlog. The "reload" channel for initial and partial loads now defaults to the "bulk" data loader with an increased max batch size of 100,000.

Ready Queues

Ready queues is a feature to make efficient use of multiple queues and their associated threads. Instead of each node communication thread querying for ready data, the cache manager now keeps a list of queues by node that are ready with changes to send, reducing the number of queries needed. Push and pull service use the ready queues to efficiently allocate only the threads needed to send or receive data. Since pull service runs on the target, it still contacts each node on the default queue to get a list of ready queues.

Faster Local Data Transfer

When engines are hosted together (by placing the engine files in the same "engines" directory), a batch for a target engine will be accessed from the source engine’s staging area directly instead of copying it, saving the I/O cost of copying the data again. Remote nodes are unaffected and still receive batches over the network as usual.

Technology Upgrades

H2 database version 2.2 is now included. Users with existing H2 1.x databases may need to add ";MODE=LEGACY" to the end of db.url or "set MODE LEGACY" to db.init.sql (the Pro installer will do this automatically). Users with existing H2 2.0 or 2.1 databases need to export their database from the old version and import into the new one (the Pro installer will do this automatically). The server property of server.http.cookies.enabled is now true by default to accept cookies used by load balancers.

Issues

New Features

3.16.0 (Pro)
6234 - Web console logins with OAuth 2.0 and OpenID Connect
6543 - Add a Configure Canvas to replace the Design tab
6544 - Bulk loader for Sybase ASE
6650 - Add a way to add a prefix or suffix to transforms when doing auto-create
6688 - Dynamic reload queue to scale initial load
6714 - Package Microsoft’s bcp utility into setup program
6715 - Package Oracle’s SQL*Loader utility into a module
6736 - Documentation for OAuth 2.0 and OpenID Connect
6750 - Exasol Support as a Target Platform

3.16.0
6538 - Use source staging for incoming batch when engines are hosted together
6687 - Ready queues for more efficient push/pulls
6703 - System channel and queue for responsive data sync

Improvements

3.16.0 (Pro)
2338 - Bulk Extract for MSSQL
6436 - Change how bulk loaders determine when to flush their data
6444 - Improve how bulk loader flag is set and how bulk loaded percentage is calculated
6466 - Adding File Sync Service for Azure Blob Storage
6549 - Process panel able to interrupt if thread is available
6551 - Bulk export for SQL Server and Sybase ASE
6585 - Postgresql binary mode for COPY statement for bulk loads
6696 - Max Batch Size insight shouldn’t check reload channels

3.16.0
6244 - Upgrade H2 library to 2.2
6429 - Upgrade dependencies to latest versions
6537 - Data gap detector should refresh cluster lock
6577 - Enable acceptance of cookies by default for compatibility with load balancers
6616 - Save column references as numeric values for faster look-up in AbstractDatabaseWriter.getRowData()
6625 - Update User Guide with Upgrade and Rollback recommendations
6691 - Skip querying of self-referencing foreign key child levels if constraints are deferred
6692 - Prevent logging of secrets in symadmin and wrapper
6695 - Increase the default max batch size for the reload channel
6734 - Multi-threaded routing by channel enabled by default

Bug Fixes

3.16.0 (Pro)
6745 - RabbitMQ Errors when "data_loader_type" is bulk

3.16.0
6351 - Text and icons disappear in SQL Explorer TreeGrid when switching nodes
6663 - Logging of exceptions on data load can show values used in the wrong order when conflict resolution is in play

3.16.1 (Pro)
5506 - Remove productionMode context parameter in web.xml
6749 - If user enters RabbitMQ username or password incorrectly RabbitMQ will fail

Tables

The following changes were made to the definition of configuration and runtime tables. Table changes are applied to the database automatically using data definition language (DDL) during startup.

New Columns

SYM_EXTRACT_REQUEST
Column Name Description

extract_thread_id

Thread number within dynamic queue assigned for extraction

load_thread_id

Thread number within dynamic queue assigned for loading over push/pull

bulk_rows_loaded

The number of rows that are bulk loaded in an extract.

SYM_NODE_GROUP_LINK
Column Name Description

sync_sql_enabled

Whether SQL/DDL events are routed and synced on this link.

SYM_NODE_SECURITY
Column Name Description

partial_load_time

The timestamp when a partial load was started for this node.

partial_load_end_time

The timestamp when a partial load was completed for this node.

partial_load_id

A reference to the load_id in outgoing_batch for the last partial load that occurred.

partial_load_create_by

The user that created the partial load. A null value means that the system created the batch.

SYM_OUTGOING_BATCH
Column Name Description

thread_id

Thread number within dynamic queue assigned for loading over push/pull

SYM_TABLE_RELOAD_STATUS
Column Name Description

row_bulk_load_count

The number of rows that were loaded with the bulk loader.

Parameters

The following changes were made to add new parameters, modify their default value, modify their description, or remove them from use.

New Parameters

ase.bulk.extract.use.bcp (Pro)

Enables the use of the Sybase ASE bulk copy program utility (bcp) for extracting initial load. See also parameters that will configure bcp out: ase.bulk.load.bcp.cmd, ase.bulk.load.field.terminator, and ase.bulk.load.row.terminator. (Default: false)

ase.bulk.load.bcp.cmd (Pro)

Specifies the path to the Sybase ASE bulk copy program utility (bcp) executable. (Default: )

ase.bulk.load.field.terminator (Pro)

Specify the field terminator used by the Sybase ASE bulk loader. Pick something that does not exist in the data in your database. (Default: |})

ase.bulk.load.row.terminator (Pro)

Specify the line terminator used by the Sybase ASE bulk loader. Pick something that does not exist in the data in your database. (Default: |>\n)

auto.resolve.capture.delete.missing.rows

If this is true, when a delete affects zero rows, if the table is configured for sync on incoming, then the auto resolver will write the deletes into the capture log. (Default: false)

bcp.bulk.load.use.source.staging (Pro)

Enables the target to bulk load (bcp in) a data file that is located in the source staging (from bcp out), when both source and target are on the same server. (Default: true)

bulk.load.max.bytes.before.flush (Pro)

Maximum number of bytes to write to file before loading data via a bulk loader (Default: 1000000000)

cache.ready.queue.time.ms

This is the amount of time ready queue entries will be cached before re-reading them from the database. (Default: 5000)

console.auth.oauth2.authorization.endpoint (Pro)

The authorization endpoint is used by SymmetricDS in order to interact with your OAuth 2.0/OIDC provider and get the authorization to access the protected resource. (Default: )

console.auth.oauth2.client.id (Pro)

This is the Client ID that your OAuth 2.0 provider gives you when you create your OAuth 2.0 Credentials. (Default: )

console.auth.oauth2.client.secret (Pro)

This is the Client secret that your OAuth 2.0 provider gives you when you create your OAuth 2.0 Credentials. (Default: )

console.auth.oauth2.discovery.jwks.uri (Pro)

URL of the OpenID Provider’s JWK Set document. This contains the signing key(s) SymmetricDS uses to validate signatures from the OpenID Provider. (Default: )

console.auth.oauth2.issuer (Pro)

The issuer identifier of the OpenID Provider. (Default: )

console.auth.oauth2.role.map (Pro)

Map of OAuth 2.0/OIDC roles to console role IDs. Allows OAuth 2.0/OIDC users to log in without having a corresponding console user if they are assigned one of the specified roles. (Default: )

console.auth.oauth2.token.endpoint (Pro)

The token endpoint is used by SymmetricDS in order to get an access token or a refresh token from your OAuth 2.0/OIDC provider. (Default: )

console.auth.saml.idp.metadata.url (Pro)

Metadata endpoint info of the SAML IdP. When set, SymmetricDS will obtain the public x509 certificate(s) from this URL. (Default: )

console.auth.saml.properties.file (Pro)

Path to a file containing properties for the OneLogin SAML Java Toolkit that SymmetricDS uses behind the scenes. The values in this file will override any default values built into SymmetricDS or the Toolkit, but not any values set by parameters. (Default: )

console.auth.saml.sp.cert.lifetime.days (Pro)

Number of days that the SAML SP certificate will last before it expires. Values below 1 are invalid and will be changed to 25 years. (Default: 9125)

console.auth.saml.sp.cert.rotate.days (Pro)

Maximum number of days before the SAML SP certificate expires when the Purge Incoming job will replace it with a new certificate. (Default: 30)

create.table.include.application.triggers

If set to true, when a table’s schema is sent to the target database it will also send the DDL script that will create an application trigger at the target if the source and the target database types are the same. SymmetricDS trigger DDL statements will not be sent. (Default: false)

dataloader.sql.event.strip.comments

Enables removal of block comments from script text by data loader processing DDLs. Applies to stores procedures and functions, when DDL replication is enabled. (Default: true)

incoming.batches.use.source.staging

If source and target engine are hosted together, the source will send a "retry" command instead of sending the batch, and the target will use the source’s outgoing staging to access the batch. This also requires the staging to be enabled (stream.to.file.enabled=true). (Default: true)

initial.load.defer.table.logging

If tables are created as part of the initial load, it will defer setting up table-level logging to improve performance. Applies to loads only. After data is loaded, the table-level logging will be switched on. This parameter needs set for the node that will send the initial load, not the node receiving it. Support for table-level logging varies by database dialect and is ignored where not applicable. Currently this is only supported by the PostgreSQL UNLOGGED feature. Oracle’s NOLOGGING feature might be implemented in the future. Note: this parameter is not compatible with PostgreSQL tables referenced by foreign keys. (Default: false)

initial.load.extract.max.process.time.ms

The number of milliseconds that the initial load extract job can run a thread for processing extracting requests. (Default: 3600000)

initial.load.queue.sync.thread.count (Pro)

The number of threads available for concurrent push/pulls of initial load batches using the reload queue. If the reload channel is placed on its own reload queue, multiple threads are created for push/pulls using dynamic queues named internally as reload!0, reload!1, reload!2, etc. Dynamic queues have the advantage of processing a table on the next available queue, preventing one queue from being full while other queues are empty. (Default: 20)

initial.load.queue.use.all.threads (Pro)

Whether to use all available threads for an initial load on the reload queue. Set to "true" to use all available threads, or "false" to use one thread. Set to "auto" and it will use all available threads only if the load is deferring constraints. (Default: auto)

mssql.bulk.extract.use.bcp (Pro)

Enables the use of the SQL Server bulk copy program utility (bcp) for extracting initial load. See also parameters that will configure bcp out: ase.bulk.load.bcp.cmd, ase.bulk.load.field.terminator, and ase.bulk.load.row.terminator. (Default: false)

mssql.bulk.load.bcp.code.page (Pro)

Specifies the code page to pass to the bcp process using the -C command line option. This parameter only applies to the bcp process running on Windows. (Default: )

postgres.bulk.copy.binary (Pro)

Enable this parameter to activate binary mode for PostgreSQL bulk loader utilizing the COPY IN statements. If a column type unsupported by the binary mode is encountered batch will be switches to the text mode. (Default: false)

sync.use.ready.queues

Whether push and pull should use ready queues to limit queries and communication. Ready queues is the list of queues with outgoing batches that are ready to be sent, queried periodically and cached. When enabled, the pull service first pulls the default queue and receives a list of ready queues that should also be pulled, instead of pulling those queues every time. The push service gets the list of ready queues to allocate threads for calling data extractor service. (Default: true)

Modified Parameters

mssql.bulk.load.bcp.trust.server.cert (Pro)

Specifies if server cert should be trusted when connecting bcp bulk loader to SQL Server. Applies to Linux and macOS only. (Old Default: false) (New Default: true)

mssql.bulk.load.use.bcp (Pro)

Enables the use of the SQL Server bulk copy program utility (bcp) as the bulk loader. (Old Default: false) (New Default: true)

outgoing.batches.copy.to.incoming.staging

When sending an outgoing batch, copy directly from the outgoing staging to the incoming staging when both nodes are on the same server. This also requires the staging to be enabled (stream.to.file.enabled=true). The HTTP transport is still used to send a batch "retry" instruction that causes the target node to read from staging. Deprecated and replaced by incoming.batches.use.source.staging. (Old Default: true) (New Default: false)

routing.use.channel.threads

When enabled, use a thread per channel for parallel routing. (Old Default: false) (New Default: true)

cloud.bulk.load.max.rows.before.flush (Pro)

{REMOVED}

mysql.bulk.load.max.rows.before.flush (Pro)

{REMOVED}

mysql.bulk.load.max.bytes.before.flush (Pro)

{REMOVED}

cloud.bulk.load.max.bytes.before.flush (Pro)

{REMOVED}

mssql.bulk.load.max.rows.before.flush (Pro)

{REMOVED}