Overview
SymmetricDS Pro 3.16.2 release includes 21 features, 55 improvements, and 40 bug fixes.
Security Fixes
| Issue | Summary | Severity |
|---|---|---|
Web console logins with OAuth 2.0 and OpenID Connect (Pro) |
Medium |
|
Prevent logging of secrets in symadmin and wrapper |
Medium |
|
Read-only users can edit Parameters via overrides (Pro) |
Medium |
|
Read-only users can save SAML configuration (Pro) |
Medium |
|
Add functionality to read the SAML IdP certificate from the IdP metadata URL (Pro) |
Medium |
|
Sign SAML requests with SHA-256 instead of SHA-1 (Pro) |
Medium |
|
Add a parameter to allow specification of OneLogin SAML properties via a properties file (Pro) |
Medium |
|
Add functionality to have the purge job automatically rotate the SAML SP x509 certificate when it’s near expiration (Pro) |
Medium |
|
A FileNotFoundException can occur when deploying multiple endpoints at once (Pro) |
Medium |
|
Disallow access to WEB-INF (Pro) |
Medium |
|
Upgrade chartjs and remove momentjs (Pro) |
Medium |
Performance Fixes
| Issue | Summary | Severity |
|---|---|---|
Bulk Extract for MSSQL (Pro) |
Medium |
|
Use source staging for incoming batch when engines are hosted together |
Medium |
|
Bulk loader for Sybase ASE (Pro) |
Medium |
|
Bulk export for SQL Server and Sybase ASE (Pro) |
Medium |
|
Postgresql binary mode for COPY statement for bulk loads (Pro) |
Medium |
|
Save column references as numeric values for faster look-up in AbstractDatabaseWriter.getRowData() |
Medium |
|
Ready queues for more efficient push/pulls |
Medium |
|
Dynamic reload queue to scale initial load (Pro) |
Medium |
|
Multi-threaded routing by channel enabled by default |
Medium |
|
During deployment, the Deploy Wizard should create engines in parallel (Pro) |
High |
|
On the Deploy Wizard’s Endpoint Summary screen, indicate how good each endpoint’s connection is (Pro) |
High |
What’s New
UI Improvements
The Configure tab now features a Canvas, replacing the Design tab with a new approach to configuring replication. Common replication scenarios are now simplified to setup using a visual canvas of connected nodes and the tables being replicated. Transforms are now created by drawing lines between tables and columns, even showing implied mappings as dashed lines. Changes are applied with a deployment wizard that collects final settings, creates engines, and prepares replication. The familiar original screens are still available with advanced settings.
Scaling Initial Load
Scaling the number of threads used for an initial load is now as easy as changing a parameter setting, one for extract threads (initial.load.extract.thread.per.server.count) and another for push/pull threads (initial.load.queue.sync.thread.count). The "reload" channel now defaults to a "reload" queue, which tells the system to use the parameter settings for the number of threads. All batches for a table are assigned to the same thread so there is no contention for the table. Users should request loads that either create tables or clear tables and defer constraints, so the system knows that foreign keys will not be present and it can use multiple threads.
Bulk Extract and Load Improvements
Bulk extract using native tools is now supported for SQL-Server and SAP (Sybase) ASE using the bcp bulk copy utility. Bulk loading using the bcp bulk copy utility is now supported on SAP ASE in addition to SQL-Server. The setup program and node wizard now perform automatic installation and configuration of native bulk loaders for SQL-Server and Oracle. The PostgreSQL bulk loader was improved to create unlogged tables and use binary mode copy, resulting in a 30% performance improvement.
File Sync with Azure Blob and S3
File sync now supports Azure Blob Storage and S3 buckets as either source or target of files. Paths to files that start with "azure://" or "s3://" will prompt the user for connection information to the cloud provider.
OAuth2 and OpenID Connect
Web console users can now authenticate with single sign-on (SSO) using the OAuth2 and OpenID Connect (OIDC) security standards with a third party identity provider. A role mapping feature allows the user’s role from OIDC to be mapped into a web console role.
Exasol Database Support
Migrate data and replicate changes to Exasol database. Exasol is an in-memory relational database built for high performance analytics. Even though it was designed to run in memory, it persists data to disk following ACID rules and follows SQL query standards. Support for Exasol makes it a compelling data migration target for building a data warehouse.
Channel Improvements
Internal operational SYM tables, such as node, node_security, and incoming/outgoing_error, now use a "system" channel so they act independently and are easier to see as batches for operations versus configuration changes (on the "config" channel). All SYM tables are using channels with a "system" queue so they sync on their own thread even when other channels have a backlog. The "reload" channel for initial and partial loads now defaults to the "bulk" data loader with an increased max batch size of 100,000.
Ready Queues
Ready queues is a feature to make efficient use of multiple queues and their associated threads. Instead of each node communication thread querying for ready data, the cache manager now keeps a list of queues by node that are ready with changes to send, reducing the number of queries needed. Push and pull service use the ready queues to efficiently allocate only the threads needed to send or receive data. Since pull service runs on the target, it still contacts each node on the default queue to get a list of ready queues.
Faster Local Data Transfer
When engines are hosted together (by placing the engine files in the same "engines" directory), a batch for a target engine will be accessed from the source engine’s staging area directly instead of copying it, saving the I/O cost of copying the data again. Remote nodes are unaffected and still receive batches over the network as usual.
Technology Upgrades
H2 database version 2.2 is now included. Users with existing H2 1.x databases may need to add ";MODE=LEGACY" to the end of db.url or "set MODE LEGACY" to db.init.sql (the Pro installer will do this automatically). Users with existing H2 2.0 or 2.1 databases need to export their database from the old version and import into the new one (the Pro installer will do this automatically). The server property of server.http.cookies.enabled is now true by default to accept cookies used by load balancers.
Issues
New Features
3.16.0 (Pro)
6234 - Web console logins with OAuth 2.0 and OpenID Connect
6543 - Add a Configure Canvas to replace the Design tab
6544 - Bulk loader for Sybase ASE
6650 - Add a way to add a prefix or suffix to transforms when doing auto-create
6688 - Dynamic reload queue to scale initial load
6714 - Package Microsoft’s bcp utility into setup program
6715 - Package Oracle’s SQL*Loader utility into a module
6736 - Documentation for OAuth 2.0 and OpenID Connect
6750 - Exasol Support as a Target Platform
3.16.1 (Pro)
6759 - Add support for timestamp with time zone data type to Postgres binary bulk loader
6765 - Add functionality to read the SAML IdP certificate from the IdP metadata URL
6770 - Add a parameter to allow specification of OneLogin SAML properties via a properties file
6776 - Add functionality to have the purge job automatically rotate the SAML SP x509 certificate when it’s near expiration
3.16.1
6777 - Add free disk space to runtime properties
3.16.2
6820 - Run compact script on shutdown H2
Improvements
3.16.0 (Pro)
2338 - Bulk Extract for MSSQL
6436 - Change how bulk loaders determine when to flush their data
6444 - Improve how bulk loader flag is set and how bulk loaded percentage is calculated
6466 - Adding File Sync Service for Azure Blob Storage
6549 - Process panel able to interrupt if thread is available
6551 - Bulk export for SQL Server and Sybase ASE
6585 - Postgresql binary mode for COPY statement for bulk loads
6696 - Max Batch Size insight shouldn’t check reload channels
3.16.0
6244 - Upgrade H2 library to 2.2
6429 - Upgrade dependencies to latest versions
6537 - Data gap detector should refresh cluster lock
6577 - Enable acceptance of cookies by default for compatibility with load balancers
6616 - Save column references as numeric values for faster look-up in AbstractDatabaseWriter.getRowData()
6625 - Update User Guide with Upgrade and Rollback recommendations
6691 - Skip querying of self-referencing foreign key child levels if constraints are deferred
6692 - Prevent logging of secrets in symadmin and wrapper
6695 - Increase the default max batch size for the reload channel
6734 - Multi-threaded routing by channel enabled by default
3.16.1 (Pro)
6767 - Sign SAML requests with SHA-256 instead of SHA-1
6782 - Add verbiage to the license key screen to explain how to get a license key
6785 - On the Define Endpoint screen, change the "Endpoint" field’s label to "Select New or Existing Endpoint"
6787 - During deployment, the Deploy Wizard should create engines in parallel
3.16.1
6678 - Support INCLUDE non-key columns for indexes in Postgres
3.16.2 (Pro)
6726 - Request a new Compare and Repair job remotely
6727 - Include primary key transforms in the Compare and Repair process
6790 - Allow sorting of more columns on Outgoing Loads screen
6801 - In the Transform Column Details dialog, the variable transform options should be sorted alphabetically
6806 - In the Configure Canvas' sidebar, update the styling of the progress stepper to clarify which step is the current step
6807 - In the Configure Canvas' transform tables dialog, draw dashed lines between the tables to indicate the default mapping
6808 - In the Configure Canvas' transform dialogs, make the column and table names on each side the same width
6824 - Batch fails with BadPaddingException and keeps retrying
6825 - Service and commands hang on Linux waiting for /dev/random
6828 - On the Deploy Wizard’s Endpoint Summary screen, display which deployed endpoint is the root registration server
6829 - Add tooltips to buttons to indicate the specific action that will be taken
6831 - Add a batch.outgoing.tosend.offline.count to runtime-stats in the Support Snapshot
6835 - Show cancelled load error message to user on load detail screen
6841 - Compare tables when columns are renamed on target
6843 - After configuring tables for the first time, the Load Data Wizard pops up to say that capture isn’t ready without any context
6844 - Resubmit of compare requests
6851 - Open active compare request automatically
6858 - When sending an outgoing load via the Load Data Wizard, have the Initial Load Queue job run immediately
6859 - Show future compare request and when they run
6860 - In the web console, replace "trigger" verbiage with "capture" verbiage
6863 - Log miner should check current SCN & if it’s less than the next SCN in sym_context, reset it to the current SCN & log an error
6864 - Compare and repair with before and after view
6868 - Compare should warn about unsupported routers
3.16.2
6705 - DataExtractorService.transferFromStaging should delete staged file if fails to decrypt staging file
6752 - Capture of DDL changes and delivery to the target should cause sync triggers at the target node after executing the SQL event
6793 - The symadmin import-config command should provide a better error message when it lacks the permissions to import a CSV file
6822 - Add sym_node_host_job_stats to snapshot
6826 - Add a batch.outgoing.tosend.offline.count to runtime-stats in the Support Snapshot
6850 - Extract error should be recorded on outgoing batch
6854 - Add console_table_stats to snapshot
6865 - Run sync-triggers on SQL events for Create/Alter DDL changes
6866 - Upgrade gradle to 8.14
Bug Fixes
3.16.0 (Pro)
6745 - RabbitMQ Errors when "data_loader_type" is bulk
3.16.1 (Pro)
5506 - Remove productionMode context parameter in web.xml
6748 - RabbitMQ should make "username" and "password" required fields
6749 - If user enters RabbitMQ username or password incorrectly RabbitMQ will fail
6754 - Read-only users can edit Parameters via overrides
6755 - Read-only users can save SAML configuration
6760 - Postgres binary bulk loader fails to parse times with 4+ digits of precision and timestamps with 4, 5, or 7+ digits of precision
6771 - RabbitMQ fails to connect when using the Connect Database wizard
6783 - Exasol icon doesn’t work in dark mode
6784 - The Deploy Wizard doesn’t display the Time-Based screen when adding new tables to an existing time-based endpoint
6791 - On the Manage Incoming/Outgoing Batches screens, filtering by a status of error can miss some batches in error
3.16.2 (Pro)
6802 - Some images in the SymmetricDS Pro 3.16 User Guide are broken
6803 - On the Canvas, the notification that opens after clicking "Details" before first deployment stays open after clicking elsewhere
6804 - In the Configure Canvas' transform columns dialog, the legend’s header text sometimes wraps to a second line
6805 - On the Configure Canvas, selecting a link that can’t be configured and then selecting one that can causes unexpected behavior
6810 - If no threads available, compare requests can cause multiple compare statuses to process in loop
6814 - Initial load fails with DB concurrent error
6815 - Detect transport issues during Compare & Repair and retry automatically
6817 - SQL Server change tracking can replicate data twice or in different order
6819 - Explore DB Compare should use target platform when using log mining
6823 - A FileNotFoundException can occur when deploying multiple endpoints at once
6840 - Can’t access license screen if engines fail to start
6842 - Dashboard customizations mess up size of dashboard
6849 - SymmetricDS Control Center does not come up on a Mac
6853 - Disallow access to WEB-INF
6857 - Upgrade chartjs and remove momentjs
3.16.2
6461 - Blob fields and Firebird 4 or 5
6779 - Postgresql DDL trigger fails when sym_trigger_hist is located in another schema
6827 - ExtractDataReader support from target platform
6830 - NPE when processing table renaming transform (without columns)
6832 - Unknown exceptions from registration not passed to listeners
6834 - Custom parameters and parameters with no default do not show up in parameters-changed in the snapshot
6836 - Fail startup if extensions XML doesn’t initialize Spring context
6847 - Postgres money data type error from initial load
6861 - Upgrade Spring, Swagger, Commons, and Vaadin libraries
Tables
The following changes were made to the definition of configuration and runtime tables. Table changes are applied to the database automatically using data definition language (DDL) during startup.
New Columns
| Column Name | Description |
|---|---|
|
Thread number within dynamic queue assigned for extraction |
|
Thread number within dynamic queue assigned for loading over push/pull |
|
The number of rows that are bulk loaded in an extract. |
| Column Name | Description |
|---|---|
|
Whether SQL/DDL events are routed and synced on this link. |
| Column Name | Description |
|---|---|
|
The timestamp when a partial load was started for this node. |
|
The timestamp when a partial load was completed for this node. |
|
A reference to the load_id in outgoing_batch for the last partial load that occurred. |
|
The user that created the partial load. A null value means that the system created the batch. |
| Column Name | Description |
|---|---|
|
Thread number within dynamic queue assigned for loading over push/pull |
| Column Name | Description |
|---|---|
|
The number of rows that were loaded with the bulk loader. |
Parameters
The following changes were made to add new parameters, modify their default value, modify their description, or remove them from use.
New Parameters
Enables the use of the Sybase ASE bulk copy program utility (bcp) for extracting initial load. See also parameters that will configure bcp out: ase.bulk.load.bcp.cmd, ase.bulk.load.field.terminator, and ase.bulk.load.row.terminator. (Default: false)
Specifies the path to the Sybase ASE bulk copy program utility (bcp) executable. (Default: )
Specify the field terminator used by the Sybase ASE bulk loader. Pick something that does not exist in the data in your database. (Default: |})
Specify the line terminator used by the Sybase ASE bulk loader. Pick something that does not exist in the data in your database. (Default: |>\n)
If this is true, when a delete affects zero rows, if the table is configured for sync on incoming, then the auto resolver will write the deletes into the capture log. (Default: false)
Enables the target to bulk load (bcp in) a data file that is located in the source staging (from bcp out), when both source and target are on the same server. (Default: true)
Maximum number of bytes to write to file before loading data via a bulk loader (Default: 1000000000)
This is the amount of time ready queue entries will be cached before re-reading them from the database. (Default: 5000)
The authorization endpoint is used by SymmetricDS in order to interact with your OAuth 2.0/OIDC provider and get the authorization to access the protected resource. (Default: )
This is the Client ID that your OAuth 2.0 provider gives you when you create your OAuth 2.0 Credentials. (Default: )
This is the Client secret that your OAuth 2.0 provider gives you when you create your OAuth 2.0 Credentials. (Default: )
URL of the OpenID Provider’s JWK Set document. This contains the signing key(s) SymmetricDS uses to validate signatures from the OpenID Provider. (Default: )
The issuer identifier of the OpenID Provider. (Default: )
Map of OAuth 2.0/OIDC roles to console role IDs. Allows OAuth 2.0/OIDC users to log in without having a corresponding console user if they are assigned one of the specified roles. (Default: )
The token endpoint is used by SymmetricDS in order to get an access token or a refresh token from your OAuth 2.0/OIDC provider. (Default: )
Metadata endpoint info of the SAML IdP. When set, SymmetricDS will obtain the public x509 certificate(s) from this URL. (Default: )
Path to a file containing properties for the OneLogin SAML Java Toolkit that SymmetricDS uses behind the scenes. The values in this file will override any default values built into SymmetricDS or the Toolkit, but not any values set by parameters. (Default: )
Number of days that the SAML SP certificate will last before it expires. Values below 1 are invalid and will be changed to 25 years. (Default: 9125)
Maximum number of days before the SAML SP certificate expires when the Purge Incoming job will replace it with a new certificate. (Default: 30)
If set to true, when a table’s schema is sent to the target database it will also send the DDL script that will create an application trigger at the target if the source and the target database types are the same. SymmetricDS trigger DDL statements will not be sent. (Default: false)
Enables removal of block comments from script text by data loader processing DDLs. Applies to stores procedures and functions, when DDL replication is enabled. (Default: true)
If source and target engine are hosted together, the source will send a "retry" command instead of sending the batch, and the target will use the source’s outgoing staging to access the batch. This also requires the staging to be enabled (stream.to.file.enabled=true). (Default: true)
If tables are created as part of the initial load, it will defer setting up table-level logging to improve performance. Applies to loads only. After data is loaded, the table-level logging will be switched on. This parameter needs set for the node that will send the initial load, not the node receiving it. Support for table-level logging varies by database dialect and is ignored where not applicable. Currently this is only supported by the PostgreSQL UNLOGGED feature. Oracle’s NOLOGGING feature might be implemented in the future. Note: this parameter is not compatible with PostgreSQL tables referenced by foreign keys. (Default: false)
The number of milliseconds that the initial load extract job can run a thread for processing extracting requests. (Default: 3600000)
The number of threads available for concurrent push/pulls of initial load batches using the reload queue. If the reload channel is placed on its own reload queue, multiple threads are created for push/pulls using dynamic queues named internally as reload!0, reload!1, reload!2, etc. Dynamic queues have the advantage of processing a table on the next available queue, preventing one queue from being full while other queues are empty. (Default: 20)
Whether to use all available threads for an initial load on the reload queue. Set to "true" to use all available threads, or "false" to use one thread. Set to "auto" and it will use all available threads only if the load is deferring constraints. (Default: auto)
Enables the use of the SQL Server bulk copy program utility (bcp) for extracting initial load. See also parameters that will configure bcp out: ase.bulk.load.bcp.cmd, ase.bulk.load.field.terminator, and ase.bulk.load.row.terminator. (Default: false)
Specifies the code page to pass to the bcp process using the -C command line option. This parameter only applies to the bcp process running on Windows. (Default: )
Enable this parameter to activate binary mode for PostgreSQL bulk loader utilizing the COPY IN statements. If a column type unsupported by the binary mode is encountered batch will be switches to the text mode. (Default: false)
Whether push and pull should use ready queues to limit queries and communication. Ready queues is the list of queues with outgoing batches that are ready to be sent, queried periodically and cached. When enabled, the pull service first pulls the default queue and receives a list of ready queues that should also be pulled, instead of pulling those queues every time. The push service gets the list of ready queues to allocate threads for calling data extractor service. (Default: true)
Modified Parameters
Specifies if server cert should be trusted when connecting bcp bulk loader to SQL Server. Applies to Linux and macOS only. (Old Default: false) (New Default: true)
Enables the use of the SQL Server bulk copy program utility (bcp) as the bulk loader. (Old Default: false) (New Default: true)
For a node operating in offline mode, specify the local directory where data files should be read from. The $(nodeGroupId) and $(externalId) variables are useful when running multiple engines in the same server. (Old Default: tmp/$(nodeGroupId)-$(nodeId)/offline/incoming) (New Default: tmp/$(nodeGroupId)-$(externalId)/offline/incoming)
For a node operating in offline mode, specify the local directory where data files should be written to. The $(nodeGroupId) and $(externalId) variables are useful when running multiple engines in the same server. (Old Default: tmp/$(nodeGroupId)-$(nodeId)/offline/outgoing) (New Default: tmp/$(nodeGroupId)-$(externalId)/offline/outgoing)
When sending an outgoing batch, copy directly from the outgoing staging to the incoming staging when both nodes are on the same server. This also requires the staging to be enabled (stream.to.file.enabled=true). The HTTP transport is still used to send a batch "retry" instruction that causes the target node to read from staging. Deprecated and replaced by incoming.batches.use.source.staging. (Old Default: true) (New Default: false)
When enabled, use a thread per channel for parallel routing. (Old Default: false) (New Default: true)
{REMOVED}
{REMOVED}
{REMOVED}
{REMOVED}
{REMOVED}