Overview
SymmetricDS Pro 3.12 release includes 47 features, 213 improvements, and 382 bug fixes.
Security Fixes
| Issue | Summary | Severity |
|---|---|---|
Provide more granular level of permissions (Pro) |
Medium |
|
authentication before using REST api |
Medium |
|
Client certificate authentication |
Medium |
|
Two-Factor Authentication (Pro) |
Medium |
|
Use HTTP/2 for encrypted HTTPS synchronization |
Medium |
|
Record node login failures and prevent too many logins |
Medium |
|
Encrypt node passwords (Pro) |
Medium |
|
Password Complexity Meter (Pro) |
Medium |
|
Forgot Password and Email Verification (Pro) |
Medium |
|
Failure to reset the Failed Login Attempts on a successful login using Two-Factor Authentication (Pro) |
Medium |
|
Improve security for changing user password (Pro) |
Medium |
|
Authentication for JMX and default to localhost only |
Medium |
|
Web server require or accept client ssl certificate authentication |
Medium |
|
Manage → Security screen import/export certificate authority (Pro) |
Medium |
|
REST API Security keys (Pro) |
Medium |
|
First time setup admin user encryption strength (Pro) |
Medium |
|
TransportManagerFactory needs to call SelfSignedX509TrustManager constructor with actual trust store used instead of null |
Medium |
|
Console user authentication with client certificate (X.509) (Pro) |
Medium |
|
Upgrade Jetty, BC, Kotlin, okhttp libraries |
Medium |
|
Prevent expired certificates from being imported (Pro) |
Medium |
|
Upgrade log4j2 again again |
Medium |
|
CVE-2022-22965: Spring Version needs to use 5.2.20.RELEASE or 5.3.18 in order to not be vulnerable to this security issue |
Medium |
|
Apache commons-text version 1.9 security vulnerability |
Medium |
What’s New
Design Screen
The web console has a new "Design" tab to visually build configuration from a higher level. The "Configure" tab is still available for editing configuration options.
Log Mining for SQL-Server
Log mining is now available for SQL-Server in additional to trigger-based change capture. It is based on change tracking provided by SQL Server 2008 and newer.
Conflicts by Time of Capture
The new default conflict detection is USE_CHANGED_DATA with a resolver of NEWER_WINS. The NEWER_WINS resolution was enhanced to automatically use the source capture time of the change. (Previously, the user had to setup a detection with USE_TIMESTAMP and specify a timestamp column from the table.) When the target node is version 3.12 or newer, the source node will send the capture time in the CSV protocol, which is used when resolving conflicts. The default for dataloader.apply.changes.only parameter was changed to true, which will help avoid conflicts of a row when different columns are updated.
Modules
A modules system was added to reduce the size of the setup program. There are modules for JDBC drivers, streaming platforms, and the Swagger UI that will download automatically as needed. The most popular drivers (Oracle, SQL-Server, PostgreSQL, MySQL, and H2) are still included in the setup program, but they may become modules in the future. For command line support, see the "symadmin help module" command.
HTTP/2
Jetty was upgraded to support HTTP/2 with ALPN for TLS 1.2 encryption. When configuring server options during installation, HTTP/2 is an enabled option by default. When upgrading, edit the symmetric-server.properties file to enable the https2.enable property. The new protocol is faster, more efficient, and more secure. To continue supporting Java 8 and Android platforms, the okhttp3 library is used for client connections that support HTTP/2.
Node Password Lockout
When a node fails to authenticate too many times in a row, it will be locked out as a security feature. A new field named failed_logins was added to the sym_node_security table, which is incremented for each failed login. A successful login resets the counter back to 0. When failed_login reaches the limit for the node.password.failed.attempts parameter, which defaults to 5, then the log changes from "it had the wrong password" to "it had too many login attempts". Set the failed_logins back to 0 to unlock the node.
Index on SYM_DATA
When upgrading, you’ll see an index on sym_data is dropped and recreated for the columns data_id and channel_id. The index was changed from non-unique to unique, which will help query execution plans associated with routing and extraction of batches to run more efficiently.
Ingres Database
The Ingres database is now supported for change capture, as well as load only and extract only.
Oracle Character Sets
The Oracle triggers were enhanced to handle multiple character sets in the same database. To enable the feature, set parameter oracle.use.ntypes.for.sync=true in the engine file. The triggers will use nchar and nclob types, and the sym_data table is altered to use nclob. As a nice side effect for all Oracle users, the triggers should be slightly more efficient because they now use nvl2() instead of decode() function.
Logging System
The logging was upgraded to use log4j2 logging system. There is now a log4j2.xml file to configure instead of log4j.xml file. The logging format and rotation of log files should continue to work the same way.
Issues
New Features
3.12.1
4415 - Ingres Dialect Implementation
3.12.2
2861 - Client certificate authentication
3.12.3 (Pro)
4500 - REST API Security keys
3.12.4 (Pro)
4520 - New Screen Called Startup Parameters for Editing Engine Files
3.12.5 (Pro)
4564 - Add option to allow changes from SQL to be captured
4569 - Add a file to snapshots that contains thread CPU and memory usage information
4603 - Indicate when nodes or users are locked out and add buttons to unlock them
4604 - Add file containing information about blocking processes to snapshots
4615 - Add a notification for when the license is about to expire
3.12.5
4579 - SQL Server supports data compression on tables and indexes, add support for this in DDLUtils
3.12.7 (Pro)
4759 - Add option to require a user to change their password on their next login
4761 - Add console.password.min.change.chars parameter
4766 - Add shortcut to skip admin user configuration
4799 - Console user authentication with client certificate (X.509)
4805 - MSSQL Log based loop prevention and sync on incoming support
4837 - Support for a new variable in trigger source table name to expand upon based on configured nodes
4846 - Add profiles to Data Load Wizard
3.12.9
1289 - Ignore row in batch from source system
3.12.11
5021 - DBFill support for load only nodes
3.12.12 (Pro)
5058 - Ability to store table stats by DML
3.12.12
5057 - Record table stats by DML type
3.12.23
5820 - System property to control use of "on conflict" on Postgres
Improvements
3.12.0 (Pro)
1432 - When a user cannot edit a panel, instead of hiding edit buttons, disable the buttons.
1663 - Windows authentication with SQL-Server
1692 - If you can’t edit configuration, at least let you view it read-only
2349 - Provide more granular level of permissions
4301 - Simplify conflicts screen to encourage valid settings
4361 - Encrypt node passwords
4388 - User Email Capture
3.12.0
2579 - authentication before using REST api
4167 - sym_router table should require router_type value, providing a default value of 'default'
4192 - Non-unique index on sym_data should be unique
4279 - Remove JAR files that are not used as often to download separately
4302 - Conflict resolution NEWER_TIME based on capture time of row
4303 - Change default of dataloader.apply.changes.only to true to support better conflict resolution
4322 - Upgrade to swagger 2, optional swagger-ui as module
4324 - Upgrade to log4j2
4351 - Change registration to send request parameters as POST instead of GET
4352 - Sybase dialect should enable row locking and set identity gap
4360 - Record node login failures and prevent too many logins
3.12.3
4485 - Log file rotation is reversed
3.12.4 (Pro)
4532 - Manage Startup Parameters screen needs its own privilege
4533 - Batch data should be easily downloadable
4534 - Update Support and Maintenance screens
4535 - Improve Manage JVM Threads screen
4539 - Show query tab by default under the Explore tab
4540 - Improve Manage Console Events Screen
4541 - Fix all warnings on all Manage screens and classes
4562 - Manage Staging Area screen should hide folders when their contents don’t match the filter value
4566 - Allow conflict edit panel to set ping back
3.12.5 (Pro)
4570 - Fix all warnings on AbstractEditPanel and the classes that extend it
4575 - Upgrade jQuery library to 3.5.1
4587 - Improve multi-select grid behavior
4588 - Improve behavior of "Active Only" checkbox on Manage JVM Threads screen
4589 - Update deprecated components on Dashboard tab
4590 - Update deprecated components on Design tab
4592 - Update deprecated components on Explore tab
4606 - Widen Process Type combo box on Manage Processes screen
4608 - Improve Configure Transforms screen
4618 - Add "View Text" button to Manage Installed Triggers screen
4621 - Registration secret for security
4623 - Update deprecated components in load, quick start and setup wizards
4624 - Update deprecated components in login dialog
4626 - Fix and improve Quick Config Wizard
4629 - Improve node group selection in the node setup wizard
3.12.5
4568 - SQL Server supports filtered indexes, need to add support of filtered indexes in ddlutils
4572 - how to stop one node’s service
4593 - Symadmin send-schema should allow specification to not send indices, foreign keys, and column default values
4596 - Upgrade dependent libraries
4614 - SQL Server trigger creation should use CREATE OR ALTER TRIGGER syntax to eliminate the loss of data capture
4625 - Multi-primary out of sync with initial load on busy system
3.12.6 (Pro)
4632 - Update deprecated classes in Outgoing Batches and Incoming Batches screens
4639 - Update deprecated classes in common UI classes
4640 - Update deprecated classes in MonitorSummaryDialog
4641 - Improve Configure Monitors and Configure Notifications screens
4654 - Add filter field to Manage JVM Properties screen
4655 - Add "Show sym_ Batches" checkbox to Manage Incoming/Outgoing Batches screens
4656 - Show the staging directory location on the Manage Staging Area screen
4659 - Allow query results to be edited
4663 - Add encryption to parameter screens
4678 - Add Edit button to Manage Staging Area screen
4683 - Update docker image to JDK slim
4690 - Make the data load wizard resizable
4691 - Alphabetize channel select fields
4692 - Add validate button to BeanShell script editors
4693 - Add confirmation dialog when canceling out of a form
4694 - Improve channel filtering on batch screens
4695 - Add grid paging to batch screens and Manage Nodes screen
4712 - Logging panel push pull requests
4716 - Improve Configure Overview screen
4718 - Improve Help Support screen
4721 - Improve Configure Extensions screen
4728 - Add confirmation dialog when disabling a TriggerRouter
4729 - Add lazy loading to Manage Staging Area screen
4731 - Add lazy loading to SQL Explorer tree
4733 - Add delay between login prompts after a failed login attempt
4734 - Set cookies as http only and secure
4735 - Improve batch details window
4739 - Add more specific privileges for Manage Staging Area screen
4740 - Add filters and export button to Manage Console Events screen
4744 - Upgrade Vaadin to 8.12
4745 - Fix and improve Manage Nodes screen
3.12.6
4661 - Initial load foreign key error auto resolve when foreign key missing at source
4668 - Extensions global BSH functions
4673 - Quick test before upgrade of sym_data_event
4674 - Notification of monitor event resolution
4708 - Data loader lookup of meta data on table does not log appropriate error message
4715 - Oracle fails to check compatible v$parameter
4737 - Parameter for create or alter syntax
4741 - DB2 zOS clob columns attempt to drop auto generated rowid on restart
4743 - Conflict resolver improvements, unique key detection
4751 - MSSQL ddl table creation support for varbinary(max)
4752 - DB2 support for XML data types
3.12.7 (Pro)
4748 - Add more console events
4753 - Add server-side lazy loading to SQL Explorer tree
4754 - Add lazy loading to Manage Console Events screen
4755 - Firebird blocking transaction in support snapshot
4756 - Prevent table & file routing from being deleted via auto create wizard
4757 - Open node setup wizard when database icon is clicked
4758 - Parameter console.ip.addr.filter should apply to welcome wizard
4772 - For upgrades, default https2 disabled
4773 - Require the interface name field when creating or editing a BeanShell extension
4774 - Add failed logins column to grid on Configure Users screen
4775 - Add validate button when entering BeanShell scripts to send to other nodes
4776 - Lazily load the full list of nodes into the data load wizard’s grid
4777 - Improve BeanShell editors
4778 - Allow disabling of node_password encryption
4781 - Improve cron schedule editor
4785 - Load user guide by section to conserve memory
4788 - Improve editor cancellation
4789 - Adjust default sync URL
4792 - Improve update statements in SQL explorer
4801 - Update wording for the 3 setup options on welcome wizard
4807 - Monitor summary clearing all errors and delay
4808 - Add configure links to monitor summary dialog
4809 - Enhance Send Table Schema window
4813 - Improve multi-select grid behavior
4815 - Set node.password.encrypt.enable=false when a new node is added with cluster locks enabled
4822 - Improve background refresher registration
4827 - Improve Manage Installed Triggers screen
4829 - Improve appearance of validation error messages
4832 - Replace invalid characters in group IDs
4834 - Improve appearance of parameterized messages in monitor summary dialog
4836 - Allow spaces in external id
4844 - Alphabetize dropdowns in quick config wizard
4845 - Fix default grid sort order on Manage Outgoing Loads screen
4848 - Improve statistics on incoming batch details window
4850 - Improve error resolution on incoming batch details window
3.12.7
4765 - Prevent blocking on Firebird with lock timeout
4780 - JMX batch and data count attributes
4782 - Reduce dependency on log4j2 for deployments without it
4819 - Improve WAR deployment
4821 - Add unit tests for massageForLimitOffset()
4828 - Log an error instead of a warning when a trigger’s source table doesn’t exist
4835 - Allow spaces in external id
3.12.8 (Pro)
4841 - Improve batch screen grid performance
4853 - Improve grids on batch details window
4855 - Enable the Ignore Batch option for any batch that is not ok or ignored
4856 - Improve columns of batch screen grids
4864 - Update design view to display log based and load only databases along with proper database icon
4868 - Add the target.* properties to the symmetric-default.properties file so that they show up on the Manage Startup Parameters
4869 - Get rid of deprecated usages in gradle build in preparation for upgrade of gradle
4870 - Improve grid on Help Support screen
4875 - Remove "Save Failed" pop-up from forms
4878 - Improve appearance of schedule field in job form
4883 - Improve batch screen grid filter
4885 - Hide Resolve Data tab when it’s empty
4893 - Add filters to the grid on the send table schema window
4894 - Oracle Log Miner: should show SQL statement in log when parsing fails
4898 - Move remaining calls to syncTrigger() to background thread
4904 - Improve data load wizard’s criteria screen
4907 - Add router selection to quick config wizard
4908 - Add option to reset row replication statistics
4910 - Oracle auto create trigger options for LONG and ROWID
4911 - Temporarily default to HTTPS 1.1 instead of 2 for stability
4921 - Improve node terminal window
4922 - Prevent node installation dialog from being dismissed on first-time setup
4923 - Improve appearance of violation panel
4924 - Add a user-specific setting for showing advanced form fields
4926 - Add insert button to transform column editor
3.12.9 (Pro)
4872 - Design screen will display transforms that are not linked up properly
4935 - Skip a specific source row in batch thru the front-end
4937 - Save grid column visibility when switching screens
4938 - Open quick config wizard when table or file sync icon is clicked
4939 - Stop resetting the position of the vertical split in the Design tab
3.12.11
5000 - Turn on Jetty access logging
5023 - Sync Triggers should execute successfully in parallel by table
5025 - Informix dialect with page size 2k
5035 - Big Query support for bulk loading data with commas
5043 - Trigger router cache getting flushed every time initial load extract job runs
5049 - Use session_context with JTDS driver
5051 - Improve performance with sync trigger call for list of triggers
3.12.12 (Pro)
4994 - Performance of sync trigger is very slow in SQL Server while synchronizing large number of triggers
3.12.16
2450 - Preserve datetime precision, support for nanoseconds
3.12.20 (Pro)
5607 - Replace deprecated cacls command with icacls command
3.12.20
4168 - SQL-Server capture rows that exceed 4000 characters
3.12.21 (Pro)
5632 - Ensure that SQL Server log miner always captures deletes in foreign key order
3.12.21
5628 - When deleting FK child rows during conflict resolution and old_data is null, query for the parent row on the target
Bug Fixes
3.12.0
1866 - PostgreSQL, postgresql_bulk handle money type
2058 - Similar long table names fail to create trigger
2278 - File sync during initial load doesn’t work
2292 - Cannot replicate the table with foreign key to non-replicated table
2439 - PostgreSQL 9.5 Beta 1 no longer supports autocommit option
3919 - Symmetric does not locate engines.properties file upon application reload in WebLogic container
3972 - Tables become out of sync due incorrect conflict detection (and/or invalid conflict resolution)
4226 - Table trigger configuration - Sync on Insert/Delete/Update Condition, wrong Sync Condition example
4229 - [Docs] Invalid column name in "Example 2. Sample Group Links"
4263 - mx4j without auth
4331 - symmetrcDS war deployment fails on tomcat
4401 - Oracle ntype characters lost from conversion in capture
4402 - Default the file.sync.fast.scan back to false since it only works reliably on Unix
3.12.1 (Pro)
4411 - NoClassDefFoundError during node setup for Kafka, Azure, Snowflake
4422 - Node registration gets Bad Padding Exception
4425 - Add load only node does not continue when node is not the master node
4431 - Failure to reset the Failed Login Attempts on a successful login using Two-Factor Authentication
4432 - Diagram table foreign key name is too long for Oracle
4438 - Table count should be case insensitive for license check
4440 - Improve security for changing user password
4447 - MSSQL Log based replication fails on sync triggers
3.12.1
4407 - Monitor Email Notifications Stopped Working
4413 - MSSQL support for binary types in the primary key
4419 - HTTP2 Connection response body not closed
4420 - Error "hostnameVerifier parameter specified as non-null is null"
4421 - Primary key ordering based on the sequence of the PK
4427 - Ingres Dialect: Foreign Key DDL Reader issue and System Generated Indexes
4435 - Kafka key.serializer issue
4436 - Unable to handle unknown csv values: ts
4441 - Kafka module missing during setup of Kafka node.
4442 - Invalid lines in batch with HTTP2
4449 - Authentication for JMX and default to localhost only
4452 - Trigger Rebuild: custom_before_insert_text is ignored when rebuilding triggers
3.12.2 (Pro)
4478 - Manage → Logging panel is slow
3.12.2
2358 - Sending required CLOB between H2 and ORACLE throws ""inconsistent datatypes: expected - got CLOB""
2362 - Sending CLOB with USE_OLD_DATA as conflict in ORACLE throws ""inconsistent datatypes: expected - got CLOB""
4458 - Null pointer exception while extracting batch
4459 - Too many rows sent in reload for table when self-referencing foreign key
4462 - Oracle failed to create trigger, identifier is too long
4463 - Invalid node in sym_table_reload_request causes NPE
4464 - Failed to load batch ORA-00932: inconsistent datatypes: expected - got CLOB
4467 - MSSQL text fields are not permitted in where clause
4473 - Convert to modules, module is already installed
4476 - Kafka avro module missing, no errors from install
4479 - Confluent jackson modules missing from install
3.12.3 (Pro)
3930 - Offline nodes shown with active spinning icon on Dashboard "nodes" panel
4117 - Outgoing Loads screen reports Load not complete (when it is) when zero tables are synced
4466 - Permission test during installation fails on MySQL Clusters
4480 - Load data wizard does not check where conditions for a select tables case when all tables selected
4482 - Control Center shows JMX is disabled after enabling from installer
4511 - NPE on incoming batch window when batch in error
4513 - Setup of load only node hangs because of NPE
3.12.3
4483 - Role tables are not filtered for pre-3.9 installations
4484 - Log exception stack traces after log file rotates
4492 - MySQL before 5.0.32 does not have drop trigger if exists
4493 - Transforms start and end table for each row causing poor performance with bulk data loading
4496 - If node registers, immediate pull causes server to indicate that node is not registered
4497 - Load Filter wildcard specification does not work when catalog and table are wildcards, and schema is specified
4502 - RouterService.routeDataForChannel checks incorrect reference for null
4504 - NodeService needs to be more persistent when looking for authenticated nodes
4506 - Regression NPE in Oracle, Tibero bulk loader and Hbase, and Mongo data loader factory
4516 - Upgrade Jetty
4517 - Stream to file threshold, Illegal State Exception: There is no content to read. Memory buffer was empty
3.12.4 (Pro)
4536 - Conflict timestamp or version column name not displayed on edit
4537 - Nodes panel doesn’t show hierarchy if multi-primary detected
4542 - Admin privilege maintenance screen on remote nodes
4554 - Staging encryption batch errors invalid lines and did not receive ack
4558 - NPE during background refresh for session expiration
3.12.4
4507 - Master to master setup fails to send initial load if registering with a node other than registration server
4519 - Database name with special character like minus sign, fails to create trigger when in same database
4522 - DbCompare does not seem to support schema/database prefixed table names for at least the target node
4526 - [dbcompare command] - Generate script wrongly
4531 - Postgres Varchar column with no defined length getting converted to a LONGVARCHAR
4547 - Data Extractor gets "java.lang.IllegalStateException: There is no content to read" from staging
4548 - Oracle Bulk Loader: quote columns in control file if db.delimited.identifier.mode is set to true
4549 - Documentation: Transform Column Policy has invalid choice
4551 - Initial load time empty at client
4553 - Transforms break deletes when transforming primary key column and not capturing old data as specified in trigger
4557 - Stream to file, Illegal State Exception: There is no content to read. Memory buffer was empty
4565 - New Group Link should send node info to nodes in source node group ID and to nodes in target node group ID
3.12.5 (Pro)
4238 - Can’t delete group links once created
4576 - Filter in Design Overview tab does not work correctly
4594 - First time setup admin user encryption strength
4601 - Prevent registration server from being unregistered
4602 - Allow a group link to be deleted when its only dependencies are system transforms
4613 - Fix query result grid bugs
4620 - Preserve sort order when switching screens
4622 - Nodes panel buttons for send, registration, and test should be enabled for multi-primary
4628 - Fix various bugs with the Design tab
3.12.5
4574 - Column match routers can mark batches unrouted with initial load select set in table reload request only
4577 - Restrict pro tables from syncing to invalid versions and open source deployments
4582 - Heartbeat job should catch exceptions from heartbeat listeners
4591 - Multi-primary out of sync race condition with NEWER_WINS conflicts
4598 - SQL Server bulk loader data conversion with decimal scientific notation
4600 - SQL-Server bulk loader not allowing create events
4607 - JMX invalid username or password on Java 11
4610 - Postgres Conflict Resolution fails to resolve
4612 - SQL Server primary key update should create update event when only one record is updated
4619 - dbcompare using only first table on command line
4627 - Hardcoded KeyManagerFactory algorithm (sunX509). Symmetric doesn’t work for IBM JRE
4630 - TransportManagerFactory needs to call SelfSignedX509TrustManager constructor with actual trust store used instead of null
3.12.6 (Pro)
3123 - License Key fails to synchronize (data truncated) to target node of DB2 on zSeries
4410 - "Reconnect" to database for SymmetricDS installation fails with security exception
4634 - Config Import from Overview Panel fails because of cleanup of design diagram records
4637 - Installer overwrites setenv.bat and keystore password no longer matches
4638 - Nodes panel allows attempt to control non-existent node engine
4677 - Allow transform column PK and order fields to be edited
4684 - Manage Incoming Batches screen should say "Source Node" when viewing a batch’s details
4688 - Disconnect a user’s session(s) when the user gets deleted
4689 - Make Interface Name field only selectable for BeanShell extensions
4703 - BigDecimal value in Row needs to use toPlainString() when returning string representation
4704 - Permission check for DB2 zOS is not valid
4713 - Console events retention parameter
4727 - Prevent problematic characters from being included in group IDs
4749 - Value too long for column node_password
3.12.6
4631 - Default Conflict resolution (IS_CAPTURE_TIME_NEWER) doesn’t work when target node is a load only node
4644 - Multiple nodes need synchronized access to keystore
4646 - Repetition in configuration file and registration batch
4648 - Postgres Conflict Resolution blocks when trying to resolve
4649 - Auto resolve ConflictException hides unique key exception
4650 - Conflict with NEWER_WINS implicit conversion error
4651 - Column match routers can mark batches unrouted with initial load select set in table reload request only
4653 - NPE in PostgreSqlDmlStatement
4660 - Multi-primary subsequent data loads blocked
4669 - Extensions should reload when changes to sym_extension
4670 - Deletes and conflict detection USE_CHANGED_DATA with resolved NEWER_WINS
4671 - sym command should log to standard error by default
4676 - Insert into sym_data replaces current_timestamp string with question mark to allow specified timestamp but fails sometimes
4679 - Snowflake module is missing AWS libraries for bulk loading S3
4680 - Spaces in sym_trigger fields causes create trigger failure
4686 - Push initial load in background gets early EOF log message
4687 - Postgres conflict when unique index uses primary key index
4696 - DB2 zOS trigger permission check
4697 - Server can’t open truststore after upgrade
4698 - Sybase Quick Config fails to get schemas when changing catalog selection
4699 - Date time values are not selectable in DB2 z/OS due to lpad function.
4700 - DB2 zOS replace function will not work with CLOBs
4701 - Windows unable to access log4j2.xml file on command line
4707 - MSSQL load only nodes do not support identity inserts
4709 - DB2 zOS uses long varchar instead of clobs
4720 - MultiBatchStagingWriter writes "\r\n" on Windows for end of lines in child batch staging files
4722 - Invalid PK order on send-schema if source db is oracle
4725 - Using same column twice in sync key names gets error
4730 - MSSQL Bulk Writer continues to call start and end for the table even during fallback to default
4732 - Do not worry about conflict winner in load only or log based target nodes
4738 - The primary keys are out of order for H2 when returned from Table.java getPrimaryKeyColumns
4742 - Postgres conflict but can’t ignore row
4746 - Log4j should not be required. LogSummaryAppenderUtils currently throws a ClassCastException if a log4j2 is not present
4750 - On SQL-Server can’t drop trigger, incorrect syntax
4779 - SQL Server error while syncing data with identity column
3.12.7 (Pro)
4760 - Fix batch screen lazy loading bugs
4767 - Cannot save integer parameters greater than 999
4768 - Hide system channels from Configure Table Triggers screen dropdowns when console.web.hide.system.info is true
4769 - Mixed Pro/OSS deployment syncs sym_console_role table
4770 - Include "https" instead of "http" in the default sync url when https.enable is true
4771 - Fix sorting on batch screens
4784 - SQL Server Log Miner formats row_data incorrectly when value contains backslash
4790 - Fix batch screen "<Not Ok>" filter
4791 - Fix side-by-side documentation being cut off
4793 - Trust store password is required when storing
4794 - Installer error String index out of range: -1
4796 - Fix batch window NullPointerException
4803 - Fix Configure Table Triggers screen slowness
4806 - Fix transaction file generation for MySQL 5.5 and earlier
4811 - Fix load wizard table selection screen
4816 - Design tab not displaying transforms correctly when "use source catalog/schema" turned on the router
4820 - Fix Oracle SqlException on batch screens
4839 - Push and pull threads panel shows file sync when not enabled
4851 - Fix download CSV button on batch details window
4854 - Fix node selection grid in data load wizard
3.12.7
4571 - Password file read access must be restricted: security/jmxremote.password
4573 - Dynamic Query Builder support for bit
4764 - Raima Create Snapshot does not work
4783 - Initial load can slow down when batches on other channels cause conflict resolution
4786 - Force trigger regenerate does not work properly
4797 - JMX local host name unknown
4798 - No data found while correcting foreign key error for reload batch
4802 - On Tomcat, ack was not sent successfully: java.io.IOException: Premature EOF
4810 - Fix casting in MsSqlDmlStatement
4814 - binary column that is zero length (not null) sends to null to target during initial load
4817 - "smallint out of range" exception when incrementing failed login count > 32767
4818 - Fix SqlException when generating snapshot from a node with an Oracle database
4823 - Postgres unique constraint causes current transaction is aborted
4831 - Initial Load events for config tables need to use correct dialect when running load only, extract only, or log miner
4833 - Can’t export batch that includes sym_node_security
4840 - Sybase ASE primary key update should create update event when only one record is updated
4842 - Exception during registration with multi-primary
4843 - Push and pull errors while node registration is pending
4852 - Registration server script using extensions
4873 - "Old Data" comparison fails on Derby for LONG VARCHAR data types
3.12.8 (Pro)
4858 - Fix grid bug on captured data tab of outgoing batch details window
4863 - Null Pointer on new Transform panel when meta data is turned off
4876 - SQL Server: log mining capture should wrap all non-null values in double quotes
4877 - Fix exception in node setup wizard
4879 - Fix message on node status dashboard panel
4881 - Sybase issue with getting update counts
4887 - Oracle log miner stops processing rows when running into a DML statement that is not parseable
4889 - Fix table count in quick config wizard
4890 - dbcp2 package requires setting of validation query, otherwise it runs jdbc driver isValid() method, causing problems
4891 - Oracle log miner fails to move forward in the log mining when starting SCN is too low
4896 - Fix TreeGrid style being changed when visiting the Design tab
4897 - Oracle Log Miner: handle parse exceptions by logging and moving on
4901 - Fix inability to open multiple form tabs at once
4905 - Fix authentication failure dialog style on Configure LDAP screen
4919 - Fix custom transforms in transform column editor
4928 - Fix incoming batch download button
4930 - Fix NullPointerException in the resolve data tab of the batch details window
3.12.8
2415 - Reload a table causes ORA-01652: unable to extend temp segment by 128 in tablespace
4454 - Monitors: Memory and CPU do not provide detailed monitor information
4849 - Firebird result set is closed after routing
4859 - SQL-Server error cannot update identity column during auto resolution
4860 - Invalid module during startup after upgrade
4862 - Create WAR gets instance.uuid does not exist
4866 - Expandable trigger support for trigger names
4867 - Fix ClassCastException in SQL explorer
4880 - Upgrade DBCP and Vaadin libraries
4884 - Docker image doesn’t start because security directory permissions
4888 - SQL Server 2016 using SESSION_CONTEXT only works with SQL Server JDBC driver, not the JTDS driver
4892 - NPE in mail service
4900 - Symmetric upgrade with Firebird database fails to upgrade when upgrading to 3.12
4902 - Auto resolve when primary key has binary/varbinary
4903 - Oracle bulk loader not being used on Oracle >= 12.2
4906 - Sybase ASE: update trigger needs to set nocount on on very first statement
4916 - SYM_DESIGN_DIAGRAM and SYM_DIAGRAM_GROUP need to be filtered out when routing to certain nodes
4929 - Fix SQL explorer row editor
3.12.9 (Pro)
4917 - Unable to configure mail server- bad credentials -irrespective of giving the correct credentials.
4936 - Fix date sorting in grids
4946 - Fix ArrayIndexOutOfBoundsException in the resolve data tab of the batch details window
4956 - Fix ClassNotFoundException in quick config wizard
4959 - Mail server password not working
4961 - Fix unknown error on the NodesPanel when there are nodes in multiple tiers
4962 - Error on outgoing batch panel on postrgres
4969 - Enable debug logging for log miner
3.12.9
4857 - ClassNotFound exception in symadmin
4920 - Initial load blocking
4931 - Stream LOB when row is deleted can get not null error when required by constraint
4933 - Invalid MySQL update statement that queries same table when starting load to multiple nodes
4940 - NullPointerException in TriggerRouterService after conflict resolution (NEWER_WINS)
4941 - Null pointer on registration when conflict resolution is needed
4944 - Docker container not being deployed with latest SymmetricDS artifacts
4945 - USE_CHANGED_DATA / NEWER_WINS not working on Android
4947 - Sybase ASE null byte with data type of TIME
4948 - Bulk conflict resolution- when falling back to default writer, clear last error in context so context is clean
4965 - AndroidSymmetricEngine: (re)start after stop fails
4966 - Upgrade Jetty, BC, Kotlin, okhttp libraries
3.12.10 (Pro)
4967 - Oracle log miner SQL parser exception from functional index
4979 - Log mining trigger hist error causes drop trigger to run
4999 - Oracle Log Miner needs to handle rollback to savepoints
5002 - Manage Installed Triggers screen should filter out the symmetric tables on the source table name instead of the trigger id
5005 - Table list from remote client is not populated
5008 - Push and pull threads panels are empty
3.12.10
4345 - Postgres array types improperly cast into non-array types
4488 - UPDATE triggers have local variables defined with hardcoded "utf8" charset
4675 - SQL-Server trigger not implemented exception for NCHAR
4970 - Auto resolve missing foreign key with load only or extract only node
4971 - Connection leak during routing when a group link is added
4973 - Unicode synchronization of utf8 strings with 4 byte characters in MySql and MariaDB does not work
4974 - Routing and extracting had X columns but expected Y
4976 - MySQL: Drop trigger silently fails
4977 - Oracle create index under same user/schema as table
4980 - MySQL 8 CHAR data type needs spaces right trimmed
4986 - Work with MySQL 8 JDBC driver
4987 - Kafka escaping for special characters
4988 - Batch in conflict won’t load when unique key value is null
4996 - Table PK column order mismatch with column order can cause errors
4998 - Firebird error while dropping trigger, token unknown
5001 - Postgres converts json and uuid types to text for create table
5003 - Oracle: Geometry column type can not be used in a where clause
5006 - Push, pull, extract thread name has wrong queue
5047 - Windows install to directory with parenthesis
3.12.11 (Pro)
4366 - Manage outgoing loads screen does not update completed tables with last set of active tables when those tables are complete
5012 - Oracle log miner uses deprecated CONTINUOUS_MINE during permission check
5018 - MS log miner needs snapshot isolation to work under heavy load
5024 - Setup wizard for Informix fails with method not supported IfxSqliConnect.isValid(int)
5026 - SQL Server: The data types time and datetime are incompatible in the equal to operator.
5031 - UIDetachedException during sync triggers
5033 - Oracle Log Miner fails to load transactions when SCN value is too old and gets stuck in loop
5034 - Initial load shows up as pending and loading at the same time
5036 - Outgoing batch panel remote status not working
5038 - Select all checkbox missing on node selection in load wizard
5039 - Process info panel uses wrong calculation for rows per second
5050 - Screens call sync triggers in contention with config router
5052 - Batch window pop-up is slow to display
5053 - SQL Explorer not showing triggers from another catalog on MS-SQL
3.12.11
1387 - DB2 cannot synchronize tables from non-default schemas.
4717 - Client ack rejected with server busy when using channel queues
4991 - Kafka XML escaping needed
5014 - BigQuery connectivity not loading security file properly
5019 - Service fails to install on SUSE 15
5022 - BigQuery type not found FLOAT
5027 - Fix NPE when a trigger hist is rebuilt with a reason code of H during routing
5029 - Push acknowledgements are not ordered by batch id
5030 - Load Filter documentation indicates that Java and BSH are the only available scripting, but there is also SQL
5041 - Upgrade from 3.7 to 3.11 fails to upgrade schema
3.12.12 (Pro)
5065 - Support for encrypted pem certificates
5071 - Oracle log miner handle multiplexed redo log file
5073 - Table names for databases with empty string ("") default catalogs not present in quick setup wizard table selection screen
5083 - Fix SQLException on Manage Nodes screen
5087 - MS log miner permission check failing
5090 - Prevent modal when adding more nodes than license allows
5103 - Prevent expired certificates from being imported
5114 - Initial Load for SQL Server shows completed when data load batches is more than calculated batches
3.12.12
2233 - IgnoreColumnException and IgnoreRowException are not handled correctly in bsh extension scripts
2382 - Source file creation time not retained at destination folder during file synchronization
4658 - Postgresql data type - character varying problem
4957 - Recording status of Batch with large payloads fails on Derby
5054 - UnixService for wrapper: needs to wait for cmdline object to show up when starting wrapper on Amazon EC2
5055 - DB2 filtering node list with a limit has invalid syntax
5056 - DBExport for AS400 fails compatibility check
5062 - Oracle SDO_GEOMETRY: problem inserting when both BLOB and SDO_GEOMETRY are in the table
5066 - Sybase column of type unitext causing error when used in "where" clause
5067 - Failing to sync stored procedures containing semi-colons
5068 - Postgres operator doesn’t exist when using domain user defined type
5070 - Error on purge on Firebird DB with Dialect 1
5072 - Conflict resolution on Derby for batch with large payloads fails with SQLException
5077 - Spaces prevent conflict configuration from working
5078 - Tibero won’t update table with LOB when missing PK
5079 - Tibero bulk loader failing because table and columns quoted in control file
5080 - Bulk loaders treat all conflicts as if their detection type is USE_CHANGED_DATA when forming where clauses
5082 - Mail service support for encrypted passwords
5085 - Data loader gets ArrayIndexOutOfBoundsException on batch
5086 - Wrong logic for deleting triggers on SQL Anywhere
5088 - Duplicate column name error
5089 - Sybase inserts a space instead of an empty string or null value
5091 - Table Reload Requests: create schema, delete data, before custom sql should honor initial load order less than 0
5092 - Lexical error when issuing a foreign key correction with a carriage return in the data
5095 - Initial load before custom sql to support table replacement with delimiters
5097 - Postgres default value support for gen_random_uuid()
5105 - Transforms should return old and new values for updates when column is primary key
5108 - Failed to route beanshell event from conflict resolution
5115 - Kafka filtering of event types that are not an INSERT, UPDATE, OR DELETE
653 - dbexport should export xml in utf-8
3.12.13 (Pro)
5120 - Sync Triggers- Parallel Mode code changes causes more than one trigger for a table to be active when in extract only
3.12.13
5121 - Data mapping is incorrect on an insert fallback to update potentially when column count on target is not the same as row data
5133 - During a full load do not ok existing batches for the same load id.
5144 - Multiple active trigger histories result when more than one trigger assigned to a table
5151 - Upgrade log4j2
3.12.14
5159 - Upgrade log4j2 again
3.12.15 (Pro)
5192 - Outgoing batches for sym_design_diagram are sent to nodes with versions earlier than 3.12
3.12.15
5123 - ASE, ASA, and MSSQL support for stream lob optimizations
5164 - Duplicate entry on file sync
5169 - Tibero reading of indexes fails when index is in another schema
5175 - JdbcBatchBulkDatabaseWriter only uses one record in the batches for deletes, and fails on record 10000
5177 - File Sync fails to create zip file when modify, followed by delete, followed by create is captured and sent in same batch
5190 - Upgrade log4j2 again again
3.12.16
5196 - "There are SymmetricDS tables that needed altered" on every start-up on Derby
5207 - Multi-tenant SymmetricDS instance fails to allow User Interface to function when some engines fail to start
5242 - DataGapFastDetector- needs to make sure that we find the last gap that is really the last gap
5248 - CVE-2022-22965: Spring Version needs to use 5.2.20.RELEASE or 5.3.18 in order to not be vulnerable to this security issue
3.12.17 (Pro)
5290 - Unable to Unregister 3rd Tier Node from Registration Server
3.12.17
5271 - Conflict Resolution of deletes of parent records fail to delete child records when child tables have no primary keys
5319 - SymmetricDS Not Properly Handling Unable to Write Batch to Kafka Error
5322 - Conflict resolution batch with token parsing error when newlines in data
5360 - Conflict resolution based on newer wins with transforms
3.12.18 (Pro)
5378 - Turning auto refresh off on batch screens prevents grid from being refreshed
3.12.18
4633 - Tables not created on reverse initial load
5383 - Snapshots do not contain log files when the files are in a non-default location specified in log4j2.xml
5390 - Initial load delete or truncate with table transform
5398 - Batch Locking issue when exceptions occur during locking (semaphore and batch file locking)
5437 - Conflict resolver fails if there are no primary keys on the target and it is set to not use primary keys from source
3.12.20
5508 - Multiple active trigger history for same table
5517 - Empty string default values not syncing to target
5521 - Create trigger hist if routing can’t find it
5527 - Cannot set up SQL Server log-based replication without "ALTER ANY DATABASE" permission
5540 - Apache commons-text version 1.9 security vulnerability
5573 - Unique index on function causes error when resolving a conflict
5578 - Script error from newer wins conflict resolution with old nodes
5584 - Trigger creation fails when 2 new triggers have trigger IDs that are identical when shortened
5594 - Failure to Flush when using ConvertToReload router.
3.12.22 (Pro)
5687 - Logging out of UI after running a job like sync triggers can interrupt job
3.12.23 (Pro)
5793 - SQL Server log miner skips over changes on a busy system
Tables
The following changes were made to the definition of configuration and runtime tables. Table changes are applied to the database automatically using data definition language (DDL) during startup.
New Tables
| Table Name | Description |
|---|---|
|
Role for users to assign privileges to use screens |
|
List of privileges for a role |
|
|
|
New Columns
| Column Name | Description |
|---|---|
|
The IP from which the user last logged in. |
|
Two Factor Authentication Key. |
|
Indicates if an email has been verified. |
|
Indicates if a password reset is required. |
|
Confirmation code for email verification and forgot password |
|
Create time for a confirmation code |
|
| Column Name | Description |
|---|---|
|
The database product name identified by SymmetricDS. |
| Column Name | Description |
|---|---|
|
The timestamp when an initial load was completed for this node. |
|
Number of failed login attempts |
| Column Name | Description |
|---|---|
|
Flag that indicates that this load is a full load for all tables, otherwise it is a partial load. |
Modified Tables
-
Index
idx_d_channel_idisUnique changed fromfalsetotrue
-
router_typedefault value changed fromnullto"default" -
router_typeisRequired changed fromfalsetotrue
Parameters
The following changes were made to add new parameters, modify their default value, modify their description, or remove them from use.
New Parameters
When auto-creating channels, this option enables removal of reload channels that weren’t used when rebalancing tables across reload channels (Default: true)
Enable use of batchps command over remote status for outgoing batches screen, which uses current list of batches being processed. When disabled, it will use batchstatus command with a list of batches to track over remote status. (Default: true)
The maximum number of characters to display in the data grid in the batch window. (Default: 100)
BeanShell script to include at the beginning of all scripts used in extensions (Default: )
Timeout Administrator Sessions in Minutes (Default: 30)
Allow console to authenticate user from a client certificate (X.509) (Default: false)
Regular expression to find the username from the distinguished name (DN) subject of a client certificate (X.509). The first group in parenthesis will be used for the username. The default regex will match on the common name (CN) field of the subject. (Default: .CN=(.?),.*)
Delay in milliseconds between login prompts following a failed login attempt (Default: 5000)
This is the retention time for the number of minutes that a console event will be retained. (Default: 20160)
The minimum number of characters that must be changed when changing a password (Default: 0)
Required two-factor authentication for login (Default: false)
Timeout for Verification Codes in Minutes (Default: 15)
The maximum number of sessions that can be active at a time (0 = unlimited) (Default: 20)
The maximum number of sessions that can be active at a time per user (0 = unlimited) (Default: 1)
Number of consecutive days a user can be logged out before getting disabled (0 = unlimited) (Default: 0)
Timeout User Sessions in Minutes (Default: 30)
Extract the capture time of each row and put it in the batch to be used by the conflict manager for picking the winning row during a conflict. Enable for the best precision in resolution, which includes a unix timestamp for each occurrence of insert, update, and delete. Disable to include a single unix timestamp to represent the entire batch, when accuracy is less important or conflict management isn’t needed. (Default: true)
Google BigQuery security credentials path for the json file containing the credentials to connect to Big Query. (Default: )
Sets the connection timeout on the internal HttpUrlConnection (Default: 90000)
For SQL-Server log miner, collect changes from all tables into memory, then sort them by change number and foreign keys. This attempts to preserve the order of changes across the database. When the same row is updated multiple times, SQL-Server will merge and return the last change number for it, which prevents accurate ordering. When disabled, the log miner collects and syncs changes from each table ordered by change number, processing each table in order by foreign keys. (Default: true)
The maximum number of rows to sort in memory. When the number of changes exceeds the maximum, it will process like the sort in memory parameter is disabled. (Default: 1000000)
Use a single T-SQL statement that returns which tables have changes waiting. The single round trip with SQL-Server can save time. When disabled, a statement is run for each table to check for changes. (Default: true)
Text displayed within a pop-up banner that must be accepted on login (Default: )
Encrypt the node_password in the sym_node_security table when saving it. Decryption is always done on reading if the node_password starts with "enc:" prefix. (Default: true)
Number of failed login attempts by a node before lockout (0 = never lockout, -1 = never lockout or record) (Default: 5)
The maximum number of children to display under each node on the node screen before dividing them into smaller groups. (Default: 100)
When auto-creating table triggers, some options will be detected and applied for the user, including enabling stream LOB for LONG columns and excluded ROWID columns. (Default: true)
This indicates how to process LOB values. "plain" is the default, which uses the JDBC interface directly. "createtemporarylob" uses the creation of CLOB/BLOB temporary objects to process the data. "streamlob" uses the stream interface of the JDBC interface (part of JDBC 4.0). (Default: plain)
Enable previous login information notification (Default: true)
Send a shared secret as part of the registration request. If the client and server secrets are non-blank and match, the registration is automatically allowed. (Default: )
Number of failed connections to remote node before falling back to follow group link direction = true (Default: 10)
Max number of batches to write to statistics listing for support snapshot. (Default: 10000)
Max number of files to write in directory listing for support snapshot. (Default: 50000)
Max number of nodes and channels for batch statistics, after which it will group by node only. (Default: 5000)
Max time for a snapshot operation to complete, such as gathering table definitions, before it will be interrupted so the snapshot completes in a reasonable amount of time. (Default: 30000)
The maximum size of a file that can be edited via the staging area screen. (Default: 25000000)
Automatically alter data, data_event and outgoing_batch tables to allow only row level locking. (Default: true)
Automatically change attribute on data, data_event and outgoing_batch tables to set the identity gap to the number provided. This prevents skipping a large number of identities that can cause routing to stop. Use 0 to disable. (Default: 1000)
When symmetric triggers are created and accessed, this is the prefix to use for the tables. If this is not set, it will use the sync.table.prefix as the trigger prefix (Default: )
When sync triggers expands a table name to include node external ID, use the first table found and clone it for all other nodes to improve performance. (Default: true)
Time out value in milliseconds for sync triggers. This timeout value will be used to time out the inactivate step and the create or update step. (Default: 3600)
These are settings that will be passed to the target JDBC driver as connection properties. Suggested settings by database are as follows: Oracle db.connection.properties=oracle.net.CONNECT_TIMEOUT=300000;oracle.jdbc.ReadTimeout=300000;SetBigStringTryClob=true (Default: )
Specify your target database driver. This applies for load only, extract only, and log based data capture. (Default: )
Specify a SQL statement that will be run when a target database connection is created (Default: )
This is the default number of rows that will be sent to the target database as a batch when SymmetricDS uses the JDBC batch API. Currently, only routing uses JDBC batch. The data loader does not. (Default: )
Override the target JDBC isolation level. The isolation level is detected by platform and automatically set, but it can be overridden here. Most platforms need at least read committed level to prevent phantom reads. (0=none, 1=read uncommitted, 2=read committed, 4=repeatable read, 8=serializable) (Default: )
This is the default fetch size for target streaming result sets. (Default: )
Specify your target database password. This applies for load only, extract only, and log based data capture. (Default: )
The initial size of the target connection pool (Default: )
The maximum number of target connections that will be allocated in the pool The http.concurrent.workers.max value should be half or less than half of this value. (Default: )
The maximum number of target connections that can remain idle in the pool, without extra ones being released (Default: )
This is how long a request for a target connection from the datasource will wait before giving up. (Default: )
This is how long a target connection can be idle before it will be evicted. (Default: )
The minimum number of target connections that can remain idle in the pool, without extra ones being created (Default: )
If set to true forces target database columns that contain character data to be read as bytes (bypassing JDBC driver character encoding) so the raw values be encoded using the system default character set (usually UTF8). This property was added to bypass MySQL character encoding so the raw data can be converted to utf8 directly. (Default: )
Most target symmetric queries have a timeout associated with them. This is the default. (Default: )
This property determines whether or not the target pool will validate objects before they are borrowed from the pool. (Default: )
This property determines whether or not the target pool will validate objects before they are returned to the pool. (Default: )
This property determines whether or not the target idle object evictor will validate connections. (Default: )
Specify your target database URL. This applies for load only, extract only, and log based data capture. (Default: )
Specify your target database user. This applies for load only, extract only, and log based data capture. (Default: )
This is the query to validate the target database connection in Connection Pool. It is database specific. The following are example statements for different databases. MySQL db.validation.query=select 1 Oracle db.validation.query=select 1 from dual DB2 db.validation.query=select max(1) from syscat.datatypes (Default: )
Defines the number of milliseconds before logging that a target query is slow. (Default: )
Defines whether the logging of target SQL statements include the values inline or not. (Default: )
Whether target binary fields should be treated as lobs (Default: )
Enable or disabled use of create or replace syntax on Oracle and MS-SQL 2016 and newer. (Default: true)
The delimiter to use when capturing changes from a DDL trigger. MS SQL-Server only. See: trigger.capture.ddl.changes (Default: $)
Modified Parameters
The maximum number of batches to display on the batch screens. (Old Default: 10000) (New Default: 0)
Indicates that old data should be used to create the update statement. If old data is equal to the new data and this property is set to true, then no update statement will be run. (Old Default: false) (New Default: true)
For Unix, fast scan will look for files that were modified since the last run of file sync tracker and commit their changes using the data loader max commit row setting. When it finds modified directories, it compares to the file snapshot to find changes. For a large file system, this is faster and more efficient than the normal tracker. This setting works on most Unix systems, but reportedly not working on some Windows systems. (Old Default: true) (New Default: false)
{REMOVED}