SymmetricDS Pro 3.8 User Guide

SymmetricDS is written in Java and requires a Java Runtime Environment (JRE) Standard Edition (SE) or Java Development Kit (JDK) Standard Edition (SE) version 7.0 or above. Most major operating systems and databases are supported. See the list of supported databases in the Database Compatibility section. The minimum operating system requirements are:

The memory, disk, and CPU requirements increase with the number of connected clients and the amount of data being synchronized. The best way to size a server is to simulate synchronization in a lower environment and benchmark data loading. However, a rule of thumb for servers is one server-class CPU with 2 GB of memory for every 500 MB/hour of data transfer and 350 clients. Multiple servers can be used as a cluster behind a load balancer to achieve better performance and availability.

SymmetricDS Pro is accessed from a web console, which requires one of the following supported web browsers:

A node is responsible for synchronizing the data from a database or file system with other nodes in the network using HTTP. Nodes are assigned to one of the node Groups that are configured together as a unit. The node groups are linked together with Group Links to define either a push or pull communication. A pull causes one node to connect with other nodes and request changes that are waiting, while a push causes one node to connect with other nodes when it has changes to send.

Each node is connected to a database with a Java Database Connectivity (JDBC) driver using a connection URL, username, and password. While nodes can be separated across wide area networks, the database a node is connected to should be located nearby on a local area network for the best performance. Using its database connection, a node creates tables as a Data Model for configuration settings and runtime operations. The user populates configuration tables to define the synchronization and the runtime tables capture changes and track activity. The tables to sync can be located in any Catalog and Schema that are accessible from the connection, while the files to sync can be located in any directory that is accessible on the local server.

At startup, SymmetricDS looks for Node Properties Files and starts a node for each file it finds, which allows multiple nodes to run in the same instance and share resources. The property file for a node contains its external ID, node group, registration server URL, and database connection information. The external ID is the name for a node used to identify it from other nodes. One node is configured as the registration server where the master configuration is stored. When a node is started for the first time, it contacts the registration server using a registration process that sends its external ID and node group. In response, the node receives its configuration and a node password that must be sent as authentication during synchronization with other nodes.

Each subsystem in the node is responsible for part of the data movement and is controlled through configuration. Data flows through the system in the following steps:

SymmetricDS offers a rich set of features with flexible configuration for large scale deployment in a mixed environment with multiple systems.

SymmetricDS is a feature-rich data synchronization solution that focuses on ease of use, openness, and flexibility. The software encourages interoperability and accessibility for users and developers with the availability of source code, an application programming interface (API), and a data model supported by documentation. Configuration includes a powerful set of options to define node topology, communication direction, transformation of data, and integration with external systems. Through scripts and Java code, the user can also extend functionality with custom behavior. With a central database for setup and runtime information, the user has one place to configure, manage, and troubleshoot synchronization, with changes taking immediate effect across the network.

The trigger-based data capture system is easy to understand and widely supported by database systems. Table synchronization can be setup by users and application developers without requiring a database administrator to modify the server. Triggers are database objects written in a procedural language, so they are open for examination, and include flexible configuration options for conditions and customization. Some overhead is associated with triggers, but they perform well for applications of online transaction processing, and their benefits of flexibility and maintenance outweigh the cost for most scenarios.

Using an architecture based on web server technology, many simultaneous requests can be handled at a central server, with proven deployments in production supporting more than ten thousand client nodes. Large networks of nodes can be grouped into tiers for more control and efficiency, with each group synchronizing data to the next tier. Data loading is durable and reliable by tracking batches in transactions and retrying of faults for automatic recovery, making it a low maintenance system.

SymmetricDS Pro is commercial software that is licensed, not sold. It is subject to the terms of the End User License Agreement (EULA) and any accompanying JumpMind Support Contract. See the standard SymmetricDS Pro license for reference, but your agreement with JumpMind may be different.

The SymmetricDS installer is an executable jar file named symmetric-pro-<version>-setup.jar. In order to run the installer, you must have the Java Runtime Environment (JRE) version 6.0 or newer installed. Start the installer by double-clicking it (if the JRE is in your path and associated with .jar files), or by running it from a command prompt, like this:

java -jar symmetric-pro-<version>-setup.jar

The default installation will run in graphical mode, but it can also be run from a command window by adding the -console argument on the end of the command.

To continue setup and configuration of SymmetricDS, refer to the Setup section.

SymmetricDS can be configured to start automatically when the system boots, running as a Windows service or Linux/Unix daemon. A wrapper process starts SymmetricDS and monitors it, so it can be restarted if it runs out of memory or exits unexpectedly. The wrapper writes standard output and standard error to the logs/wrapper.log file.

For SymmetricDS Pro, you may have already installed as a service, so this section will show you how to manually install the service from command line.

A single SymmetricDS node may be deployed across a series of servers to cooperate as a cluster. A node can be clustered to provide load balancing and high availability.

When using clustering, a hardware load balancer is typically used, but a software load balancer, such as a reverse proxy, can also be used.

For clustered nodes running SymmetricDS 3.8 and later, the recommended approach is to configure the load balancer to use sticky sessions and ensure the staging directory for all nodes in the cluster are using a shared network drive. Sticky sessions are needed to support reservation requests, which allows for nodes to connect and obtain a reservation before connecting again and pushing their changes. The shared staging directory is needed to support extract in background of the initial load, which is extracted by one node, but served by different nodes in the cluster. If the start.initial.load.extract.job property is disabled, then shared staging is not required, but the performance of the initial load may be degraded.

For clustered nodes running SymmetricDS 3.7 and earlier, it is recommended to round robin client requests to the cluster and configure the load balancer for stateless connections.

Also, the sync.url (discussed in Registration URL) SymmetricDS property should be set to the URL of the load balancer.

If the cluster will be running any of the SymmetricDS jobs, then the cluster.lock.enabled property should be set to true. By setting this property to true, SymmetricDS will use a row in the LOCK table as a semaphore to make sure that only one instance at a time runs a job. When a lock is acquired, a row is updated in the lock table with the time of the lock and the server id of the locking job. The lock time is set back to null when the job is finished running. Another instance of SymmetricDS cannot acquire a lock until the locking instance (according to the server id) releases the lock. If an instance is terminated while the lock is still held, an instance with the same server id is allowed to reacquire the lock. If the locking instance remains down, the lock can be broken after a period of time, specified by the cluster.lock.timeout.ms property, has expired. Note that if the job is still running and the lock expires, two jobs could be running at the same time which could cause database deadlocks.

By default, the locking server id is the hostname of the server. If two clustered instances are running on the same server, then the cluster.server.id property may be set to indicate the name that the instance should use for its server id.

When deploying SymmetricDS to an application server like Tomcat or JBoss, no special session clustering needs to be configured for the application server.

It is recommended that SymmetricDS is installed as a standalone service, however there are two other deployment options.

When opening the SymmetricDS Pro web console, if there are no SymmetricDS nodes defined within the running SymmetricDS instance, the Node Setup Wizard will be displayed to guide you through the process of creating one. There are two types of SymmetricDS nodes:

In addition to the two standard node types above, there is also an option to create a Demo installation. This Demo installation does the following:

The Demo option is a great way to look at an existing synchronization scenario configuration and setup. If your goal is to begin synchronizing your own databases, select the Master or Node options from above.

Once the SymmetricDS software is installed, an instance of it is running, and a master node is setup, the synchronization scenario must be configured (i.e. what tables and/or files do I want to synchronize from which source node groups to which target node groups). The Quick Config wizard simplifies this process by walking you through several simple configuration steps.

The Quick Config Wizard can be found under the Configure→Overview screen.

Once configuration is complete SymmetricDS will automatically pop up a Node Setup Wizard if there is only one node currently setup in your network.

See Add Node for more information.

The node setup wizard generates a properties file that contains all the information a node needs to bootstrap itself.

Each node that is deployed to a server is represented by a properties file that allows it to connect to a database and register with a parent node. Properties are configured in a file named xxxxx.properties. It is placed in the engines directory of the SymmetricDS install. The file is usually named according to the engine.name, but it is not a requirement.

To give a node its identity, the following properties are required. Any other properties found in conf/symmetric.properties can be overridden for a specific engine in an engine’s properties file. If the properties are changed in conf/symmetric.properties they will take effect across all engines deployed to the server.

See Startup Parameters, for additional parameters that can be specified in the engine properties file.

There are two ways a SymmetricDS configuration can be imported. Both ways support importing an export that is in CSV format. In order to import the SQL format you would run the SQL via database specific tools.

Export your configuration at any time through the web console. Use the drop down to select the format you would like to use for the export.

Exporting your configuration might be useful in order to save a configuration in source control or to be used as an import into another environment. Export and Import are frequently used to migrate configuration between test and production environments.

In SymmetricDS, configuration rules are applied to groups of nodes versus individual nodes. A group is a categorization of nodes with similar synchronization needs. For example, in a synchronization scenario where a corporate office database is synchronized with field office databases, two node groups would be created, one for the corporate office database (Corporate), and one for the field office databases (Field_office). In the corporate group, there would be a single node and database. In the field_office group, there would be many nodes and databases, one for each field office. Configuration rules/elements are applied to the node group versus the individual nodes in order to simplify the configuration setup (no need to configure each individual field office node, just how the field office nodes sync with the corporate office node).

Group links define at a high level how data moves throughout your synchronization scenario. The group link defines which node groups will synchronize data to other node groups and within that exchange, which node group will initiate the conversation for that exchange.

Routers ride on top of group links. While a group link specifies that data should be moved from nodes in a source node group to nodes in a target node group, routers define more specifically which captured data from a source node should be sent to which specific nodes in a target node group, all within the context of the node group link.

Once group links and routers are defined, configuration must be completed to specify which data (tables, file systems, etc.) should be synchronized over those links and routers. The next step in defining which specific data in the database is moved is to define logical groupings for that data. Channels define those logical groupings. As an example, a set of tables that hold customer data might be logically grouped together in a Customer channel. Sales, returns, tenders, etc. (transaction data) might be logically grouped into a transaction channel. A default channel is automatically created that all tables will fall into unless other channels are created and specified. The default channel is called 'default'.

Channels can be disabled, suspended, or scheduled as needed.

The next step in defining the synchronization scenario is to define which tables should be synchronized. The simplest way to do this is by using the Auto Create…​ button on the Configure → Table Triggers screen. The Auto Create functionality introspects the master node’s database and looks for available tables to synchronize.

Note that you will be configuring tables to synchronize for specific Routers. You will get a chance to select tables for each configured Router. Select the channel you desire from the dropdown, select the tables that should be synchronized using that channel, and then click the Next button. A summary will be presented at the end of the configuration. Click Finish to create the triggers.

The Auto Create functionality can be used repeatedly to select different table, router and channel combinations. If you are configuring a new set of tables triggers that should use an independent Trigger definition be sure to enter values for either the Trigger Prefix or the Trigger Suffix fields. These will be used to create a unique Trigger Id.

For each table selected in the Auto Create wizard, a corresponding Table Trigger configuration element will be added that can be maintained within the Table Triggers list screen. New Table Trigger configuration elements can also be created manually from this screen.

As discussed previously, Routers define more specifically which captured data from a source node should be sent to which specific nodes in a target node group. Table Triggers define which tables should be synchronized. Table Routing defines how the Routers and Table Triggers work together, specifying what tables are associated with each router (and thus the direction the table data flows).

The simplest way to set up Table Routing is to use the Auto Create…​ button.

You can also select routers on the first screen of the auto create if you would like to select routers first then triggers. Both approaches will result in the same table routing.

The table routing elements created by the wizard can be maintained from the Table Routing List Screen. New table routing entries can also be created manually from this screen if desired.

In addition to supporting database synchronization, SymmetricDS also supports File Synchronization. Similar to database synchronization which allows configuring [Table Triggers], SymmetricDS also supports setting up File Triggers. A file trigger is equivalent to specifying a directory structure or path that should be "watched" for files that need to be synchronized.

The file routing concept is identical to the Table Routing concept except that it is for files versus database tables. The File Triggers specify which directory structures on the source node to watch, and the Routers define where the data is sent based on the configured router. Similar to Table Routing the simplest way to set up file routing is via the use of the Auto Create button.

You can also select routers on the first screen of the auto create if you would like to select routers first then file triggers. Both approaches will result in the same file routing.

The file routing elements created by the wizard can be maintained from the File Routing List Screen. New file routing entries can also be created manually from this screen if desired.

Conflict detection is the act of determining if an insert, update or delete is in "conflict" due to the target data row not being consistent with the data at the source prior to the insert/update/delete.

Transforms allow you to manipulate data on a source node or on a target node, as the data is being loaded or extracted.

The source trigger creates the synchronization data, while the transformation configuration decides what to do with the synchronization data as it is either being extracted from the source or loaded into the target. You have the flexibility of defining different transformation behavior depending on whether the source change that triggered the synchronization was an Insert, Update, or Delete. In the case of Delete, you even have options on what exactly to do on the target side, be it a delete of a row, setting columns to specific values, or absolutely nothing at all.

Load Filters are a way to take a specific action when a row of data is loaded by SymmetricDS at a destination database node.

Grouplets are user defined sub groups of nodes as defined by a node’s external id. They are used to enable or disable synchronization at a finer grained level than group. Grouplets are typically used to accomplish piloting or rolling out partial synchronization functionality to a smaller group of nodes in a large network of nodes.

Extensions are custom code written to a plug-in interface, which allows them to run inside the engine and change its default behavior. Saving extension code in the configuration has the advantage of dynamically running without deployment or restarting. Configured extensions are available to other nodes and move between environments when configuration is exported and imported.

Parameters can be used to help tune and configure your SymmetricDS configuration. Parameters can be set for an individual node or for all nodes in your network.

See Parameter List for a complete list of parameters.

Additional users may be set up to access the SymmetricDS web console.

Configuring a Users authentication on SymmetricDS using LDAP is as simple as setting up the following three Parameters.

A list of base DNs can be specified by separating each entry by a pipe ("|") symbol.
For example:
ou=Users,o=IT,c=US,dc=corp,dc=local|ou=Users,o=Tech,c=US,dc=corp,dc=local

These parameters can be set on the Configure LDAP page by filling out the form and clicking Save.

These parameters can also be set by manually by entering the desired values in Configure→Parameters.

You may test the values that have been input into the form to test the connection to your LDAP directory. To do this, click the Test button, enter the user credentials and click Test to test the authentication. If the authentication succeeded, save the form.

The web console requires a valid license key before it will allow access to screens. The user is prompted automatically to enter a valid license if one is missing, or it can updated from the configuration screen. The license key contains a group of fields to define how the software is licensed for use.

The license key is a block of encoded text that is sent to the user over email, or it can be retrieved from the Support Center. When an installed license key expires, the web console will require a new valid key before allowing access, but data synchronization and command line access will continue to function normally.

A mail server can be configured for sending email notifications.

The "Test" button will use the current settings on the screen to connect to the mail server with the configured transport and report any errors.

A monitor watches some part of the system for a problem, checking to see if the monitored value exceeds a threshold. (To be notified immediately of new monitor events, configure a notification.)

A notification sends a message to the user when a monitor event records a system problem. First configure a monitor to watch the system and record events with a specific severity level. Then, configure a notification to match the severity level and write to the log or send an email.

Nodes can be added and managed in the Nodes screen under the Manage tab.

Most work done by SymmetricDS is initiated by jobs. Jobs are tasks that are started and scheduled by a job manager. Jobs are enabled by the start.<name>.job parameter.

Most jobs are enabled by default. The frequency at which a job runs is controlled by one of two parameters:

If a valid cron property exists in the configuration, then it will be used to schedule the job. Otherwise, the job manager will attempt to use the period.time.ms property.

From the manage jobs screen the frequency that a job will run for the current node in the cluster can be managed. If you want to change the frequency a job runs across a group or for everyone, you modify the parameter in the Parameters screen.

A job can also be stopped and restarted. If a job is scheduled to run infrequently and the job is needed to run immediately you can select job and click Run Now.

SymmetricDS utilizes Spring’s CRON support, which includes seconds as the first parameter. This differs from the typical Unix-based implementation, where the first parameter is usually minutes. For example, */15 * * * * * means every 15 seconds, not every 15 minutes. See Spring’s documentation for more details.

Some jobs cannot be run in parallel against a single node. These jobs use the LOCK table to get an exclusive semaphore to run the job. This table is only used if the cluster.lock.enabled is set to true.

SymmetricDS installs database triggers to capture changes in the DATA table. A record of the triggers that were installed and what columns are being captured is stored in the TRIGGER_HIST table. When data is captured in DATA it references the TRIGGER_HIST record that represented the trigger at the time data was captured.

This is necessary because if a trigger is rebuilt after columns are added or removed and data that was captured by the old trigger has not yet been delivered, we need a record of what columns were in play at the time the data had been captured.

The Installed Triggers Screen is used to view tables that have successfully been installed with triggers.

This screen allows you to Rebuild an individual database trigger or Rebuild All database triggers. It also allows you to Drop triggers. Only reason you might need to drop triggers to alter a table or perform a batch job where you don’t want data to be captured.

The TRIGGER_HIST table records the reason a trigger was rebuilt. The following reasons are possible:

A configuration entry in Trigger without any history in Trigger Hist results in a new trigger being created (N). The Trigger Hist stores a hash of the underlying table, so any alteration to the table causes the trigger to be rebuilt (S). When the last_update_time is changed on the Trigger entry, the configuration change causes the trigger to be rebuilt ©. If an entry in Trigger Hist is missing the corresponding database trigger, the trigger is created (T).

The Outgoing Loads screen shows the number of loads that have been queued. It also lists loads that have had Outgoing Batches created. The screens shows loads that are sourced from the current node.

You can cancel a load that is in progress by selecting the load and pressing the Cancel button.

The Outgoing Batches screens allows you to monitor data that has been routed.

Outgoing batches are delivered to the target node when the source node pushes or when the target node pulls.

A single push or pull connection is called a synchronization.

For one synchronization, each enabled channel will be processed. Channels are processed in the order defined by the Processing Order setting on the channel with two exceptions:

When outgoing batches are selected for a node and a channel, the maximum number of batches that are extracted per synchronization is controlled by the Max Batch To Send setting on the channel.

There is also a setting that controls the max number of bytes to send in one synchronization. If SymmetricDS has extracted more than the number of bytes configured by the transport.max.bytes.to.sync parameter, then it will finish extracting the current batch and then finish synchronization so the client has a chance to process and acknowledge the "big" batch. This may happen before the configured Max Batch To Send has been reached.

When extracting a batch, data is first extracted to the Staging Area and then sent across the network from the Staging Area. The staging area is used to minimize the amount of time a database connection is being used when streaming over slower networks. The use of the staging area can be turned off by setting the stream.to.file.enabled parameter.

The Incoming Batches screens allows you to monitor batches that have been delivered to the target node.

Incoming batches are delivered to the target node when the source node pushes or when the target node pulls.

Incoming batches are written to the Staging Area first and then loaded. The use of the staging area can be turned off by setting the stream.to.file.enabled parameter.

SymmetricDS creates temporary extraction and data load files with the CSV payload of a synchronization when the value of the stream.to.file.threshold.bytes SymmetricDS property has been reached. Before reaching the threshold, files are streamed to/from memory. The default threshold value is 0 bytes. This feature may be turned off by setting the stream.to.file.enabled property to false.

SymmetricDS creates these temporary files in the directory specified by the java.io.tmpdir Java System property.

The location of the temporary directory may be changed by setting the Java System property passed into the Java program at startup. For example,

The Staging Area screen shows files that are currently in the staging area. Staging area can be cleared at any time by clicking the Clear Staging button.

The Processes screen shows what is actively going on in SymmetricDS for the current node.

Both the Pull Job and the File Sync Pull Job can be configured to pull multiple nodes in parallel. In order to take advantage of this the pull.thread.per.server.count or file.pull.thread.per.server.count should be adjusted (from the default value of 1) to the number to the number of concurrent pulls you want to occur per period on each SymmetricDS instance.

This screen shows pull activity for both table and file synchronization.

Pull activity is recorded in the NODE_COMMUNICATION table. This table is also used as a semaphore to lock pull activity across multiple servers in a cluster.

Both the Push Job and the File Sync Push Job can be configured to push multiple nodes in parallel. In order to take advantage of this the push.thread.per.server.count or file.push.thread.per.server.count should be adjusted (from the default value of 1) to the number to the number of concurrent pushes you want to occur per period on each SymmetricDS instance.

This screen shows pull activity for both table and file synchronization.

Push activity is recorded in the NODE_COMMUNICATION table. This table is also used as a semaphore to lock push activity across multiple servers in a cluster.

The JVM Properties screen is a convenience screen that shows JVM System Properties.

The JVM Threads Screen shows active threads in the JVM.

The Monitors screen allows you to view events of system problems recorded by both local and remote nodes. The list of events can be filtered to limit the number of events displayed, event type, severity level, and node ID. Filtering by severity level will match the level you choose and any level above it. Events are listed in descending order by event time, but the order can be changed by clicking column headings. The remove button will clear the event from the table on the current node. Events are purged automatically each night using the purge.retention.minutes parameter to remove by event time.

When a Monitor is configured, it is run periodically to check the current value of a system metric and compare it to a threshold value. Different monitor types can check the CPU usage, disk usage, memory usage, batch errors, outstanding batches, unrouted data, and number of data gaps. Custom monitor types can be created using Extensions that use the IMonitorType interface. When the value returned from the check meets or exceeds the threshold value, a MONITOR_EVENT is recorded. The MONITOR_EVENT table is synchronized on the heartbeat channel, which allows a central server to see events from remote nodes, but this behavior can be disabled by setting the monitor.events.capture.enabled parameter to false.

To be immediately notified of a monitor event, use Notifications to match on the severity level. Different notification type can send a message by writing to the log or sending an email. Custom notification types can be created using Extensions that use the INotificationType interface. In order to send email, the Mail Server should be configured.

The standalone SymmetricDS installation uses Log4J for logging. The configuration file is conf/log4j.xml. The log4j.xml file has hints as to what logging can be enabled for useful, finer-grained logging.

There is a command line option to turn on preconfigured debugging levels. When the --debug option is used the conf/debug-log4j.xml is used instead of log4j.xml.

SymmetricDS proxies all of its logging through SLF4J. When deploying to an application server or if Log4J is not being leveraged, then the general rules for for SLF4J logging apply.

You can use the logging screen to dynamically adjust the log level in order to help triage issues. When developing transforms it is useful to turn on Log Sync Details.

The db.user and db.password properties will accept encrypted text, which protects against casual observation. The text is prefixed with enc: to indicate that it is encrypted. To encrypt text, use the following command:

or

The text is encrypted using a secret key named "sym.secret" that is retrieved from a keystore file. By default, the keystore is located in security/keystore. The location and filename of the keystore can be overridden by setting the "sym.keystore.file" system property. If the secret key is not found, the system will generate and install a secret key for use with Triple DES cipher.

Generate a new secret key for encryption using the keytool command that comes with the JRE. If there is an existing key in the keystore, first remove it:

Then generate a secret key, specifying a cipher algorithm and key size. Commonly used algorithms that are supported include aes, blowfish, desede, and rc4.

If using an alternative provider, place the provider JAR file in the SymmetricDS lib folder. The provider class name should be installed in the JRE security properties or specified on the command line. To install in the JRE, edit the JRE lib/security/java.security file and set a security.provider.i property for the provider class name. Or, the provider can be specified on the command line instead. Both keytool and sym accept command line arguments for the provider class name. For example, using the Bouncy Castle provider, the command line options would look like:

To customize the encryption, write a Java class that implements the ISecurityService or extends the default SecurityService, and place the class on the classpath in either lib or web/WEB-INF/lib folders. Then, in the symmetric.properties specify your class name for the security service.

Remember to specify your properties file when encrypting passwords, so it will use your custom ISecurityService.

By specifying the "https" protocol for a URL, SymmetricDS will communicate over Secure Sockets Layer (SSL) for an encrypted transport. The following properties need to be set with "https" in the URL:

For incoming HTTPS connections, SymmetricDS depends on the webserver where it is deployed, so the webserver must be configured for HTTPS. As a standalone deployment, the "sym" launcher command provides options for enabling HTTPS support.

Monitoring and administrative operations can be performed using Java Management Extensions (JMX). SymmetricDS uses MX4J to expose JMX attributes and operations that can be accessed from the built-in web console, Java’s jconsole, or an application server. By default, the web management console can be opened from the following address:

In order to use jconsole, you must enable JMX remote management in the JVM. You can edit the startup scripts to set the following system parameters.

More details about enabling JMX for JConsole can be found here.

Using the Java jconsole command, SymmetricDS is listed as a local process named SymmetricLauncher. In jconsole, SymmetricDS appears under the MBeans tab under the name defined by the engine.name property. The default value is SymmetricDS.

The management interfaces under SymmetricDS are organized as follows:

With the proper configuration SymmetricDS can publish XML messages of captured data changes to JMS during routing or transactionally while data loading synchronized data into a target database. The following explains how to publish to JMS during synchronization to the target database.

The XmlPublisherDatabaseWriterFilter is a IDatabaseWriterFilter that may be configured to publish specific tables as an XML message to a JMS provider. See Extensions for information on how to configure an extension point. If the publish to JMS fails, the batch will be marked in error, the loaded data for the batch will be rolled back and the batch will be retried during the next synchronization run.

The following is an example extension point configuration that will publish four tables in XML with a root tag of 'sale'. Each XML message will be grouped by the batch and the column names identified by the groupByColumnNames property which have the same values.

The publisher property on the XmlPublisherDatabaseWriterFilter takes an interface of type IPublisher. The implementation demonstrated here is an implementation that publishes to JMS using Spring’s JMS template. Other implementations of IPublisher could easily publish the XML to other targets like an HTTP server, the file system or secure copy it to another server.

The above configuration will publish XML similar to the following:

To publish JMS messages during routing the same pattern is valid, with the exception that the extension point would be the XmlPublisherDataRouter and the router would be configured by setting the router_type of a ROUTER to the Spring bean name of the registered extension point. Of course, the router would need to be linked through TRIGGER_ROUTERs to each TRIGGER table that needs published.

SymmetricDS not only supports the synchronization of database tables, but it also supports the synchronization of files and folders from one node to another.

SymmetricDS has a pluggable architecture that can be extended. A Java class that implements the appropriate extension point interface, can implement custom logic and change the behavior of SymmetricDS to suit special needs. All supported extension points extend the IExtensionPoint interface. The available extension points are documented in the following sections.

When SymmetricDS starts up, the ExtensionPointManager searches a Spring Framework context for classes that implement the IExtensionPoint interface, then creates and registers the class with the appropriate SymmetricDS component.

Extensions should be configured in the conf/symmetric-extensions.xml file as Spring beans. The jar file that contains the extension should be placed in the web/WEB-INF/lib directory.

If an extension point needs access to SymmetricDS services or needs to connect to the database it may implement the ISymmetricEngineAware interface in order to get a handle to the ISymmetricEngine.

The INodeGroupExtensionPoint interface may be optionally implemented to indicate that a registered extension point should only be registered with specific node groups.

SymmetricDS has its web-enabled, fault-tolerant, database synchronization software available on the Android mobile computing platform. The Android client follows all of the same concepts and brings to Android all of the same core SymmetricDS features as the full-featured, Java-based SymmetricDS client. The Android client is a little bit different in that it is not a stand-alone application, but is designed to be referenced as a library to run in-process with an Android application requiring synchronization for its SQLite database.

By using SymmetricDS, mobile application development is simplified, in that the mobile application developer can now focus solely on interacting with their local SQLite database. SymmetricDS takes care of capturing and moving data changes to and from a centralized database when the network is available

The same core libraries that are used for the SymmetricDS server are also used for Android. SymmetricDS’s overall footprint is reduced by eliminating a number of external dependencies in order to fit better on an Android device. The database access layer is abstracted so that the Android specific database access layer could be used. This allows SymmetricDS to be efficient in accessing the SQLite database on the Android device.

In order to convey how to use the SymmetricDS Android libraries, the example below will show how to integrate SymmetricDS into the NotePad sample application that comes with the Android ADK.

The NotePad sample application is a very simple task list application that persists notes to a SQLite database table called Notes. Eclipse 3.7.2 and Android ADK 20.0.3 were used for this example. The NOTES table definition is below.

Create the NotePad project. You do this by adding a new Android Sample Project. Select the NotePad project.

SymmetricDS for Android comes as a zip file of Java archives (jar files) that are required by the SymmetricDS client at runtime. This zip file (symmetric-ds-3.4.7-android.zip) can be downloaded from the SymmetricDS.org website. The first step to using SymmetricDS in an Android application is to unzip the jar files into a location where the project will recognize them. The latest Android SDK and the Eclipse ADK requires that these jar files be put into a libs directory under the Android application project.

Unzip the symmetric-ds-x.x.x-android.zip file to the NotePad project directory. Refresh the NotePad project in Eclipse. You should end up with a libs directory that is automatically added to the Android Dependencies.

The Android version of the SymmetricDS engine is a Java class that can be instantiated directly or wired into an application via a provided Android service. Whether you are using the service or the engine directly you need to provide a few required startup parameters to the engine:

In order to integrate SymmetricDS into the NotePad application, the Android-specific SymmetricService will be used, and we need to tell the Android application this by adding the service to the AndroidManifest.xml file. Add the following snipped to the Manifest as the last entry under the <application> tag.

The other change required in the Manifest is to give the application permission to use the Internet. Add this as the first entry in the AndroidManifest.xml right before the <application> tag.

The only additional change needed is the call to start the service in the application. The service needs to be started manually because we need to give the application a chance to provide configuration information to the service.

In NotePadProvider.java add the following code snippet in the onCreate method.

This code snippet shows how the SQLiteOpenHelper is shared. The application’s SQLiteOpenHelper is registered in a static registry provided by the SymmetricDS Android library. When the service is started, the key used to store the helper is passed to the service so that the service may pull the helper back out of the registry.

The various parameters needed by SymmetricDS are being set in the Intent which will be used by the SymmetricService to start the engine.

Next, set up an Android Emulator. This can be done by opening the Android Virtual Device Manager. Click New and follow the steps. The higher the Emulator’s API, the better.

Run your NotePad project by pressing Run on NotePadProvider.java in Eclipse. When prompted, select the emulator you just created. Monitor the Console in Eclipse. Let the NotePad.apk install on the emulator. Now watch the LogCat and wait as it attempts to register with your SymmetricDS Master Node.

A minimal implementation of the SymmetricDS client is written in C, which includes a shared library named "libsymclient" and a command line executable named "sym" for synchronizing a database. It currently only supports the SQLite database. The SymmetricDS C library and client are built from the following projects:

The binaries are built using Eclipse CDT (C/C++ Development Tooling), which is an Integrated Developer Environment based on the Eclipse platform. A distribution of Eclipse CDT can be downloaded or an existing Eclipse installation can be updated to include the CDT. (See https://eclipse.org/cdt/ for information and downloads.) In the future, the projects above will switch to a general build system like Autotools for automating builds, but for now Eclipse is required.

The "sym" executable can be run from the command line and expects the "libsymclient.so" library to be installed on the system. If running from the project directories during development, the path to the library can be specified with the LD_LIBRARY_PATH environment variable on Linux, the DYLD_LIBRARY_PATH on Mac OS X, or PATH on Windows. The executable will look for a "symmetric.properties" file containing startup parameters in the user’s home directory or in the current directory:

It will also accept an argument of the path and filename of the properties file to use:

The client uses Startup Parameters to connect to a database, identify itself, and register with a server to request synchronization. Here is an example client.properties file:

The symmetric-client-native project is an example of how to use the SymEngine API provided by the C library. The C library uses an object-oriented pattern and follows the same naming conventions as the Java project. All symbol names in the C library are prefixed with "Sym". Each Java class is represented in C with a struct that contains member data and pointers to member functions. Here is an example C program that runs the SymmetricDS engine:

Use SymmetricDS to capture changes in your database and publish the changes to a Kafka message queue.

This table represents a category of data that can be synchronized independently of other channels. Channels allow control over the type of data flowing and prevents one type of synchronization from contending with another.

Defines how conflicts in row data should be handled during the load process.

Context variables used by runtime services on a single node

The captured data change that occurred to a row in the database. Entries in data are created by database triggers.

Each row represents the mapping between a data change that was captured and the batch that contains it. Entries in data_event are created as part of the routing process.

Used only when routing.data.reader.type is set to 'gap.' Table that tracks gaps in the data table so that they may be processed efficiently, if data shows up. Gaps can show up in the data table if a database transaction is rolled back.

Dynamic extensions stored in the database that plug-in to the running engine and receive callbacks according to their interface.