Copyright © 2018 JumpMind, Inc
Version 3.3.0
Permission to use, copy, modify, and distribute this Metl User Guide for any purpose and without fee is hereby granted in perpetuity, provided that the above copyright notice and this paragraph appear in all copies.
Preface
This user guide introduces Metl, a simple, web-based integration platform that allows for several different styles of data integration including file based Extract/Transform/Load (ETL), messaging, and remote procedure invocation via Web Services. This guide is intended for users, developers, and administrators who want to install the software, configure integrations, and manage its operation. Thank you to all the members of the open source community whose feedback and contributions helped us build better software and documentation. This version of the guide was generated on 2018-11-08.
1. Introduction
Metl is a simple, web-based integration platform that allows for several different styles of data integration including messaging, file based Extract/Transform/Load (ETL), and remote procedure invocation via Web Services. Metl was built to solve fairly simple day to day integration tasks without the need for custom coding, heavy infrastructure, or high costs. It can be deployed in the cloud or in an internal data center, and was built to allow developers to extend it to fit their needs by writing their own components that can be included and leveraged by the existing Metl infrastructure.
1.1. Overview
Metl is a web application. It can be deployed as a web application archive (war) file to an application server (i.e. Tomcat, JBoss, Websphere) or can be deployed stand-alone as a war file deployed under a bundled Jetty Servlet container. See installation section for details.
Once installed, the web application is accessed via a web browser and is used to design, deploy and manage integrations.
Integrations can be grouped or packaged into one or more projects. Projects are a means to logically group integrations together in some logical way. As an example, a project might contain all of the integrations in or out of a specific system. See projects for additional details.
Projects contain Flows, Models and Resources.
Flows are the integrations themselves. That is, flows allow the definition of data be retrieved from source systems, transformed, and written to target systems. Flows are graphical in nature and allow the developer to configure data movement.
Flows are constructed of components connected by links that describe the path of data through the flow. There are various components available, and custom components can be built and integrated in to the existing Metl infrastructure.
See flows for additional details.
Models provide structure to your data. When dealing with structured data, the data can be defined by modeling. Metl Models allow you to describe your data. Models can either be relational or hierarchical in nature. See models for additional details.
Resources represent connections to physical endpoints where data is read from or written to. The following resource types are available in Metl:
-
Database - A JDBC connection to a JDBC compliant relational database
-
Directories - Connections to file systems. Local, FTP, SFTP and SMB are all supported
-
HTTP Resource - An HTTP connection to REST or SOAP based services
-
Mail Session - An SMTP connection that can be used to send and receive email
See resources for additional details.
1.2. Use Cases
-
Flat File Data Integrations
-
RDBMS Data Integrations
-
Pub / Sub (Queue based) Data Integrations
-
API / Service Based Data Integrations
1.3. Why Metl?
-
Integrations can be configured in hours vs coding them in days or weeks
-
Metl is easy to install. Simply download the .war file and run
-
Metl is web based. No tools to install on each desktop. Deploy locally, in your data center or in the cloud
-
Metl doesn’t require heavy infrastructure. Any windows or linux box will do
-
Metl is open. No black box. Download the source. Write your own components if desired
-
Metl is proven in large integration scenarios
-
Metl is free. The entire toolset is open source under the GPL license
-
Metl is supported. JumpMind, Inc. provides professional support and services for any need
1.4. System Requirements
Metl is written in Java and requires a Java Runtime Environment (JRE) Standard Edition (SE) or Java Development Kit (JDK) Standard Edition (SE). Most major operating systems and databases are supported. The minimum operating system requirements are:
-
Java SE - Runtime Environment or Development Kit version 8 or above
-
Memory - 1 (GB) available
-
Disk - 256 (MB) available
Metl is accessed from a web console, which requires one of the following supported web browsers:
-
Google Chrome 23 or newer
-
Internet Explorer 8 or newer
-
Mozilla Firefox 17 or newer
-
Safari 6 or newer
1.5. License
METL is free software licensed under the GNU General Public License (GPL) version 3.0. See http://www.gnu.org/licenses/gpl.html for the full text of the license. This project includes software developed by JumpMind (http://www.jumpmind.com/) and a community of multiple contributors. METL is licensed to JumpMind as the copyright holder under one or more Contributor License Agreements. METL and the METL logos are trademarks of JumpMind.
2. Installation
Metl is provided as a web application archive (war) file. The .war file can be deployed in two different ways.
-
Deployed stand-alone using Metl’s embedded Jetty server
-
Deployed to an existing servlet container (i.e. Tomcat, JBoss, Websphere, Oracle Application Server, etc.)
Deployed stand-alone using Metl’s embedded Jetty server
The simplest way to run Metl is to run it stand-alone using the internal Jetty container. In order to run Metl stand-alone, simply run the following from a command line.
| A Java 8 run-time environment must be installed and in the path to run java. |
java -jar metl.war
In the simplest form shown above, Metl will choose a configuration direction based on the logged in user’s home directory (i.e. /users/myusername/.metl) for linux based systems and (c:/users/myusername/.metl) on windows based systems. In this form metl will use a file-based H2 database for for the Metl data store.
Deployed to an existing servlet container
In order to deploy Metl to an existing servlet container, simply copy the metl.war file to the deploy directory of the application server.
Specifying the Metl configuration directory
The default configuration directory can be set by specifying the system property org.jumpmind.metl.ui.init.config.dir.
java -Dorg.jumpmind.metl.ui.init.config.dir=/opt/metl -jar metl.war
Application properties file (metl.properties)
Metl will create a default properties file in its configuration directory if one does not exist. The property file is named metl.properties and has the following content.
db.driver=org.h2.Driver db.url=jdbc:h2:./metl-dev execution.retention.time.ms=3600000 execution.retention.time.ms.cancelled=60000 log.to.console.enabled=true log.to.file.enabled=true table.prefix=METL #log.file=
Specifying an alternate database resource for metl configuration and run-time data is as simple as specifying a jdbc driver and url in the metl.properties file.
db.driver=net.sourceforge.jtds.jdbc.Driver db.url=jdbc:jtds:sqlserver://myserver:1433/mydatabase db.user=myuserid db.password=mypassword
By default, Metl uses a DBCP connection pool. In addition to those above, all DBCP configuration settings can be set in the metl.properties file including:
db.validation.query= db.pool.initial.size= db.pool.max.active= db.pool.max.idle= db.pool.min.idle= db.pool.max.wait.millis= db.pool.min.evictable.idle.millis= db.test.on.borrow= db.test.on.return= db.test.while.idle=
Installing Metl as a service
Metl can be installed as a linux/unix or windows service. To install Metl as a service, run the following from a command line.
java -Dorg.jumpmind.metl.ui.init.config.dir=/opt/metl -jar metl.war install
By default, Metl will be installed as a service with service name metl. A service configuration file named metl_service.conf will be written to the configuration directory and can be used to further configure the Metl service.
Starting the Metl service
To start the Metl service, run the following from a command line
java -Dorg.jumpmind.metl.ui.init.config.dir=/opt/metl -jar metl.war start
or, use the operating system tools to start and stop the service. On linux:
service metl start
For windows, use the local services control panel to start the Metl service.
Stopping the Metl service
To stop the Metl service, run the following from a command line
java -Dorg.jumpmind.metl.ui.init.config.dir=/opt/metl -jar metl.war stop
or, use the operating system tools to start and stop the service. On linux:
service metl stop
For windows, use the local services control panel to stop the Metl service.
Uninstalling Metl as a service
To uninstall Metl as a service, run the following from a command line.
java -Dorg.jumpmind.metl.ui.init.config.dir=/opt/metl -jar metl.war uninstall
Once Metl has been started, it can be accessed via a web browser as follows:
http://myservername:myport/metl
If Metl is running on the default port (42000) on the local machine, the url would be as follows:
http://localhost:42000/metl
3. The Metl User Interface
Accessing the Metl User Interface
Once Metl has been started, it can be accessed via a web browser as follows:
http://myservername:myport/metl
If Metl is running on the default port (42000) on the local machine, the url would be as follows:
http://localhost:42000/metl
The following home screen will be displayed:
Navigation Menu
The navigation menu shown below is the mechanism for traversing the system.
The Metl UI is split into five major areas:
-
Design - Design and Develop Integrations
-
Deploy - Deploy integrations to a Metl run time agent
-
Manage - Manage integrations that have been run on a given agent including looking at past executions, etc.
-
Explore - Explore resources that are configured for this Metl instance
-
Admin - Administrative management of Metl including defining system users, managing logging levels, etc.
In addition to these five major navigation points, the navigation menu also allows for:
-
Help - Access this help documentation
-
Logout - Log the current user out of the Metl UI
4. Design
The Design screens are used to design and develop integrations. The main design screen is shown below.
The design screen is split into two main panels, the navigation pane and the content/editor pane. The navigation pane shown below allows for traversing projects, flows, models and resources.
When Metl is first installed, it comes installed with a sample project that includes sample flows, models and resources. These samples allow the user to immediately see Metl in action. In addition, these sample flows are referenced in the component documentation in order to better understand how individual components work.
The content/editor pane is where flows, resources and models are edited. The screenshot below shows a database resource being edited.
4.1. Projects
Projects are a means to group integrations together in some logical way. Typically, the boundaries of a project should equate to a deployable unit. That means the scope or context of the project should be those flows that must be bundled together when deployed. I.E. if two flows depend on each other and are so tightly coupled that changes to one dictate changes in the other, then those flows should be bundled together in a single project. The project is the level at which flows and their associated resources and models are versioned (see Versions) and imported/exported (see Import/Export).
A new project can be added via the menu.
4.1.1. Versions
Projects can have multiple versions. When a project is first created an initial version is also created with name master. After the master project is released (deployed to production), future changes to the project can be completed in a new version such that each production migration of the project can be tracked. By making each production release of a project under a new project version visibility to differences between versions is also available. New versions of a project can be created from any subsequent version. I.E. if you have versions 1.0.0, 1.1.0, and 1.2.0, a new version can be created from ANY of the previous 3 versions. A patch release (1.1.1) version can be created from the 1.1.0 version while simultaneously making changes to the 1.2.0 version for some future release date. This is very similar to branching and merging concepts that are available in most source control systems.
To create a new version of a project from an existing version, first select the source version from which the new version should be created and then click File New Project Version as shown below.
4.1.2. Project Version Dependencies
One project version can be dependent on another project version. A project dependency can be added with File New Project Dependency. When a project is dependent on another it gains access to Resources, Models and Flows in the child project. A good use case for adding Project Dependencies is for sharing common resources.
4.1.3. Import/Export
Project configuration can be exported and imported. Use File Export… and File Import…. The export is to a JSON file. Import and Export are used to migrate from one environment to another or to backup your configuration.
Also note that a subset of your project can be exported if need be.
4.2. Flows
An integration is called a flow. A flow defines the input to be read, the processing rules to be applied, and the output to be written.
Flows are a graphical set of instructions for how data is integrated (flows) from a source system to a target system.
4.3. Components
Flows consist of a set of components (readers, processors or writers) connected together with component links.
A typical component (the rdbms-reader) component is shown below.
Each component has an input port and an output port (blue circles in the image above). The input and output ports are used to connect the component to other components with component links. Each port has a designation that dictates the type of message that can be sent into the component and the type of message that the component generates. The designations are:
-
E - A model based message
-
T - A text based message
-
B - A binary based message
-
* - Any message type
Each component completes some action when an input message is received and sends one or more resultant messages when it is complete. The RDBMS-Reader as an example takes any input message type and produces model based message types. When an input message is received, the RDBMS-Reader executes a configured sql script and returns the resultant rows from execution of the script out as model based messages.
The following shows a simple flow that reads a table from a database and writes it to three different flat files, a delimited file, a fixed length file and an xml file.
In the flow above, each component completes a specific task as follows:
| Component | Type | Task |
|---|---|---|
Setup Person Database |
Sql Executor |
Executes a sql sript to set up the database and tables that will be used in the sample flow |
Person Table Reader |
RDBMS Reader |
Executes a sql statement to read the person table from the database created by the Setup Person Database component |
Delimited Formatter |
Format Delimited |
Formats the person data sent from the Person table reader in a delimited format |
Delimited File Writer |
Text File Writer |
Writes the delimited formatted data to a text file |
Fixed Length Formatter |
Format Fixed |
Formats the person data sent from the Person table reader in a fixed length format |
Fixed Length Writer |
Text File Writer |
Writes the delimited formatted data to a text file |
XML Formatter |
Format XML |
Formats the person data sent from the Person table reader in an xml format |
XML File Writer |
Text File Writer |
Writes the delimited formatted data to a text file |
When a flow executes, all components in the flow are initialized in parallel and begin waiting for messages to process. Any component with NO inbound component links receives a Control Message from the Metl framework telling it to perform its action.
The outbound messages from those components started by the Metl framework are fed to downstream components, and the flow runs until completion. In the example above, when the flow is run, all components are initialized and wait for inbound messages.
The Setup Person Database component receives a startup message, executes its sql script to create the initial database used in the sample flow, and then sends an output message to the Person Table Reader component which runs its sql statement to select the person records from the database. The output of the RDMBS reader is messages that contain rows which are sent to the Formatters, and so forth down the chain until all components are complete, and then the flow ends.
4.4. Messages
Data flows between components over the component links in a series of messages that are generated and consumed by each component. There are three different types of messages.
-
Model based message - A message that contains records in model format. I.E. data parsed and stored according to a given Metl Model
-
Text based message - A message that contains text based records. I.E. data in string format
-
Binary based message - A message that contains binary based data. I.E. A binary object
The difference in the message types, is the type of payload that is carried by each. A model based message has a payload that contains an array of model based records. Each model based record has attributes and values. Each value is tied to a specific attribute in the model.
A text based message has a payload that contains an array of strings (text data), and a binary based message has a payload of binary data.
4.4.1. Unit of Work
Several components can deal with multiple units of work in the execution of a given flow.
As an example, the Text File Reader can take one or more inbound messages that specify the path and name of a file to read. In this instance, the text file reader receives three messages, and will read data from the three files specified in each input message. When the text file reader completes reading of a given file and sends all data in output messages, it will send a control message that indicates it is complete with the given "unit of work" (i.e. reading the file that was specified by the input message).
Downstream components can utilize that control message in different ways depending on the implementation and configuration of the downstream component. As an example, many components have a "Run When" configuration option which can be set to "Per Message" or "Per Unit of Work".
Setting the "Run When" option to "Per Unit of Work" will enable the component to run when a unit of work control message is received".
Setting the "Run When" option to "Per Message" will enable the component to run each time any data message is received.
4.5. Components
The following table shows a list of available components.
| Icon | Name | Use When |
|---|---|---|
Readers |
||
|
Data needs read from a binary file |
|
|
Data needs read from an Excel file |
|
|
A file based resource needs to be checked for the existence of a file or set of files |
|
|
Data needs read from a relational database |
|
|
A text constant is needed in message content in the flow |
|
|
Data needs read from a text file |
|
|
A compressed zip file needs to be extracted in order to process its content |
|
|
An XML file needs to be read and element content of that file sent as individual messages |
|
Processors |
||
|
Check for an assertion |
|
|
A flow needs to be embedded in another flow |
|
|
Content within a message needs to be routed to different downstream components in the existing flow |
|
|
Input model based records must be deduplicated before being sent to downstream components |
|
|
A delay is needed when sending data between components |
|
|
A component that will parse XML or JSON to a Model |
|
|
There is a need to execute an operating system command as part of a flow |
|
|
Files need to be renamed, copied, moved or deleted |
|
|
Model based data needs to be formatted into delimited text |
|
|
Model based data needs to be formatted into fixed length text |
|
|
Model based data needs to be formatted into xml text |
|
|
Reference data exists that needs to be looked up / retrieved by a specified key, and another attribute of the the resulting reference data passed to downstream components. |
|
|
Data needs to be mapped from one set of entities/attributes (model structure) to another set of entities/attributes |
|
|
Data needs to be mapped from a relational model to a hierarchical model |
|
|
Different instances of model based records need to be merged into a single instance that combines the attributes from all instances. |
|
|
Messages need to be logged to the Metl console and log file |
|
|
A set of model based data needs compared to another set of model based data in order to generate insert/update/delete instructions for a writer component |
|
|
Inbound messages need to be multiplied (duplicated n number of times) and the sent to downstream components. |
|
|
A message needs to be passed through to another component without any action being taken on the message |
|
|
An inbound text message in some delimited text format needs to be parsed into a model based message structure |
|
|
An inbound text message in a fixed length text format needs to be parsed into a model based message structure |
|
|
An inbound text message in xml text format needs to be parsed into a model based message structure |
|
|
A custom action should be performed that can be written using Groovy |
|
|
A component that will format an entity to XML or JSON |
|
|
A sequence number needs to be created for a given attribute in a model based record, and the starting value of the sequence can be derived by an sql statement. |
|
|
Data within a model based message needs to be sorted by one of the attributes within the model |
|
|
Entity message data needs to be staged and queried using SQL. |
|
|
A piece of text in a text based message needs replaced with alternate text |
|
|
An attribute in a model based message needs transformed in one way or another (trim, substring, constants, etc.) |
|
|
Inbound messages need to be combined into a single message and then sent to downstream components. |
|
|
A web service (either REST or WSDL) needs to be called to perform an action such as retrieving or writing data |
|
|
XSLT needs applied to a set of input data in order to transform it into a given output format |
|
Writers |
||
|
Data needs written to a binary file |
|
|
An email needs to be sent |
|
|
Data needs written to an Excel file |
|
|
Data needs written to a relational database |
|
|
Data needs written to or deleted from a database using a specific sql statement |
|
|
Data needs written to a text based file |
|
|
Data needs to be put in a compressed file |
|
Services |
||
|
A RESTful service needs to be created and hosted by the Metl server |
|
|
A RESTful service needs to be created and hosted by the Metl server |
|
|
A component that registers with a pub/sub system (like JMS). Works with resources that support subscriptions |
|
Controls |
||
|
Flow needs controlled by one input before remaining process continues |
|
|
Flow needs to be held at this point until all inputs complete before continuing |
|
4.5.1. Readers
Binary File Reader
Icon |
|
|---|---|
Use When |
Data needs read from a binary file |
Samples |
|
Description |
The Binary File Reader is used to read data from one or more binary based files. The Binary File Reader can read one or more files and pass the data read to downstream components. The file name(s) and path(s) of the file(s) to be read can be specified in the File Path parameter below, or can be specified in the payload of the inbound message. |
Inbound Message Type |
Text Based Message |
Output Message Type |
Binary Based Message |
Properties
| Name | Description |
|---|---|
Resource |
The file based resource (either local or remote) on which the file to be read resides. |
Get File Name From Message |
If selected, the path and file name to be read will be determined by the payload of the incoming message. This option is used frequently with the File Poller component. If this setting is not selected, the path and file to be read is based on the File Path property below. |
Must Exist |
Whether the file must exist |
File Path |
The relative path and file of the file to be read. The relative path and file are concatenated with the absolute path specified in the File based resource. |
Size / Message (MB) |
TODO: implement if the file is bigger than the allowable message size |
Send Control Message on EOF |
Whether to send a control message at EOF for each file read |
Excel File Reader
Icon |
|
|---|---|
Use When |
Data needs read from an Excel file |
Samples |
|
Description |
The Excel File Reader is used to read data from one or more Excel based files. The Excel File Reader can read one or more files and pass the data read to downstream components. The file name(s) and path(s) of the file(s) to be read can be specified in the File Path parameter below, or can be specified in the payload of the inbound message. Excel files that are read are parsed as defined within the component editor. |
Inbound Message Type |
Text Based Message |
Output Message Type |
Entity Based Message |
Properties
| Name | Description |
|---|---|
Resource |
The file based resource (either local or remote) on which the file to be read resides. |
Get File Name From Message |
If selected, the path and file name to be read will be determined by the payload of the incoming message. This option is used frequently with the File Poller component. If this setting is not selected, the path and file to be read is based on the File Path property below. |
Must Exist |
Whether the file to read must exist for the flow to execute. |
Ignore on Formula Error |
Whether to substitute null values for cells containing an error. |
File Path |
The relative path and file of the file to be read. The relative path and file are concatenated with the absolute path specified in the File based resource. |
Records in the Excel file reader are defined by a row of data. |
|
Header Lines to Skip |
The number of header lines in the file that should be skipped and not sent to downstream components |
Send Control Message on EOF |
Whether to send a control message after receiving an EOF |
| Specifying None on the Action on Error does NOT stop the flow from failing when a file cannot be read. It simply specifies what should be done with the file when the error condition does occur. |
Component Editor
Double clicking on the Excel File Reader component in the flow will result in the Excel File Reader editor being displayed as shown below.
The editor displays a row for every entity and attribute of the output model, and allows setting the Sheet:Column of each.
The syntax is: Sheet Name:Column Letter
Sheet Name is the exact name as it appears on the tab to be imported.
Column Letter is the corresponding column the desired data exists in.
File Poller
Icon |
|
|---|---|
Use When |
A file based resource needs to be checked for the existence of a file or set of files |
Samples |
|
Description |
The File Poller is used to look at a file based directory structure and send a message to downstream components if a file or files match a given file pattern exists. The file poller sends a payload of an array list of strings with the file names of each file found that matches the File Poller’s File Pattern parameter. The File pattern parameter can be a single pattern or a list of patterns separated by commas. Wildcards can be used in any pattern. |
Inbound Message Type |
None |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Resource |
The File based resource on which the File Poller should operate |
Get File Pattern From Message |
Get the file pattern from an inbound text message versus the File Pattern parameter below. |
File Pattern |
The File Pattern specifies the pattern(s) of files to look for. Multiple patterns can be specified in a comma delimited list. Wildcards (*) can be used. See examples below. |
File Sort Order |
Indicates the order files will be reported. |
Sort Descending |
Order the sort in descending order. |
Files/Msg |
The number of file paths that should be published per message. The default value is 1. |
Min Files to Poll |
Files will not be reported unless at least this number of files are found |
Max Files to Poll |
No more than this number of files will be reported during on execution |
Use Trigger File |
Many systems post a single trigger file to designate that other data files are ready to process. The existence of the trigger file indicates the other files are ready to process. If "Use Trigger File" is true, the File Poller will first check for the existence of a file named the same as the Relative Trigger File Path parameter. If that file exists, the File Poller will then check for files matching the File Pattern parameter. If the trigger file does not exist, the File Poller will send a shutdown message to any downstream linked components. If the Use Trigger File is set to false, no file triggering mechanism will be used |
Relative Trigger File Path |
The relative path and file name of a trigger file that will be used to indicate the files that match the File Pattern parameter are ready for processing. |
Example 1. Search for all .csv files
*.csv
Example 2. Search for all.csv and all .txt files
*.csv,*.txt
RDBMS Reader
Icon |
|
|---|---|
Use When |
Data needs read from a relational database |
Samples |
Relational Database to Flat File, Normalized Table to Denormalized File, Normalized Table to Denormalized File v2 |
Description |
The RDBMS Reader is used to read data from a sql compliant relational database by specifying a SQL query. The RDBMS Reader can take one or more input messages. The reader will be executed (i.e. the sql query run) based on the 'Run When' option. One time, once for every input message or once for every entity record within an input message that it receives. Any entity/attribute within an input message can be used as a parameter in the SQL parameter by specifying the input model entity and attribute name in the query prefaced with a colon. See Example 1 below. By default, the sql results will be mapped directly to the output model based on the table and column name of the selected fields. When aggregate or other functions are used within the sql statement the table and column name will not be available for mapping to the output model, so hints may be used to map the field directly to an output model entity and attribute. See Example 2 below. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Resource |
The SQL Database Resource on which the SQL query should be run to read data. |
Sql |
The sql query or script that will be executed for this reader. May have one or more statements to execute. |
This component also has the option of PER ENTITY. This means that the component will execute SQL for each entity. |
|
Unit of Work |
When the 'Run When' option is PER UNIT OF WORK this defines when an 'end of transaction' (control message) is forwarded to downstream components. COMPONENT LIFETIME - Send one final control message after the entire SQL content has been completed SQL SCRIPT - Send a control message once per SQL script completed SQL STATEMENT - Send a control message after each SQL statement completed |
Trim Columns |
Whether leading and trailing spaces should be trimmed from character fields after being selected from the database and before they are placed in the output model format |
Match On Column Name |
Whether when matching SQL results to the output model, you match on table and column or column name only |
Pass Input Rows Through |
Whether to forward input messages along with the output SQL results |
Example 1. Using Input Message Entity/Attribute as a parameter
select field1, field2 from mytable where field3=:INPUT_MODEL_ENTITY_NAME.INPUT_MODEL_ATTRIBUTE_NAME
In the example above, the SQL Reader query will be run for each input record within each input message. Each time the query is run, field3 will be restricted by the incoming row’s INPUT_MODEL_ENTITY_NAME.INPUT_MODEL_ATTRIBUTE field.
| All parameters in the sql statement should be prefixed with a colon. Parameters prefixed with a colon will use a prepared statement parameter to execute the query with the parameter. Using the deprecated $(parameter_name) will use string substitution versus a prepared statement parameter and is not recommended. |
Example 2. Using Hints to Map to the Output Model
select field1, field2, count(*) /* OUTPUTMODEL_ENTITY_NAME.OUTPUT_MODEL_ATTRIBUTE_NAME */ from mytable
In the example above, the results from the query execution will be mapped to:
-
field1 ⇒ MYTABLE.FIELD1
-
field2 ⇒ MYTABLE.FIELD2
-
count(*) ⇒ OUTPUTMODEL_ENTITY_NAME.OUTPUT_MODEL_ATTRIBUTE_NAME
| If a single result set / query results from several queries using the union operator, only place the sql column hints on the first query’s columns, not each set of columns in every sql. Also, hints need to directly follow the field they correspond to prior to any comma if other fields follow the hint. |
Text Constant
Icon |
|
|---|---|
Use When |
A text constant is needed in message content in the flow |
Samples |
|
Description |
The Text Constant allows a constant text literal to be placed in the payload of a message and sent to downstream components. |
Inbound Message Type |
Any Message Type |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Text |
The text constant that will be placed into the message payload and sent to downstream systems |
Split On Line Feed |
Whether to generate separate entities based on line feeds in the Text box entries or send as one entity regardless of line feeds |
Send Control Message on Text Send |
If Run When is PER MESSAGE this option will trigger a control message to be sent after each text message sent |
Example 1. Sample Text Constant Text
4,Diane,Prince,F 3,Clark,Kent,M 1,Peter,Parker,M 2,Bruce,Wayne,M
Text File Reader
Icon |
|
|---|---|
Use When |
Data needs read from a text file |
Samples |
Flat File to Relational Database |
Description |
The Text File Reader is used to read data from one or more text based files. The Text File Reader can read one or more files and pass the data read to downstream components. The file name(s) and path(s) of the file(s) to be read can be specified in the File Path parameter below, or can be specified in the payload of the inbound message. Text files that are read can be in any format (fixed length, comma delimited, etc.) and can be parsed by downstream processing components like the Parse Delimited and Parse Fixed components. |
Inbound Message Type |
Text Based Message |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Resource |
The file based resource (either local or remote) on which the file to be read resides. |
Get File Name From Message |
If selected, the path and file name to be read will be determined by the payload of the incoming message. This option is used frequently with the File Poller component. If this setting is not selected, the path and file to be read is based on the File Path property below. |
Must Exist |
Whether the file to read must exist for the flow to execute. |
File Path |
The relative path and file of the file to be read. The relative path and file are concatenated with the absolute path specified in the File based resource. |
Records in the text file reader are defined by a line of data up to the files row delimiter (i.e. <CR> or <CR><LF>) |
|
Header Lines to Skip |
The number of header lines in the file that should be skipped and not sent to downstream components |
Send Control Message on EOF |
Whether to send a control message after receiving an EOF |
| Specifying None on the Action on Error does NOT stop the flow from failing when a file cannot be read. It simply specifies what should be done with the file when the error condition does occur. |
Unzip
Icon |
|
|---|---|
Use When |
A compressed zip file needs to be extracted in order to process its content |
Samples |
|
Description |
The Unzip component allows a compressed archive/zip file to be extracted so that its contents can be processed in the flow. |
Inbound Message Type |
Text Based Message |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Zip Source Directory Resource |
The directory where the zip file will be polled at |
Unzip Target Directory Resource |
The target file resource where extracted files will be written |
Target Path |
The path to where the unzipped files will be written |
Create Sub Dir From File Name |
Whether to create a root sub-directory for the extracted files that will match the zip file name |
Extract Empty Files |
Whether to extract empty files |
Overwrite |
Whether to overwrite the target file if it already exists |
Delete Source File |
Whether the source zip file should be deleted after its contents have been extracted |
XML File Reader
Icon |
|
|---|---|
Use When |
An XML file needs to be read and element content of that file sent as individual messages |
Samples |
|
Description |
The XML Reader component allows XML data to be read from a file and data within tags to be sent as individual messages |
Inbound Message Type |
Text Based Message |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Resource |
The file based resource (either local or remote) on which the file to be read resides. |
Get File Name From Message |
If selected, the path and file name to be read will be determined by the payload of the incoming message. This option is used frequently with the File Poller component. If this setting is not selected, the path and file to be read is based on the File Path property below. |
File Path |
The relative path and file name for the file to read unless relative path and file name come from the inbound message |
Tag to find and send, defaults to root |
The tag that encloses data to be sent as individual messages |
Number of read tags to send together |
The number of instances of tags from above to be packed into a single message before sending the message |
4.5.2. Processors
Assert
Icon |
|
|---|---|
Use When |
A flow needs to check for certain assumptions |
Samples |
|
Description |
The Assert component is used to test certain assumptions about the flow. The Assert can check for a specified number of messages by the various types or by a count of records from a defined SQL statement. If any of the checks fail in the Assert it will Fail the flow. |
Inbound Message Type |
Any Message Type |
Output Message Type |
Any Message Type |
Properties
| Name | Description |
|---|---|
Expected Number of Entity Messages |
Number of entity messages expected. |
Expected Number of Entities Per Entity Message |
Number of entities per entity message expected. |
Expected Number of Text Messages |
Number of empty payload messages expected. |
Expected Number of Binary Messages |
Number of binary messages expected. |
Expected Number of Control Messages |
Number of control messages expected. |
Expected Number of Empty Payload Messages |
Number of empty payload messages expected. |
Assert DataSource |
The SQL Database Resource on which the SQL query should be run to read data. |
Assert Sql |
The SQL query used to check for assertion. |
Expected Count from Assert Sql |
Number of records expected returned from the Assert Sql. |
Content Router
Icon |
|
|---|---|
Use When |
Content within a message needs to be routed to different downstream components in the existing flow |
Samples |
Partition By Gender |
Description |
The content router routes content within a message to one or more downstream components based on the routing criteria configured in the component. |
Inbound Message Type |
Any Message Type |
Output Message Type |
Any Message Type |
Properties
| Name | Description |
|---|---|
Only Route First Match |
If checked, the content router will only route an inbound content to the first criteria in the criteria list that evaluates to true, not all criteria in the list that evaluate to true. |
Routing Criteria
Expression
A groovy script expression that returns true or false. The groovy script is executed/evaluated for each record in an inbound message. If the expression evaluates to true, the message is passed to the step at the right. If the expression is evaluates to false, the message is NOT passed to the step at the right.
Target Step
The target step is the connected downstream component that will receive the message if the groovy expression evaluates to true.
| The inbound message will go to ALL target steps where the expression evaluates to true (i.e. a message can go to more than one target step) unless the "Only Route First Match" option is checked. |
Example 1. Sample groovy expressions for an entity based message
if (ENTITY.ATTRIBUTE=='Value 1') return true; if (ENTITY.ATTRIBUTE=='Value 2') return true;
| If an entity based message has multiple entities within a single record, the record will be treated as a whole, not as its individual entities within the record. This means that the entire record (including all of its entities) will be delivered to a downstream component if the groovy expression evaluates to true, and NONE Of the entities will be delivered to the downstream component if the expression evaluates to false. For example, if the Person model contains entities NAME and ADDRESS, and the groovy expression is something like "NAME.FIRST_NAME='TOM'", a record with NAME.FIRST_NAME='TOM' will send both the NAME and ADDRESS entities to the downstream component. A record with NAME.FIRST_NAME='BOB' will send NEITHER the NAME nor ADDRESS entities of the record to the downstream component. |
Example 2. Sample groovy expressions for a text based message
if (text.indexOf("Some text string") > 0) return true;
if (text.indexOf("Some other text string") > 0) return true;
Example 3. Sample groovy expressions for routing based on a message header. This example tests to see if the header variable is available before testing it
binding.variables.containsKey('chainId') && chainId.equals('001')
Example 4. Sample groovy expressions for routing based on the change type of an entity message - whether the entity message is passed to downstream components as in Add, Change or Delete ('ADD', 'CHG', or 'DEL')
CHANGE_TYPE == 'DEL'
Example 5. Sample groovy expressions for whether a specific entity exists in a given message
ENTITY_NAMES.contains("my_entity_name")
| Note that the expression can explicitly return true or false or can simply evaluate to true or false as shown in the examples above. |
Model Data Diff
Icon |
|
|---|---|
Use When |
A set of model based data needs compared to another set of model based data in order to generate insert/update/delete instructions for a writer component |
Samples |
Calculate Differences Only |
Description |
The Model Data Diff component compares two sets of model based data in order to generate insert/update/delete instructions for a writer component in order to get the model based data sets to match. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Source of Old Version |
The source component that is sending data to the data diff component that represents the old or prior data for the compare |
Source of New Version |
The source component that is sending data to the data diff component that represents the new or current data for the compare |
In Memory Compare |
Whether the compare should be done in memory or on disk. If checked, the compare will be done completely in memory |
Deduper
Icon |
|
|---|---|
Use When |
Input model based records must be deduplicated before being sent to downstream components |
Samples |
Remove Duplicates |
Description |
The Deduper removes duplicate records from a message or set of messages in a given unit of work. Once the complete unit of work has been received by the deduper, the unit of work will repackage the records from the inbound message(s) to a set of output messages based on the configured number of records per message, and send those deduped outbound messages to downstream component(s). Records may be deduped by Entity (full record) or Attribute (selected column(s)). |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Dedupe Type |
This is the type of dedupe to execute. Options are ENTITY or ATTRIBUTE. Entity will execute a full record compare and Attribute will only look for duplicate records based on selected attributes defined through the component editor screen. |
Preserve Record |
Applicable only when dedupe type is Attribute. Options are 'First Record' or 'Last Record'. If a duplicate record is found and 'First Record' is chosen then the first matched record is forwarded. If 'Last Record' is chosen, then the final record matching the selected attribute(s) is forwarded and the earlier record(s) are dropped. |
Component Editor
Double clicking on the Deduper component in the flow will result in the Deduper editor being displayed as shown below.
The Deduper editor displays the entities contained within the input model in a list. To selected which attribute to dedupe on, select the Entity the attribute desired belongs to and click the 'Edit Columns' button at the top.
In the window that appears, click the checkbox next to the attribute(s) to use as the dedupe key shown below.
Delay
Icon |
|
|---|---|
Use When |
A delay is needed when sending data between components |
Samples |
|
Description |
The Delay causes messages being sent through the delay component to be delayed/paused for the configured number of seconds before being sent to downstream components. |
Inbound Message Type |
Any Message Type |
Output Message Type |
Any Message Type |
Properties
| Name | Description |
|---|---|
Delay (ms) |
The amount of time in milliseconds that the messages will be delayed before being passed to downstream components |
Deserializer
Icon |
|
|---|---|
Use When |
An entity message needs to be automatically parsed into a model from XML or JSON |
Samples |
If the structure is BY_TABLE, then the expected format will be as follows: [{"name":"PERSON","rows":[{"GENDER":"M","ID":"1","LAST_NAME":"Arbuckle","FIRST_NAME":"Garfield"},{"GENDER":"M","ID":"2","LAST_NAME":"Arbuckle","FIRST_NAME":"Odie"},{"GENDER":"M","ID":"3","LAST_NAME":"Arbuckle","FIRST_NAME":"Jon"},{"GENDER":"F","ID":"4","LAST_NAME":"Wilson","FIRST_NAME":"Liz"},{"GENDER":"F","ID":"5","LAST_NAME":"Cat","FIRST_NAME":"Arlene"}]}] If the structure is BY_INBOUND_ROW, then the expected format will be as follows: [{"name":"PERSON","data":{"GENDER":"M","ID":"1","LAST_NAME":"Arbuckle","FIRST_NAME":"Garfield"}},{"name":"PERSON","data":{"GENDER":"M","ID":"2","LAST_NAME":"Arbuckle","FIRST_NAME":"Odie"}},{"name":"PERSON","data":{"GENDER":"M","ID":"3","LAST_NAME":"Arbuckle","FIRST_NAME":"Jon"}},{"name":"PERSON","data":{"GENDER":"F","ID":"4","LAST_NAME":"Wilson","FIRST_NAME":"Liz"}},{"name":"PERSON","data":{"GENDER":"F","ID":"5","LAST_NAME":"Cat","FIRST_NAME":"Arlene"}}] |
Description |
Parses JSON input into a model entities. |
Outbound Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Format |
Can have values of AUTOMATIC, JSON or XML. Default is AUTOMATIC. If AUTOMATIC is chosen, then the deserializer will attempt to look at the flow parameters to determine what format should be used. If an Http Request started the flow, then the Accept header, the Content-Type header and the format parameter will all be inspected in order to determine the type. |
Structure |
Options for how to format the document. Options are: BY_TABLE and BY_INBOUND_ROW |
Embed Flow
Icon |
|
|---|---|
Use When |
A flow needs to be called from another flow |
Samples |
|
Description |
The Call Flow component allows a flow to be called from another flow. |
Inbound Message Type |
Any Message Type |
Output Message Type |
Any Message Type |
Properties
| Name | Description |
|---|---|
Flow |
The flow to be called |
Execute
Icon |
|
|---|---|
Use When |
There is a need to execute an operating system command as part of a flow |
Samples |
|
Description |
The execute component executes an operating system command on the host on which the agent is running. |
Inbound Message Type |
Any Message Type |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Command |
The operating system command to be executed for every inbound message to the component |
Continue on Error |
Whether the flow should continue if the Command fails to execute properly |
Success Code |
The code that should be returned in the text based outbound message if the Command is successful |
Parameter replacement |
Enables token replacement for the Command. Message headers and flow parameters can be token replaced. The following token formats can be used: $(HeaderKey) or $(FlowParameter). Note that you can link:Stamp entity and text message values in a message header if you want to use message values. |
File Util
Icon |
|
|---|---|
Use When |
Files need to be renamed, copied, moved or deleted |
Samples |
|
Description |
The File Util component provides a sets of file utilities that can be used to manipulate files at the operating system level including copying, reanmeing, moving or deleting. |
Inbound Message Type |
Any Message Type |
Properties
| Name | Description |
|---|---|
Resource |
The source file based resource (either local or remote) on which the source file(s) to be manipulated reside |
Action |
Either Copy, Rename, Move, Delete or Touch. Copy will copy the file, move will move the file, delete will delete the file, and touch will update the files last updated date/time. |
Source Path |
The path and file name of the source file(s) to be manipulated. Wildcard character (*) can be used. |
Target Path |
The target path used to write files if the Action is Copy, Move, or Rename. This path will be appended to the path of specified Resource be specified as a path only, and the target file names will be derived from the source file names. |
Get Input File Name From Message |
Whether to get input file name from the payload of inbound messages or from the Source Path parameter. If selected, source file names come from the payload of inbound messages |
Target File Name |
The name to use for the target file. If not set, the original name will be used. |
Append To Name |
A text string that will be appended to the end of each target file name (before the extension). |
Format Delimited
Icon |
|
|---|---|
Use When |
Model based data needs to be formatted into delimited text |
Samples |
Relational Database to Flat File |
Description |
The format delimited component takes a model based message and formats its content into a delimited text message |
Inbound Message Type |
Model Based Message |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Delimiter |
The character that should be used to delimit the attribute values |
Quote Character |
The character that should be used to quote the attribute values that are of type String |
Header line |
Whether a header line should be created that contains attribute column names |
Component Editor
Double clicking on the Format delimited component in the flow will result in the Format Delimited editor being displayed as shown below.
The editor displays a row for every entity and attribute in the input model, and allows setting the ordinal position of each for the delimited output. Order the attributes in the order in which they should be displayed in the delimited result. Attributes can be ordered by:
-
Selecting an attribute and dragging it up or down in the list
-
Selecting an attribute clicking the "Move Up", "Move Down", "Move Top" or "Move Botton" buttons
-
Selecting an attribute or set of attributes, clicking "Cut" to remove them and then "Pasting them elsewhere in the list
In addition to specifying the order of the attributes in the delimited list, transform functions can also be applied to each attribute. These transform functions are good for applying formatting, etc.
Format Fixed
Icon |
|
|---|---|
Use When |
Model based data needs to be formatted into fixed length text |
Samples |
Relational Database to Flat File |
Description |
The format fixed component takes a model based message and formats its content into a fixed length text message |
Inbound Message Type |
Model Based Message |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Header line |
Whether a header line should be created that contains attribute column names |
Component Editor
Double clicking on the Format Fixed component in the flow will result in the Format Fixed editor being displayed as shown below.
The editor displays a row for every entity and attribute in the input model, and allows setting the ordinal position and width of each attribute for the fixed length output. Order the attributes in the order in which they should be displayed in the delimited result. Attributes can be ordered by:
-
Selecting an attribute and dragging it up or down in the list
-
Selecting an attribute clicking the "Move Up", "Move Down", "Move Top" or "Move Botton" buttons
-
Selecting an attribute or set of attributes, clicking "Cut" to remove them and then "Pasting them elsewhere in the list
Set the width by typing the width in to the Width field. Start and end positions are automatically calculated based on the order and width.
In addition to specifying the order and width of the attributes, transform functions can also be applied to each attribute. These transform functions are good for applying formatting, etc.
Format XML
Icon |
|
|---|---|
Use When |
Model based data needs to be formatted into xml text |
Samples |
Relational Database to Flat File, DB request to WSDL to DB results |
Description |
The format xml component takes a model based message and formats its content into an xml text message |
Inbound Message Type |
Model Based Message |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Ignore namespaces for XPath matching |
|
XML Format |
|
Component Editor
Double clicking on the Format XML component in the flow will result in the Format XML editor being displayed as shown below.
The first step in configuring the xml output is to specify the xml template that will be used to define the xml tag structure of the document. The XML template can be imported from an XSD or WSDL source that resides in a file or at a specified URL. Clicking the "Import Template" button opens the import dialog show below.
An XML template can also be manually entered by clicking the "Edit Template" button.
Once an XML template is defined XPath is used to map entities/attributes from the model to the xml template. See examples below.
Example 1. Format Person Entity to XML
XML Template
<Persons>
<Person>
<Id></Id>
<FirstName></FirstName>
<LastName></LastName>
<Gender></Gender>
</Person>
</Persons>
XPath Mapping
Lookup
Icon |
|
|---|---|
Use When |
Reference data exists that needs to be looked up / retrieved by a specified key, and another attribute of the the resulting reference data passed to downstream components. |
Samples |
|
Description |
The Lookup component receives reference data from a source component, and then uses that reference data as a lookup source for data coming from another source component. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Lookup Data Source |
The source component that will send reference data (to be looked up) to the Lookup component |
Lookup Key Attribute |
The model attribute in the reference data that will be used as the lookup key to the reference data |
Lookup Key Value |
The model attribute in the reference data that will be passed back from the lookup request |
Replacement Key Attribute |
The model attribute that will be used to lookup data from the Lookup Data Source. This value will be matched to the Lookup Key Attribute |
Replacement Value Attribute |
The model attribute that will be replaced by the Lookup Key Value from the resultant row in the Lookup Data Source |
| Ensure Lookup Key Attribute and Replacement Key Attribute are of the same data type, or the lookup will not work. I.E. model based messages are passed into the lookup component. These two keys must have the same data type in order to compare properly. |
Mapping
Icon |
|
|---|---|
Use When |
Data needs to be mapped from one set of entities/attributes (model structure) to another set of entities/attributes |
Samples |
Flat File to Relational Database, Normalized Table to Denormalized File, Partition by Gender |
Description |
The mapping component allows you to map messages from one model structure to another. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Set Unmapped Attributes To Null |
Whether attributes in the output model that don’t have a mapping specified from the input model should be set to null. |
Entity Per Record |
Determines whether inbound records will be split into multiple output records, one for each entity. Defaults to false. If set, the output model will have null values for every entity/attribute combination that is in the output model but not mapped from the input model. If not set, the output model will not contain entities/attributes that are not mapped from the input model. |
Component Editor
Double clicking on the Mapping component in the flow will result in the Mapping editor being displayed as shown below.
The Mapping editor displays the input model on the left hand side of the screen, and the output model on the right hand side of the screen. Data can be mapped from the input model to the output mode by:
-
Clicking on an attribute in the input model and dragging until your mouse cursor is over the desired output model attribute
-
Clicking on an attribute in the input model, then clicking again on the desired output model attribute
| For large models, typing an entity or attribute name in the Filter box will narrow the displayed model to entites or attributes matching the Filter criteria. |
The AutoMap feature will look at both input and output models and attempt to map the models based on similarly named attributes.
Merger
Icon |
|
|---|---|
Use When |
Different instances of model based records need to be merged into a single instance that combines the attributes from all instances. |
Samples |
Normalized Table to Denormalized File |
Description |
The Merger component takes different instances of entities (records) and merges them together based on a set of join criteria. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Component Editor
Double clicking on the Merger component in the flow will result in the Merger editor being displayed as shown below. The Merger editor displays a row for every entity/attribute in the input model and allows the selection of the columns on which to use as the key for the merge.
Example 1. Sample Merge
Model with Entity DENORMALIZED_PERSON having the following attributes:
* ADDRESS1 * ADDRESS2 * CELL_PHONE * CITY * FIRST_NAME * GENDER * HOME_PHONE * LAST_NAME * POSTAL_CODE * STATE * WORK_PHONE
A merger component with two sources, Person and Address Mapper which sends the following columns:
* ADDRESS1 * ADDRESS2 * CITY * FIRST_NAME * GENDER * LAST_NAME * POSTAL_CODE * STATE
The second source to the merger is the Home Phone Mapper which sends the following columns:
* FIRST_NAME * LAST_NAME * HOME_PHONE
The merge criteria selected on the Component Editor is
* FIRST_NAME * LAST_NAME
The merger will merge rows with the same FIRST_NAME and LAST_NAME, combining their attributes into a single entity instance (record), and then forward that record to downstream components.
Message Logger
Icon |
|
|---|---|
Use When |
Messages need to be logged to the Metl console and log file. |
Samples |
List Files in Temp Directory, Remove Duplicates, Sort Records, WSDL Service Call |
Description |
The message logger component takes inbound messages and logs them to the Metl console and log file. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Qualify Entity Attributes with Entity Name |
Whether attributes passed in a Model Based Message should be displayed with the Entity Name. If checked, the Entity Name will be displayed. |
Multiplier
Icon |
|
|---|---|
Use When |
Inbound messages need to be multiplied (duplicated n number of times) and then sent to downstream components. |
Samples |
|
Description |
The multiplier component multiplies or duplicates an inbound message a number of times based on a second seed set of inbound messages. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Multiplier Source |
The component that provides the multiplier. I.E. if an inbound record from another component should be multiplied n times, the Multiplier Source is the component that would send n records into the Muliplier component. The Multiplier Source are NOT the messages to be multiplied, but the messages that dictate how many times messgaes from other components will be multiplied. |
No-Op
Icon |
|
|---|---|
Use When |
A message needs to be passed through to another component without any action being taken on the message |
Samples |
|
Description |
The No Op component passes messages to a downstream component without any modifications to the original message. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Parse Delimited
Icon |
|
|---|---|
Use When |
An inbound text message in some delimited text format needs to be parsed into a model based message structure |
Description |
The parse delimited component takes an inbound text message in delimited format and parses it into a model based outbound message. |
Inbound Message Type |
Text Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Delimiter |
The character that represents the delimiter in the inbound text message. Delimiters can be specified with a simple character like a , or ^, or they can be specified by a hex string (for special characters) like \u00fd |
Quote |
The string or character that is used to quote string text in the inbound text message. |
Component Editor
Double clicking on the Parse delimited component in the flow will result in the Parse Delimited editor being displayed as shown below.
The editor displays a row for every entity and attribute in the output model, and allows setting the ordinal position of each for the delimited output. Order the attributes in the order in which they should be displayed in the delimited result. Attributes can be ordered by:
-
Selecting an attribute and dragging it up or down in the list
-
Selecting an attribute clicking the "Move Up", "Move Down", "Move Top" or "Move Botton" buttons
-
Selecting an attribute or set of attributes, clicking "Cut" to remove them and then "Pasting them elsewhere in the list
In addition to specifying the order of the attributes in the delimited list, transform functions can also be applied to each attribute. These transform functions are good for applying formatting, etc.
Parse Fixed
Icon |
|
|---|---|
Use When |
An inbound text message in a fixed length text format needs to be parsed into a model based message structure |
Samples |
|
Description |
The parse fixed length component takes an inbound text message in fixed length format and parses it into a model based outbound message. |
Inbound Message Type |
Text Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Delimiter |
The character that represents the delimiter in the inbound text message |
Quote |
The character that is used to quote string text in the inbound text message |
Component Editor
Double clicking on the Parse Fixed component in the flow will result in the Parse Fixed editor being displayed as shown below.
The editor displays a row for every entity and attribute in the output model, and allows setting the ordinal position and width of each attribute for the fixed length output. Order the attributes in the order in which they should be displayed in the delimited result. Attributes can be ordered by:
-
Selecting an attribute and dragging it up or down in the list
-
Selecting an attribute clicking the "Move Up", "Move Down", "Move Top" or "Move Botton" buttons
-
Selecting an attribute or set of attributes, clicking "Cut" to remove them and then "Pasting them elsewhere in the list
Set the width by typing the width in to the Width field. Start and end positions are automatically calculated based on the order and width.
In addition to specifying the order and width of the attributes, transform functions can also be applied to each attribute. These transform functions are good for applying formatting, etc.
Parse XML
Icon |
|
|---|---|
Use When |
An inbound text message in xml text format needs to be parsed into a model based message structure |
Samples |
DB Request to WSDL to DB Results |
Description |
The parse xml component takes an inbound text message in xml format and parses it into a model based outbound message. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Ignore namespaces for XPath matching |
|
Component Editor
Double clicking on the Parse XML component in the flow will result in the Parse XML editor being displayed as shown below.
The first step in configuring the parser is to specify the xml template that will be used to define the xml tag structure of the document. The XML template can be imported from an XSD or WSDL source that resides in a file or at a specified URL. Clicking the "Import Template" button opens the import dialog show below.
An XML template can also be manually entered by clicking the "Edit Template" button.
Once an XML template is defined XPath is used to map entities/attributes from the model to the xml template. See examples below.
Example 1. Parse Person XML to Entity
XML Template
<Persons>
<Person>
<Id></Id>
<FirstName></FirstName>
<LastName></LastName>
<Gender></Gender>
</Person>
</Persons>
XPath Mapping
Script
Icon |
|
|---|---|
Use When |
A custom action should be performed that can be written using Groovy |
Samples |
See Script Templates |
Description |
The script component executes one or more configurable groovy scripts. Groovy scripts can be configured to run at different points in the component lifecycle including Component Initialization (onInit), Inbound message arrival (onHandleMessage), Successful flow completion (onSuccess), and Flow completion with error (onError). The different run points are implemented as methods in Script Helper. All methods and fields in There are several Templates that can be referenced or used as a starting point to building your script. |
Inbound Message Type |
Any Message Type |
Output Message Type |
Any Message Type |
Properties
| Name | Description |
|---|---|
Resource |
A resource that can be accessed and utilized by the script. |
Import Statements |
Import statements for external dependencies |
Init Script |
The script to be executed upon component initialization |
Handle Msg Script |
The script to be executed upon receipt of an inbound message |
Ended with Success Script |
The script to be executed upon successful completion of the flow |
Ended with Error Script |
The script to be executed upon a failed completion of the flow. Note: this will not be called if a flow uses an Assert component to capture an error. This only triggers on regular flow errors. |
Example 1. Send Text Message
String body = "My Text Message Body"; callback.sendTextMessage(null, body);
Sequence
Icon |
|
|---|---|
Use When |
A sequence number needs to be created for a given attribute in a model based record, and the starting value of the sequence can be derived by a sql statement. |
Samples |
|
Description |
The sequence component allows an attribute in a model based message to be filled with a generated sequence number. The initial value for the sequence is derived based on a sql statement configured in the component. From that point forward, every new message/record that comes into the component gets an incremented sequence number placed in the designated sequence number attribute field. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Resource |
The Database Resource that will be used to execute the Starting Sequence Sql |
Sequence Attribute Name |
The name of the attribute in form [entity].[attribute] that should be filled in with the sequence number generated |
Select Starting Sequence Sql |
The Sql statement that will be executed in order to retrieve the starting sequence value. This sql will be run once on component start up to retrieve the sequence seed value |
Is Sequence Shared |
Whether the sequence is shared amongst multiple sequence generators |
Shared Name |
If the sequence is shared, this is the name that each of the sequence generators will use to reference the shared sequence |
Serializer
Icon |
|
|---|---|
Use When |
An entity message needs to be automatically formatted to JSON or XML |
Samples |
Format a model object to JSON prior to sending to a link:http-response(Http Response) for a web service |
Description |
Formats entity messages based on the input model |
Inbound Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Format |
Can have values of AUTOMATIC, JSON or XML. Default is AUTOMATIC. If AUTOMATIC is chosen, then the serializer will attempt to look at the flow parameters to determine what format should be used. If an Http Request started the flow, then the Accept header, the Content-Type header and the format parameter will all be inspected in order to determine the type. |
Structure |
Options for how to format the document. Options are: BY_TABLE and BY_INBOUND_ROW |
Sorter
Icon |
|
|---|---|
Use When |
Data within a model based message needs to be sorted by one of the attributes within the model or data needs to be sorted by multiple attributes within an entity in the model |
Samples |
|
Description |
The Sorter is used to sort data elements within a message that is being processed through a flow. The sort can be defined at either the message level or by entity/attribute using the component editor screen. When sorting by entity/attribute the data will be sorted per entity, where each entity will be treated as though it is a separate database table and the ordering occurs across the selected attributes (columns). NOTE: Using the entity/attribute sort option will use an H2 database to load the records into using the model entity as a table and the attributes as columns. Then it will perform an 'order by' in a sql statement to re-order the records. The tables will determine the key structure based on the way the entity is modeled in Metl and will use the key to load the H2 table(s). Word of caution, the key needs to be unique otherwise if your flow allows duplicate records this sort will remove these duplicates. Also, you may not have null values in the attributes you select as the key to the table. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Sort Entity.Attribute (optional) |
The single model entity and attribute to sort the data by. The entity and attribute must be concatenated with a period (entity.attribute). NOTE: Either an entry must be defined here or attributes selected in the Component Editor screen for the component to function. A value in this field overrides any attribute(s) selected on the Component Editor screen. |
Component Editor
Double clicking on the Sorter component in the flow or clicking the Component Editor button will result in the Sorter editor being displayed as shown below.
The Sorter editor displays a row for every entity/attribute contained within the input model and allows the selection of the attributes on which to use as the sort along with the ability to order them.
When a sort checkbox for an attribute is selected, the screen will re-sort all selected attributes to the top by entity. To change the sort order, select one or more attributes within an entity and use the buttons at the top to Move the selected item(s) up, down, to the top or to the bottom.
| You can only re-order the attributes within a single entity at any one time. |
Format Fixed
Icon |
|
|---|---|
Use When |
Model based data needs to be staged and queried. |
Samples |
|
Description |
The Temp RDBMS component stages model based message data in a temporary database to be queried. This component is the same as the RDBMS Reader but instead of providing a data source, the data source is automatically created using the user provided Input Model. As model messages are received, they are loaded into the temporary database. If the Run When parameter is configured to 'PER UNIT OF WORK', only one database is created and the SQL expression is executed after all model based messages have been received. If the Run When parameter is configured to 'PER MESSAGE', each new model based message will initiate the creation of a new temporary database. The provided SQL will be executed after each message has been loaded into the database. Once the SQL has been executed against the temporary database, the database is cleaned up and removed from memory or the file system. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Sql |
The sql query or script that will be executed for this reader. SQL hints are supported. |
In Memory Database |
Whether the temporary database should be created in memory or on the file system. |
| If you would like to see the contents of the temporary h2 database, replace the Temp RDBMS SQL with the text "script to '/home/user/temp_rdbms_db.sql'". When this runs, the database contents will be exported to the local file system. After the export completes, Metl will correctly throw an error since the query did not return any results. This should be done for debugging only. |
Text Replace
Icon |
|
|---|---|
Use When |
A piece of text in a text based message needs replaced with alternate text |
Samples |
|
Description |
The text replace component uses a regular expression to replace a piece of text with another piece of text within a text based message. |
Inbound Message Type |
Text Based Message |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Search For (regex) |
A regular expression that describes the text for which to search |
Replace With |
A text string that should be used to replace the text found by the Search For (regex) parameter |
Transformer
Icon |
|
|---|---|
Use When |
An attribute in a model based message needs transformed in one way or another (trim, substring, constants, etc.) |
Samples |
|
Description |
The transformer component allows transformation functions to be applied to record elements from inbound messages. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Component Editor
Double clicking on the Transformer in the flow will result in the Transformer editor being displayed as shown below.
The editor displays a row for every entity and attribute in the input model, and allows and allows transform functions to be applied to each. The following transform functions are available:
| Function | Description |
|---|---|
abbreviate(maxwidth) |
Abbreviates the attribute using ellipses. This will turn "Now is the time for all good men" into "Now is the time for…" |
currentdate() |
Replaces the attribute with the current date and time |
currentdate(format) |
Replaces the attribute with the current system date and time formatted based on the format expression. See Date Format Expression for details on the date format specification. |
daysFromNow(numberDays) |
Replaces the attribute with the date of current date + numberDays |
flowParameter(parameterName) |
Replaces the attribute wih the flow parameter specified by name parameterName |
format(spec) |
Formats the attribute into a string according to the provided spec. See Java Formatter class for details on the specification string |
formatdate(pattern) |
Formats the date attribute according to the pattern parameter. See Date Format Expression for details on the date format specification. |
getAttributeValueByName(attributeName) |
Gets the value of the specified attribute. Could be used to reference a different attribute from within the current one. |
integer() |
Converts the attribute to an integer |
left(length) |
Truncates the attribute to the left-most length digits |
lower() |
Converts the attribute to lower case |
lpad(padChar,length) |
Left pads the attribute with length padChar characters |
messageParameter(paramterName) |
Replaces the attribute with the given message parameter specified by name parameterName |
nullvalue() |
Replaces the attribute with the null value |
nvl(substituteForNull) |
Conditionally replaces the attribute with substituteForNull value if the attribute is null |
parseAndFormatDate(parsePattern,formatPattern) |
Parses the string attribute into a date based on the parsePattern and then formats it into a date time string according to the formatPattern parameter |
parseBigDecimal() |
Parses the string attribute into a Big Decimal |
parseDouble() |
Parses the string attribute into a Double |
parseInt() |
Parses the string attribute into an Integer |
parseLong() |
Parses the string attribute into a Long |
parsedate(pattern) |
Parses the string into a Date based on the pattern. See Date Format Expression for details on the parse pattern |
replace(searchString,replacement) |
Find and replace searchString with replacement. Tip: You can chain replace() calls. e.g., replace("1","a").replace("2","b") |
right(length) |
Truncates the attribute to the right-most length digits |
rpad(padChar,length) |
Right pads the attribute with length padChar characters |
sequence(seedValue, incrementValue, breakField) |
Increments a sequence by incrementValue starting at seedValue. Restarts at seedValue when breakField changes. breakField is an attribute name for the current entity record. |
substr(start,end) |
Returns the substring of the value beginning with the character at the specified start index and extends to the end index - 1 |
stringConstant(value) |
Replaces the attribute value with the constant value provided |
trim() |
Removes leading and trailing whitespace |
upper() |
Converts the attribute to uppercase |
Format and parse expressions for dates can be defined using the following designations.
| Letter | Date or Time Component | Presentation | Examples |
|---|---|---|---|
G |
Era designator |
Text |
AD |
y |
Year |
Year |
1996; 96 |
M |
Month in year |
Month |
July; Jul; 07 |
w |
Week in year |
Number |
27 |
W |
Week in month |
Number |
2 |
D |
Day in year |
Number |
189 |
d |
Day in month |
Number |
10 |
F |
Day of week in month |
Number |
2 |
E |
Day in week |
Text |
Tuesday; Tue |
a |
Am/pm marker |
Text |
PM |
H |
Hour in day (0-23) |
Number |
0 |
k |
Hour in day (1-24) |
Number |
24 |
K |
Hour in am/pm (0-11) |
Number |
0 |
h |
Hour in am/pm (1-12) |
Number |
12 |
m |
Minute in hour |
Number |
30 |
s |
Second in minute |
Number |
55 |
S |
Millisecond |
Number |
978 |
z |
Time zone |
General time zone |
Pacific Standard Time; PST; GMT-08:00 |
Z |
Time zone |
RFC 822 time zone |
-0800 |
Union
Icon |
|
|---|---|
Use When |
Inbound messages need to be combined into a single message and then sent to downstream components. |
Samples |
|
Description |
The union component combines all inbound messages from each input link and then sends one combined message to the downstream components. The combination of messages continues until all messages have been processed and the unit of work boundary is reached. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Web Request
Icon |
|
|---|---|
Use When |
A web service (either REST or WSDL) needs to be called to perform an action such as retrieving or writing data |
Samples |
WSDL Service Call, DB Request to WSDL to DB results |
Description |
The web component allows REST or WSDL based web services to be called, sending the response data to downstream components. The Web component can be used to call either WSDL or REST based services. The Web component takes a text based message as input. This text based input message is formatted XML that represents the web service request. Thus, the web component is frequently used with the XML Formatter component which takes entity data and formats it into an xml formatted text message. |
Inbound Message Type |
Text Based Message |
Output Message Type |
Text Based Message |
Properties
| Resource | An Http resource where the service resides and can be called |
|---|---|
When providing the web service request message set this to PER UNIT OF WORK. When the request message is received from a prior component set this to PER MESSAGE |
|
Append to URL |
Text that will be appended to to the Connection url forming the base url for the web service call |
Body From |
Either 'Message' or 'Provided'. If Message the body of the web service request message will come from the inbound message. If 'Provided" the body of the web service request message will come from the Body Text parameter |
Body Text |
If the Body From parameter is 'Provided', this parameter is used to specify the body of the text message for the web service request |
Http Headers |
A list of line feed separated Http Headers that will be passed in the web request. The name value pairs should be separated by a colon (:). |
Http Parameters |
A list of line feed separated Http Parameters that will be passed on the URL of the web request. The name value pairs should be separated by a colon (:). |
Parameter replacement |
Enables token replacement for the Command. Message headers and flow parameters can be token replaced. The following token formats can be used: $(HeaderKey) or $(FlowParameter). Note that you can link:Stamp entity and text message values in a message header if you want to use message values. |
XML Formatter
Icon |
|
|---|---|
Description |
The XML Formatter component takes an inbound model based message and formats into an xml formatted text message. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Text Based Message |
XSLT Processor
Icon |
|
|---|---|
Use When |
XSLT needs applied to a set of input data in order to transform it into a given output format |
Samples |
|
Description |
The XSLT Processor component applies XSLT to a set of input data and sends the output of the XSLT processing as text messgaes to downstream components. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Text Based Message |
Properties
| Input Model | |
|---|---|
Output all attributes |
|
XML Format |
|
Parameter replacement |
Enables token replacement for the Command. Message headers and flow parameters can be token replaced. The following token formats can be used: $(HeaderKey) or $(FlowParameter). Note that you can link:Stamp entity and text message values in a message header if you want to use message values. |
4.5.3. Writers
Binary File Writer
Icon |
|
|---|---|
Use When |
Data needs written to a binary based file |
Samples |
|
Description |
The Binary File Writer is used to write data to a binary based file. The Binary File Writer can write to one output file either locally or to a remote destination. |
Inbound Message Type |
Binary Based Message |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Resource |
The file based resource (either local or remote) on which the file will be written. |
Path and File |
The relative path and file of the file to be written. The relative path and file are concatenated with the absolute path specified in the File based resource |
Must Exist |
Requires the file to exist prior to writing |
Append |
Add input data to the end of an existing file |
Get File Name From Message |
If selected, the path and file name to be written will be determined by the payload of the incoming message. If this setting is not selected, the path and file to be read is based on the File Path property. |
Param Name For Msg Based File Name |
If 'Get File Name From Message' property is checked, this is the name of parameter to pull the filename from. |
Email Writer
Icon |
|
|---|---|
Use When |
An email needs to be sent |
Samples |
You have a situation or business event where someone needs to be notified via email. |
Description |
The Email Writer is used to send one or multiple emails to one or multiple recipients. The Email Writer can take one or more input messages. The writer will be executed according to the 'Run When' setting. Depending on the setting and the type of input message the email settings (subject, body, to …) that have tokens will be filled out. Attachments can also be included with the email message. The file(s) to be attached must be located on the local server of the Metl instance. |
Inbound Message Type |
Any |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Resource |
A mail server session. If this is left blank, then the global mail server session will be used |
Subject |
The subject line for the email. This can be templated |
Body |
The body text for the email. This can be templated |
FROM: |
The from line for the email. If this is not set, the value will be used that is configured in the resource |
TO: |
The TO line for the email. This can be templated. The TO values can also be provided from a source step |
CC: |
The CC line for the email. This can be templated. The CC values can also be provided from a source step |
BCC: |
The BCC line for the email. This can be templated. The BCC values can also be provided from a source step |
Source Step For Email Addresses |
This is optional. If the recipients come from a database or source file this can be used to feed them to the component |
Source Step Email Addresses Type |
How to use the source recipients. Values can be TO, CC, and BCC |
One Email per Recipient |
Setting that indicates if multiple emails should be sent or just one |
Include Attachment |
Setting that indicates if an attachment should be included with the email sent |
Attachment File Full Path |
Fully qualified local server file path. To include multiple attachments include a pipe '|' between files. For example: C:\Temp\testfile.txt|C:\Some Folder\My Filename Here.pdf |
Run When |
When this component should send email |
Excel File Writer
Icon |
|
|---|---|
Use When |
Data needs written to an Excel based file |
Samples |
|
Description |
The Excel File Writer is used to write data to an Excel based file. The Excel File Writer can write to one output file either locally or to a remote destination. Excel files that are written can be in either Microsoft Excel (.xls) or Microsoft Excel XML (.xlsx) format. Note: The current version of the Excel File Writer only allows the writing of a single tab in a workbook and does not allow updates to a file, it will only replace the file if the file already exists. |
Inbound Message Type |
Entity Based Message |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Resource |
The file based resource (either local or remote) on which the file will be written. |
Excel Output Type |
The type of Excel file to generate. (.xls or .xlsx) |
Must Exist |
Requires the file to exist prior to writing |
Empty File On No Input |
Output an empty file if no input messages occur |
Path and File |
The relative path and file of the file to be written. The relative path and file are concatenated with the absolute path specified in the File based resource. Note, the extension must match the output type selected or flow will error. Also, if no extension is provided the component will automatically add the proper extension based on the type. |
Sheet Name |
String to use as the Excel worksheet (tab) name. Must follow Excel guidelines for naming. If invalid characters are identified they will be replaced with spaces. Not providing will use the default of Sheet1. |
Include Header Row |
Whether to add a header row with attribute names as column headings in the output file. |
Get File Name From Message |
If selected, the path and file name to be read will be determined by the payload of the incoming message. If this setting is not selected, the path and file to be read is based on the File Path property. |
Param Name For Msg Based File Name |
Name of the parameter containing the file to be read. |
Notes |
General notes about this component. |
| The output rows/columns will start in cell A1. |
RDBMS Writer
Icon |
|
|---|---|
Use When |
Data needs written to a relational database |
Samples |
Flat File to Relational Database, DB request to WSDL to DB results |
Description |
The RDBMS Writer is used to write data to a SQL compliant relational database by specifying an input model. The RDBMS Writer can take one or more input messages. The writer will be executed (i.e. the update and/or insert sql query run) for every entity record within an input message that it receives. By default, input records will attempt to be inserted into the defined Input Model. Parameters can be selected that will allow an update only or a combination of an update and insert in either order depending on selections. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Resource |
The SQL Database Resource on which the SQL query should be run to write the data. |
Catalog |
The catalog where the target table resides |
Schema |
The schema where the target table resides |
Replace rows if they exist |
Whether to execute an update sql if a unique key exception occurs on the default insert sql attempt |
Update rows first instead of insert |
Whether to execute an update sql first |
Fallback to insert if no rows updated |
Whether to execute an insert sql if at first the update sql updates no rows |
Quote table and column names |
Whether or not to include table and column names in quotes |
Trim character data to fit within column |
Whether to trim the character field data if the incoming values length is larger than the target column size |
Continue on Error |
Whether to continue processing records on a failure and count the error record in the ignore count |
Batch Mode |
Whether to execute the insert and/or update as a transaction allowing rollback capability upon failure (assuming the RDBMS is not an auto commit system) |
Create table if missing |
Whether to create the target table if it does not exist on the database. |
| Specifying both a Catalog and Schema, the flow will only use the Catalog value and ignore the Schema Specifying 'Fallback to insert…' has no affect unless 'Update rows first…' is also selected When no options are checked the default is to attempt an insert only |
Sql Executor
Icon |
|
|---|---|
Use When |
Data needs written to or deleted from a database using a specific sql statement |
Samples |
|
Description |
The Sql Executor is used to write data to a SQL compliant relational database by specifying a sql query. The Sql Executor can take one or more input messages. The sql will be executed based on the 'Run When' option, either Per Message - one time per message received, Per Entity - for every entity record in an input message or On Success - one time at flow completion. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Resource |
The SQL Database Resource on which the SQL query should be run to write the data. |
Run When |
Timing of when to run the defined sql. Whether the sql executes after each message received, once per entity record in a message or on flow completion. |
File |
File containing Sql statements to run. If a value is included here it will be used instead of any Sql Statements in the next box. |
Sql |
Sql statement(s) to run. Could be an insert, update or delete statement. Multiple statements can be executed by including a semi-colon (;) between sql statements. |
Text File Writer
Icon |
|
|---|---|
Use When |
Data needs written to a text based file |
Samples |
|
Description |
The Text File Writer is used to write data to a text based file. The Text File Writer can write to one output file either locally or to a remote destination. Text files that are written can be in any format (fixed length, comma delimited, etc.) and can be formatted by an upstream processing component like the Format Delimited and Format Fixed components. |
Inbound Message Type |
Text Based Message |
Output Message Type |
Text Based Message |
Properties
| Name | Description |
|---|---|
Resource |
The file based resource (either local or remote) on which the file will be written. |
Path and File |
The relative path and file of the file to be written. The relative path and file are concatenated with the absolute path specified in the File based resource |
Must Exist |
Requires the file to exist prior to writing |
Append |
Add input data to the end of an existing file |
Line Terminator |
String to append to the end of each line written to the file |
Get File Name From Message |
If selected, the path and file name to be read will be determined by the payload of the incoming message. If this setting is not selected, the path and file to be read is based on the File Path property. |
Param Name For Msg Based File Name |
Name of the parameter containing the file to be read. |
Empty File On No Input |
Output an empty file if no input messages occur |
| Specifying None on the Action on Error does NOT stop the flow from failing when a file cannot be written. It simply specifies what should be done with the file when the error condition does occur. |
Zip
Icon |
|
|---|---|
Use When |
Data needs to be put in a compressed file |
Samples |
|
Description |
The Zip component is used to generate a compressed file of data provided to it. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Source Directory Resource |
The file based resource (either local or remote) on which the source files reside. |
Target Directory Resource |
The file based resource (either local or remote) on which the compressed file will be written. |
File Path |
The relative path and file of the zip file to be written. The relative path and file are concatenated with the absolute path specified in the File based resource. |
Delete Source Files |
Whether to delete the source files upon completion of creating the compressed file |
Encoding |
File encoding type, defaults to UTF-8 |
4.5.4. Services
Http Request
Icon |
|
|---|---|
Use When |
A RESTful service needs to be created and hosted by the Metl server |
Samples |
Service Get All Persons, Service Get All Persons Custom XML, Service Get Person By Id, Service Upsert Persons |
Description |
The HTTP Request component is used to define the expected inbound HTTP Request for a RESTful service that will be hosted by the Metl server |
Inbound Message Type |
None |
Output Message Type |
Any |
Properties
| Name | Description |
|---|---|
Output Model |
If an output model is specified, Metl will attempt to automatically unmarshal the payload of the inbound message (either xml or json) into the structure specified by the output model definition. See Example 1 below. |
HTTP Method |
The http method for which http requests will be directed to this component. Either, GET, PUT, POST or DELETE. |
Path |
The path definition for which http requests will be directed to this component. This is a partial path that will be used in conjunction with the base metl path. I.E. if Metl is accessed at http://myserver:42000/metl and the path specIfied is /customer/listAll, then HTTP requests to http://myserver:42000/metl/api/ws/customer/listAll will be directed to this component and flow. Parameters for a service request can be specified in one of two ways. The first is with a standard http request parameter. For example, with a path of /customer/getById and a base Metl url of http://myserver:42000/metl, http requests to http://myserver/api/ws/customer/getById?customerId=1234 will pass http requests to the component with customerId available as a parameter. Parameters can be accessed by enclosing them in $(). I.E. $(customerId). The second way to specify a parameter is within the url itself. I.E. with the following path /customer/getById/{customerId} and base metl url of http://myserver:42000/metl, requets to http://myserver/api/ws/customer/getById/1234 will also pass http requests to the component with customerId available as a parameter. |
Example 1. Automatic unmarshelling of xml or json in the http request component
As discussed above, the http request component will attempt to automatically unmarshal the payload of the inbound message (either xml or json) if the Output Model is specified. The format of the xml or json must look as follows to be successfully unmarshalled.
XML Example
<ArrayList> <item> <name>PERSON</name> <!-- The model entity --> <data> <GENDER>M</GENDER> <!== The model attributes --> <ID>1</ID> <LAST_NAME>Arbuckle</LAST_NAME> <FIRST_NAME>Garfield</FIRST_NAME> </data> </item> <item> <name>PERSON</name> <!-- The second row of entity and attributes --> <data> <GENDER>M</GENDER> <ID>2</ID> <LAST_NAME>Arbuckle</LAST_NAME> <FIRST_NAME>Odie</FIRST_NAME> </data> </item> </ArrayList>
JSON Example
[
{
"name":"PERSON",
"data":
{
"GENDER":"M",
"ID":"1",
"LAST_NAME":"Arbuckle",
"FIRST_NAME":"Garfield"
}
},
{
"name":"PERSON",
"data":
{
"GENDER":"M",
"ID":"2",
"LAST_NAME":"Arbuckle",
"FIRST_NAME":"Odie"
}
}
]
Http Response
Icon |
|
|---|---|
Use When |
A RESTful service needs to be created and hosted by the Metl server |
Samples |
Service Get All Persons, Service Get All Persons Custom XML, Service Get Person By Id, Service Upsert Persons |
Description |
The HTTP Response component is used to define the expected outbound HTTP Response for a RESTful service that will be hosted by the Metl server |
Inbound Message Type |
None |
Output Message Type |
Any |
Properties
| Name | Description |
|---|---|
Input Model |
If an input model is specified, Metl will attempt to automatically marshal the payload of the entity based inbound message to the http response component into a json or xml outbound message See Example 1 below. |
Content Type |
The content type of the output. If an input model is specified, and the http response component is automatically marshalling the output, the content type will be set automatically based on the accept parameter of the request. If no input model is specified and the flow is formatting the outbound message as part of the flow, the content type should be set as approproiate. |
Example 1. Automatic marshelling of xml or json in the http response component
As discussed above, if the Inbound Model is specified, the http response component will attempt to automatically marshal the payload of the inbound entity message into either an xml or json outbound payload. The format of the xml or json will look as follows:
XML Example
<ArrayList> <item> <name>PERSON</name> <!-- The model entity --> <data> <GENDER>M</GENDER> <!== The model attributes --> <ID>1</ID> <LAST_NAME>Arbuckle</LAST_NAME> <FIRST_NAME>Garfield</FIRST_NAME> </data> </item> <item> <name>PERSON</name> <!-- The second row of entity and attributes --> <data> <GENDER>M</GENDER> <ID>2</ID> <LAST_NAME>Arbuckle</LAST_NAME> <FIRST_NAME>Odie</FIRST_NAME> </data> </item> </ArrayList>
JSON Example
[
{
"name":"PERSON",
"data":
{
"GENDER":"M",
"ID":"1",
"LAST_NAME":"Arbuckle",
"FIRST_NAME":"Garfield"
}
},
{
"name":"PERSON",
"data":
{
"GENDER":"M",
"ID":"2",
"LAST_NAME":"Arbuckle",
"FIRST_NAME":"Odie"
}
}
]
If an input model is NOT specified, the flow will need to format the output in either xml or json format accordingly.
Subscriber
Icon |
|
|---|---|
Use When |
You need to consume messages from a Pub/Sub system (like JMS) |
Samples |
|
Description |
This component requires a resource that supports subscriptions. When enabled, it will subscribe with the resource. Each message it receives results in an execution of a flow. |
Inbound Message Type |
None |
Output Message Type |
Text |
Properties
| Name | Description |
|---|---|
4.5.5. Controls
Gate
Icon |
|
|---|---|
Use When |
A component(s) needs to wait until another component has completely finished |
Samples |
Service Get All Persons, Service Get All Persons Custom XML, Service Get Person By Id, Service Upsert Persons |
Description |
The Gate component passes all incoming messages except for messages from the control source component to a downstream component once the gate component receives the final unit of work control message from the control source. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
Gate Control Source |
The source component that controls the opening of the gate |
Last Unit of Work
Icon |
|
|---|---|
Use When |
A component needs to wait until one or more other components complete prior to continuing |
Samples |
|
Description |
The Last Unit of Work component waits until all end of work control messages are received from all input components before a message is forwarded to the remaining downstream components. |
Inbound Message Type |
Model Based Message |
Output Message Type |
Model Based Message |
Properties
| Name | Description |
|---|---|
4.5.6. Common Properties
Action on Error
Either None, Archive or Delete. If Delete, the input file will be deleted if there is an error when reading. If Archive, the input file will be archived to the Archive on Error Path property location if there is an error when reading. If None, the input file will be left in its original position if there is an error when reading.
Archive on Error Path
If Action on Error is set to Archive, this property sets the archive location where the input file should be archived if there is an error when reading.
Action on Success
Either None, Archive or Delete. If Delete, the input file will be deleted after it is successfully read. If Archive the input file will be archived to the Archive on Success Path property location. If None, the input file will be left it its original location once read.
Archive On Success Path
If Action on Success is set to Archive, this property sets the archive location where the input file should be archived once successfully read.
Cancel On No Files
If there are not files to process then cancel the entire flow (versus putting the flow in ERROR status.)
Enabled
Whether the component is enabled for the given flow.
Encoding
The encoding to use when reading or writing of text data.
Inbound Queue Capacity
The number of pending inbound messages this component can store as it processes other requests.
Input Model
The input model that describes input data sent to this component.
Log Input
Whether to the log input messages to this component to the log file.
Log Input
Whether to the log output messages from this component to the log file.
Output Model
The output model that describes data sent from this component.
Rows/Msg
The number of entity rows that will be packaged into a single message.
Run When
Indicates when the component will run. Most components have options of:
PER UNIT OF WORK |
The component will run when it receives a control message. |
PER MESSAGE |
The component will run when it receives a content message. |
Shared
Whether this component definition is shared across multiple flows.
4.6. Models
Data integrated with Metl can be structured or unstructured data. When dealing with Structured data, the structured data can be defined by modeling the data. Metl Models allow you to describe that structured data. Models can either be relational or hierarchical in nature. All structured data must be modeled in Metl. For example, a delimited file may exist that looks as follows:
Smith,John,male,10/19/2000 Smith,Sally,female,5/4/2004 Smith,Sarah,female,2/1/2015
This file represents people including their last name, first name, gender and date of birth. In order to use this file in a structured way, you must model the data contents of the file. I.E. define an entity named something like Person, and then define attributes for that person (lastName, firstName, gender, dateOfBirth).
4.6.1. Relational Models vs Hierarchical Models
A model in Metl can either be a relational model or a hierarchical model. The model type needed depends directly on the structure of the data to be modeled. Data coming from or going to a relational database, most commonly utilizes a relational model. A relational model has tables, columns, primary keys, foreign keys, etc. A relational model in Metl is largely like a relational model in any relational database platform. Hierarchical models are traditionally used when the data you are modeling doesn’t have explicit relationships defined in the data, but instead has relationships that are implicitly defined based on nesting of entities. Hierarchical models are commonly used for xml and json structures which are used in service based architectures.
4.6.2. Working with Models
To create a new model, click on the "Models" folder in the navigation pane, and then click "New" - "Model", and then either "Relational" or "Hierarchical" as shown below. For this example, we will create a Relational Model as the contents of the file will be sent to a relational database.
Once the model has been created, double click the model in the navigation pane in order to edit the model. From the content editor pane, there are buttons for adding and entities and attributes as well as positing those attributes within the model. Highlighting an entity and clicking the "Add Attribute" button will add an attribute to the given entity.
Importing Relational Models from a Database
Similarly when dealing with data in a relational database, there is an inherent model defined in the relational database itself. I.E. tables and columns equate to entities and attributes. Metl relational models for relational databases can be automatically imported from the database structure itself, or they can be maintained manually. In order to import a model from a relational database, click the "Import…" button. The following screen will be displayed which displays all database resources. Drilling into a database resource allows the user to multi-select one or more tables in order for those tables to be imported into the model.
4.6.3. Importing Relational Models from a Comma Separated File
If a relational model contains several entity and/or attributes you can use a comma separated file to import the model layout. In order to import a model from a file, click the "Import…" button, then select "Relational CSV File". The following screen will be displayed where you can click the button to choose your file. Then click "Import".
The CSV file must contain only 5 columns with no header row. The columns are as follows:
Entity Name |
Required *first row of each Entity |
This is the name of your Entity, it is only required on the first row for a given Entity. Subsequent rows (attributes) will assume the prior Entity if none is provided. |
Attribute Name |
Required |
This is the name of the Attribute |
Description |
Optional |
A description of the Attribute |
Data Type |
Optional |
This is the data type of the attribute. Note: if none is provided or if the value is not a valid type, a type of "OTHER" will be used. These can then be manually edited later. |
Is Primary Key |
Optional |
This defines whether the attribute should be treated as the PK of the Entity. Valid values are 'Y', 'Yes', 'X' or 1. Any other value entered will be ignored and the attribute will not be marked as a PK. |
4.6.4. Example file
ENTITY_1,Attribute_1,Description of Attribute 1,INTEGER,X ,Attribute_2,Description of Attribute 2,VARCHAR(20), ,Attribute_3,,CHAR, ENTITY_2,Attr1,,BIGINT, ENTITY_2,Attr2,,STRING, ENTITY_2,Attr3,Key Field,INTEGER,Y ENTITY_2,Attr4,,DECIMAL(9,2),
4.6.5. Importing Relational Models from a source file containing a Column Header row
If a source file contains a delimited header row of column names you can use this to create a relational model with attributes of the column names from the file. In order to import a model from the header row in a file, click the "Import…" button, then select "Source File Header Row". The following screen will be displayed where you will enter the Entity Name, Delimiter (single character delimiter ie: , \ | \t …) and click the button to choose your file. Then click "Import".
The source file must have a header row that contains a delimited set of column names to use as attribute names. The attributes will be created as VARCHAR data types and none will be selected as PK and no descriptions will be added.
4.7. Resources
Resources represent connections to physical end-points where data is read from or written to. A resource is configurable per environment (agent). This allows a flow to be tested against a resource configured for testing and then deployed to an agent with resources configured for production.
The following resource types are available in Metl:
| Rsource Type | Description |
|---|---|
Database |
|
A JDBC connection to a JDBC compliant relational database |
|
Directory |
|
A connection to a local file system |
|
An FTP connection to a file system |
|
An SFTP connection to a file system |
|
An SMB connetion to a server |
|
A JMS connection to a specified queue and topic |
|
HTTP(S) |
|
An HTTP(S) connection to REST or SOAP based services |
|
Mail Session |
|
An SMTP connection to a Mail server |
|
Subscribe |
|
A JMS connection to a specified queue or topic that can be subscribed to |
|
Resources are defined on a project by project basis. In order to create a new resource, Click on the "Resources" folder in the Navigation pane, click the "New" menu item, and then select the type of resource to be created.
If resources will be shared across multiple projects, a project can be defined that only contains resources, and then that project can be used as a Project Dependency for other projects that will leverage the shared resources.
When a flow is deployed, its associated resources are automatically deployed with it. A flow, and thus its associated resources, can be deployed to one or more Metl agents. Each deployment of that resource on a given agent can have different settings. For example, companies typically have several environments for a given system (i.e. development, test, user acceptance test, production). A flow and its resources can be deployed to the development agent/environment with settings specific to the development environment (i.e. server names, ids, passwords, etc.) When that same flow is deployed to the test agent/environment the resources on the test environment can be changed for the specifics of that environment. The flow itself stays the same regardless of the environment in which it is running. This allows seamless migrations from environment to environment.
To edit the settings of a resource, double click the resource in the navigation pane, and the settings details will be displayed in the content pane.
4.7.1. Database
The following settings are available for the Database resource
| Setting | Description |
|---|---|
Driver |
The fully qualified Java class name of the JDBC driver to be used |
Url |
The connection URL to be passed to the JDBC driver to establish a connection |
User |
The connection username to be passed to the JDBC driver to establish a connection |
Password |
The connection password to be passed to our JDBC driver to establish a connection |
Validation Query |
The SQL query that will be used to validate connections from this pool before returning them to the caller. If specified, this query MUST be an SQL SELECT statement that returns at least one row. If not specified, connections will be validated by calling the isValid() method |
Initial Size |
The initial number of connections that are created when the pool is started |
Max Active |
The maximum number of active connections that can be allocated from this pool at the same time, or negative for no limit |
Max Idle |
The maximum number of connections that can remain idle in the pool, without extra ones being released, or negative for no limit |
Min Idle |
The minimum number of connections that can remain idle in the pool, without extra ones being created, or zero to create none |
Wait Time (ms) |
The maximum number of milliseconds that the pool will wait (when there are no available connections) for a connection to be returned before throwing an exception, or -1 to wait indefinitely |
Evict Time (ms) |
The minimum amount of time an object may sit idle in the pool before it is eligible for eviction by the idle object evictor (if any) |
Test on Borrow |
The indication of whether objects will be validated before being borrowed from the pool. If the object fails to validate, it will be dropped from the pool, and we will attempt to borrow another |
Test on Return |
The indication of whether objects will be validated before being returned to the pool |
Test while Idle |
The indication of whether objects will be validated by the idle object evictor (if any). If an object fails to validate, it will be dropped from the pool |
Init Sql |
An SQL statement that will be sent to the jdbc driver upon each connection to the database. Can be used for custom initialization |
Fetch Size |
The number of rows to fetch at a time for a sql select statement Query timeout (seconds) |
The allowable time a query can run without completing before being cancelled Connection Properties |
The connection properties that will be sent to our JDBC driver when establishing new connections. Format of the string must be [propertyName=property;] |
Adding a JDBC Driver .jar file
Metl comes pre-packaged with the following jdbc drivers:
-
SQL Server jTDS jdbc driver
-
H2 jdbc driver
-
Postgres jdbc driver
For other databases such as Oracle, jdbc drivers can be downloaded from the respective vendor and added to the classpath of Metl. Adding the jdbc driver to the classpath depends on whether Metl is being run as a service.
If Metl is being run as a service:
-
Download the jdbc driver and place it on the local file system
-
Edit the Metl metl_service.conf file and change the wrapper.java.classpath line in the file.
wrapper.java.classpath.1=/path/to/metl/metl.war:/path/to/oracle/jdbc/driver/ojdbc6.jar
| The separator for the classpath shown above depends on the operating system. For OSX and Linux, use ":", for Windows use ";" |
If Metl is NOT being run as a service, set the classpath to include the additional jdbc drivers as well as the metl.war and launch metl by specifying the class name within the .war file as shown below.
java -cp /path/to/oracle/jdbc/driver/ojdbc6.jar;/path/to/metl/metl.war org.jumpmind.metl.Metl
Example database drivers and urls
In Memory H2 Database
| parameter | value |
|---|---|
Driver |
org.h2.Driver |
Url |
jdbc:h2:mem:mydatabasename |
File based H2 Database (linux, osx)
| parameter | value |
|---|---|
Driver |
org.h2.Driver |
Url |
jdbc:h2:~/foldername/databasename |
File based H2 Database (windows)
| parameter | value |
|---|---|
Driver |
org.h2.Driver |
Url |
jdbc:h2:C:/foldername/databasename |
SQL Server Using JTDS driver
| parameter | value |
|---|---|
Driver |
net.sourceforge.jtds.jdbc.Driver |
Url |
jdbc:jtds:sqlserver:/servername.domainname:1433;databaseName=databasename; |
Oracle
| parameter | value |
|---|---|
Driver |
oracle.jdbc.OracleDriver |
Url |
jdbc:oracle:thin:@//servername.domainname:1521/servicename |
Postgres
| parameter | value |
|---|---|
Driver |
org.postgresql.Driver |
Url |
jdbc:postgresql://servername:5432/databasename |
MySql
4.7.2. Local File System
The following settings are available for the Local File Resource
| Setting | Description |
|---|---|
Path |
The base path that should be used when accessing files from this resource |
4.7.3. FTP
The following settings are available for the FTP resource
| Setting | Description |
|---|---|
Server |
The name of the FTP server including domain name. i.e. servername.domainname |
Port |
The TCP port over which to communicate. Default FTP port is 21 |
User |
The user name that should be used to connect to the FTP resource |
Password |
The password that should be used to connect to the FTP resource |
Base Path |
The base path that should be used once the connection has been made (i.e. the connection will default to this base directory on the server for subsequent requests) |
Connection Timeout |
The amount of time to wait for connection before canceling the connection |
4.7.4. SFTP
| Setting | Description |
|---|---|
Server |
The name of the SFTP server including domain name. i.e. servername.domainname |
Port |
The TCP port over which to communicate. Default SFTP port is 22 |
User |
The user name that should be used to connect to the SFTP resource |
Password |
The password that should be used to connect to the SFTP resource (if password is used to authenticate) |
Key File Location |
The full path and file name of a key file that should be used for authentication (if public key authentication is used to authenticate) |
Base Path |
The base path that should be used once the connection has been made (i.e. the connection will default to this base directory on the server for subsequent requests) |
Connection Timeout |
The amount of time to wait for connection before canceling the connection |
4.7.5. Web Resources
The following settings are available for Web Resources
| Setting | Description |
|---|---|
Url |
The URL of the web resource to be utilized |
Http Method |
The Http method t be sent for requests sent to this resource, either GET, PUT or POST |
Http Timeout |
The max time to wait for a response from the request before canceling the request |
Content Type |
The content type for the body for requests sent to this resource |
Security |
Either None, Basic Auth, OAuth 1.0 or Token |
User |
If Security is set to Basic Auth, the user name to be used to authenticate requests sent to this resource |
Password |
If Security is set to Basic Auth, the password to be used to authenticate requests sent to this resource |
Token |
If Security is set to Token, this is the token provided by the web service provider to be used to authenticate requests sent to this resource |
5. Release
6. Deploy
Once flows have been designed and configured, they can be deployed to a Metl agent to be run. When a flow is deployed to an agent, its associated resources are also deployed to that agent and can be configured differently for each agent deployment. In a typical scenario, a company might have application environments for development, system test, user acceptance test and production. Similarly with Metl, you can have a development, system test, user acceptance test and production Metl agents. After a flow has been designed and configured, it can be deployed to the development agent for dev testing. The resources that are deployed on the Metl development agent can point to the external systems development resources. When the flow is ready for system testing, it can be deployed to a Metl system test agent, and the associated resources on that agent can point to the external systems system test resources, etc.
Each running instance of the Metl application can have one or more Metl agents running underneath it. As an example, development, test and production agents could all be run under a single running instance of the Metl application. More commonly each agent is deployed on a different physical server or virtual machine.
When a flow is run from the design screen, Metl uses a pre-configured agent on the current running instance of Metl called the design-time agent. This agent is like any other Metl agent except that it is created automatically.
Similar to the design screen, the deploy screen is split into different panels. The navigation panel show below, allows viewing of the Metl agents that are configured and running for the current instance of the Metl application.
Agents are organized within folders. To create a new folder, click "New" - "Folder", and type the new name of the folder.
Similarly, to create a new Metl Agnet, click on the folder under which you would like the agent organized, click "New" - "Agent", and type the name of the new agent.
Double clicking on an agent in the navigation panel, or selecting an agent and clicking "Edit" - "Open" will open the agent details in the content editor panel as shown below.
The Start Mode for each agent can be "AUTO" or "MANUAL". When in "AUTO" mode, all agents defined in this instance of the Metl application will be started automatically when the Metl software is run. The Hostname lists the computer’s hostname on which the agent is being run.
Agent specific parameters can be set on the Agent screen. Clicking the "Parameters" button displays a dialog box where these parameters can be maintained. These parameters are available to flow components and can be used when things (in addition to resources) need to be specified on an environment by enironment basis.
The main list on the content editor panel of the deploy screen displays the flows and resources that have been deployed to a given agent.
| Column Name | Description |
|---|---|
Project Name |
The name of the project to which the deployed item belongs |
Deployment |
The name of the item that has been deployed to this agent. This can be a resource name or a flow name |
Type |
The type of the item that has been deployed. Either "Resource" or "Flow" |
Status |
The status of the deployed item. Either "DISABLED" or "ENABLED". If "ENABLED" the flow will can be run on a scheduled or manual basis from the agent. If "DISABLED" the job must be enabled before it can be run. This column is only applicable for a flow. |
Log Level |
The log level for the deployed item. Either "DEBUG", "INFO", "WARN", or "ERROR". Determines the log level that will be output for the flow when the flow is run on the agent. This column is only appicable for a flow. |
Start Type |
Determines how the flow will be started or run on the agent. Either "MANUAL", "ON_DEPLOY", or "SCHEDULED_CRON". If "MANUAL" the flow will not be invoked automatically by the system, but can be run manually by a user from the deploy screen. If "ON_DEPLOY", the flow will be run immediately after the flow has been deployed and enabled. If "SCHEDULED_CRON", the flow will be run on a scheduled cron basis that can be maintained by double clicking the flow in the list. |
Start Expression |
If Start Type is set to "SCHEDULED_CRON", this is the cron expression that will determine the run time of the job |
Deploying a flow
To deploy a flow to an agent, click on the "Add Deployment" button and the Add deployment screen will be displayed as show below:
All projects and flows will be displayed. Selecting a flow, and clicking the "select" button will deploy the flow to the Metl agent. When a flow is first deployed to an agent, its status will be "Disabled." To enable the flow for the agent, select the flow, and click the "Enable" button. The flow status will be changed to "REQUEST_DEPLOY" and once deployed will be changed to "DEPLOYED."
Running a flow
Once a flow has been deployed to an agent and enabled, the flow can be run from the deploy screen. Select the flow to be run, and click the "RUN" button. After the flow has run, the flow’s run statistics will be displayed as follows.
| Column Name | Description |
|---|---|
Component Name |
The name of the component in the flow |
Thread |
The number of threads that were used when running the component |
Status |
The completion status of the component. Either "DONE", "ERROR", or "CANCEL". If "DONE", the component completed all of its work successfully. If "ERROR", the component finished in error. If "CANCEL" the component was cancelled before it could complete its work |
Msgs Rcvd |
The number of messages this component received during the run of the flow. |
Msgs Sent |
The number of messages this component sent during the run of the flow |
Entities Prcd |
The number of entities processed (if the component takes a model based inbound message) |
Start |
The start run time of the component |
End |
The end run time of the component |
Selecting the component in the top pane will result in run information for that component being display in the bottom pane. Run information for the component can be filtered by typing a filter expression in the Filter box.
Running a flow from an enterprise scheduler
Metl has a REST API that can be used to call a deployed flow from an external enterprise scheduler. Information about the Rest API can be found here Rest API. The Rest service allows an HTTP GET request to run a flow and get a response back that provides the status of the flow run.
One way to initiate the REST API is to create a .bat file using cURL as shown in the example below.
curl -H "Accept: application/xml" http\://[SERVER]\:8080/metl/api/agents/[AGENT]/deployments/[FLOW_NAME]/invoke?paramName=paramValue | grep '<status>DONE</status>'
7. Manage
The Manage screen allows viewing run information for flows. The manage screen is shown below:
The Manage screen is split into two panels, the navigation panel on the left hand side of the screen and the content panel on the right hand side of the screen. The navigation panel allows selection of the flows to be viewed. The following options are available.
| Option | Description |
|---|---|
Currently Running |
Displays all flows that are currently executing on any Metl run-time agent in this Metl instance |
In Error |
Displays all flows that are currently in an error status on any Metl run-time agent in this Metl instance |
Agents |
Allows selection of flows to be viewed based on the run-time agent to which they are deployed. When a specific run-time agent is selected in the navigation panel, all flows that have been run on that agent will be displayed in the content panel. |
Flows |
Allows selection of flows to be viewed based on flow name. When a flow name is selected, every run of the given flow will be listed in the navigation panel. |
Double clicking a row/run in the content panel will display the run statistics for that run of the flow. There are two modes for displaying run statistics for a flow, table and graphical. If the Show Diagram checkbox is checked, a graphical representation of statistics for the flow will be displayed. If it is not checked, a tabular representation of the stats will be displayed. Both representations are shown below.
Tabular
Graphical
In the tabular representation, the following data is available:
| Column Name | Description |
|---|---|
Component Name |
The name of the component in the flow |
Thread |
The number of threads that were used when running the component |
Status |
The completion status of the component. Either "DONE", "ERROR", or "CANCEL". If "DONE", the component completed all of its work successfully. If "ERROR", the component finished in error. If "CANCEL" the component was cancelled before it could complete its work |
Msgs Rcvd |
The number of messages this component received during the run of the flow |
Payload Rcvd |
The number of content messages this component received during the run of the flow |
Msgs Sent |
The number of messages this component sent during the run of the flow |
Payload Sent |
The number of content messages this component sent during the run of the flow |
Entities Prcd |
The number of entities processed (if the component takes a model based inbound message) |
Start |
The start run time of the component |
End |
The end run time of the component |
Selecting the component in the top pane will result in run information for that component being display in the bottom pane. Run information for the component can be filtered by typing a filter expression in the Filter box.
8. Explore
The explore screen is for users to explore deployed resources. Metl currently supports exploring Sql Datasources via a Sql Explorer and Directory based resources via a Directory Explorer.
9. Admin
The administration screens allow setup of admin data including:
-
Users - Managing users
-
Groups - Managing User groups
-
REST - Displaying information about the REST API
-
General Settings - Managing purge, auto backup and password settings
-
Plugin Repositories - Managing maven repositories from which plugins are downloaded
-
Mail Server - Configuring the SMTP mail server
-
Notifications - Configuring notifications
-
Active Users - Displaying active users
-
Audit Events - Displaying system events such as logins, restarts, config imports, etc.
-
Logging - Displaying Metl log information
-
About - Displaying information about the running Metl system
9.1. Users
Selecting "Users" in the Admin navigation panel displays a list of Metl users in the content panel as shown below.
Clicking on the "New" button allows creation of a new user, while clicking on the "Edit" button allows editing of an existing user. In either case the edit user dialog will be display as follows:
| Field | Description |
|---|---|
Login ID |
The ID the user will use to log into the Metl user interface |
Full Name |
The full name for the user. For documentation purposes only |
Password |
The password the user will use to log into the Metl user interface |
Available Groups |
The available groups configured in the "Groups" section that are available to be assigned to the user |
Selected Groups |
The groups to which this user currently belongs |
9.2. Groups
Selecting "Groups" in the Admin navigation panel displays a list of Metl groups in the content panel as shown below.
Clicking on the "New" button allows creation of a new group, while clicking on the "Edit" button allows editing of an existing group. In either case the edit user dialog will be display as follows:
| Field | Description |
|---|---|
Group Name |
The name of the group that will be availble to assign to users |
Available Privileges |
The available privileges in the system. The available privileges are "DESIGN", "DEPLOY", "MANAGE", "EXPLORE", and "ADMIN". Each of these privileges allows the user to access the the corresponding high level screens in Metl and perform the actions on those screens. For example, design privileges allow the user to design new flows. Deploy allws users to deploy existing flows to Metl agents. "Manage" allows users to look at run time data for flows running on the given agent, etc. |
Selected Privileges |
The selected privileges the user group will have. The default user group has access to all privileges. |
9.3. REST
Selecting "REST" in the Admin navigation panel displays a list of Metl Rest Services in the content panel as shown below.
The current REST service that is available for Metl is a service to allow flows deployed on an agent to be run by invoking the REST service. This is used frequently when scheduling a Metl flow with an existing enterprise scheduler. Documentation for the service includes:
-
HTTP Method - GET
-
URL - /api/agents/{agentName}/deployments/{deploymentName}/invoke. {agentName} should be replaced with the agent that hosts the flow to be run, and {deploymentName} should be replaced with the flow name to be run.
-
Parameters - agentName and deploymentName from above
-
Example Response - The response that is sent by the REST service after invoked.
See Running a flow from an enterprise scheduler for an example.
9.4. Mail Server
Metl allows notifications to be sent to an email address or distribution list for certain flow run statuses. See Notifications for details. In order for notifications to be sent, the mail server must be configured. Selecting "Mail Server" in the Admin navigation panel allows the mail server to be configured in the content panel as shown below.
| Field | Description |
|---|---|
Host Name |
The server name on which the mail server resides |
Transport |
The transport over which to connect to the mail server. Either "SMTP" or "SMTPS". SMTPS is secure. |
Port |
The port on which to connect to the mail server |
From Address |
The email address notification emails should be sent FROM |
Use TLS |
Whether to use transport layer security protocol over SMTP to provide certificate based authentication. |
Use Authentication |
Whether the mail server requires authentication |
Username |
The username to be used to connect to the mail server if authentication is required |
Password |
The password to be used to connect to the mail server if authentication is required |
9.5. Notifications
Metl allows notifications to be sent to an email address or distribution list for certain flow run statuses. In order for notifications to be sent, the notifications must be configured. Selecting "Notifications" in the Admin navigation panel shows a list of configured notifications in the content panel as shown below.
Clicking on the "New" button allows creation of a new notification, while clicking on the "Edit" button allows editing of an existing notification. In either case the edit notification dialog will be display as follows:
| Field | Description |
|---|---|
Level |
The level at which this notification applies. Either "GLOBAL", "AGENT", or "DEPLOYMENT". If "GLOBAL" the notification will be sent for all flows and all agents in this instance of Metl. If "AGENT", the notification will be sent for the agent specified by the "Linked To" field, and if "DEPLOYMENT", the notification will be sent for the flow and specified agent in the "Linked To" field. |
Linked To |
If "AGENT" or "DEPLOYMENT" is specified for the "Level" field, this field specifies for which agent or deployment (agent and flow) the notification should fire |
Event |
The event for which the flow should fire. Either "FLOW_START", "FLOW_END" or "FLOW_ERROR". This field is used in conjunction with the "Level" and "Linked To" fields to determine when to fire events. "FLOW_START" indicates a notification should be fired when the flow begins, "FLOW_END" when the flow finishes (either successfully or on error), and "FLOW_ERROR" when the flow finishes with an error condition. |
Name |
The name of the notification |
Recipients |
The recipient list. One or more email addresses may be entered |
Subject |
The subject message that should be displayed for the email notification |
Message |
The message body for the email notification |
Enabled |
Whether this notification is enabled or not. If not enabled, the notification will not be sent |
Notification variables that can be used in Subject or Message
There are several variables that can be used in the notification subject and message. These are specifically helpful when the notification Level is at the Agent or Global level as details can be added that specify which flow fired the notification.
-
$(_flowName) - The name of the flow that fired the notification message
-
$(_agentName) - The name of the agent that fired the notification message
-
$(_date) - The date the notification message was fired
-
$(_time) - The time the notification message was fired
9.6. Logging
Selecting "Logging" in the Admin navigation panel shows Metl’s log file as shown below.
The log file can be filtered or downloaded by using the appropriate fields on the logging screen.
Unresolved directive in user-guide.asciidoc - include::examples/examples.asciidoc[] === Creating RESTful Services Metl can be used to create and host RESTful services. Creating a RESTful service in Metl is as simple as creating a flow and deploying it to one of the configured agents. Metl comes with several sample flows that show the creation of RESTful services. The samples can be reviewed and run from the Samples project.
The core of Metl’s service capabilities come from the Http Request and Http Response components.
Following are instructions on how to create and host a REST service that lists all persons from a relational database. Metl REST services are created by configuring a Metl flow to accept the Http request, using other components to complete the logic of the flow, and then using the Http response component to return the results to the caller.
The first step in creating the sample REST service is to create the resource that will be used to gather the person data from the database. For this sample, an in memory H2 database will be used and is configured as follows:
Next, a model must be created to represent a person. In this example, the structure of the data of a person will be the same in the relational database source and the response of the REST service, so the same model will be used for both. Different models for each could be used and mapped within the flow, but for this sample, we’ll keep things simple. The person model looks as follows:
Now that our source resource and model are defined, the flow can be created that handles the serivce logic. The completed flow looks as follows.
First, let’s go through the non-service aspects of the flow. The SQL Executor labeled "Setup Person Database" is used in the sample flow to create the person table and populate it with data. The Gate component labeled "Ensure DB Setup Completes" is used to control the order of execution of the flow when an HTTP response triggers the flow to begin. It simply ensures SQL Executor "Setup Person Database" completes before reading the data from the database itself. In most scenarios, these two steps would not be necessary as the database where the data resides most likely already exists. These two steps are included in the sample simply to make it run stand-alone without the need for predefined database resources.
Next, let’s look at the Http Request component and its associated properties.
Each property above is described in the Http Request component definition. For this example, the "HTTP Method" of type "GET" and the "Path" of "/person/getAll" tells the flow to listen for HTTP GET requests to a URL of http://myserver:port/metl/api/ws/person/getAll, where "myserver" and "port" are the server and port on which Metl is running. If Metl is running locally on its standard port of 42000, HTTP GET requests to http://localhost:42000/metl/api/ws/person/getAll would be handled by the flow.
Once the request hits the flow, the "Setup Person" SQL Executor would run and setup the database. The "Ensure DB Setup Completes" Gate would ensure the SQL Executor is finished setting up the database before sending the HTTP request to the RDBMS Reader labeled "Read From Database". The RDBMS Reader component is set up as follows:
Note the output model of "Person" and the Sql of select * from person. The RDBMS Reader will simply read all rows from the person table and return
the results based on the Person model. Once the data has been read from the database, it will be forwarded to the Serializer component configured
as follows:
Now that the response has been formatted it send to the Http Response configured as follows:
Each property above is described in the Http Response component definition.
The following shows the results of a GET request to the service with an Accept header of application/xml.
<ArrayList> <item> <name>PERSON</name> <data> <GENDER>M</GENDER> <ID>1</ID> <LAST_NAME>Arbuckle</LAST_NAME> <FIRST_NAME>Garfield</FIRST_NAME> </data> </item> <item> <name>PERSON</name> <data> <GENDER>M</GENDER> <ID>2</ID> <LAST_NAME>Arbuckle</LAST_NAME> <FIRST_NAME>Odie</FIRST_NAME> </data> </item> <item> ... </ArrayList>
With an Accept header of application/json, the result would look as follows:
[
{
"name": "PERSON",
"data":
{
"GENDER": "M",
"ID": "1",
"LAST_NAME": "Arbuckle",
"FIRST_NAME": "Garfield"
}
},
{
"name": "PERSON",
"data":
{
"GENDER": "M",
"ID": "2",
"LAST_NAME": "Arbuckle",
"FIRST_NAME": "Odie"
}
}
...
]
If different XML or JSON formatting is desired, the flow can be configured to format entity based data into XML or JSON using the XML Formatter or custom scripts and the resulting text messages from the formatters can be passed to the HTTP Response component. The HTTP Response component will return the pre-formatted text (XML, JSON, or other) accordingly.
| If the flow formats its own output, the HTTP Response component should be configured to communicate the content-type to the caller. This can be done by setting the HTTP Response "Content Type" accordingly (i.e. application/xml, etc.) |
In order for a REST service flow to be used (run), it must be deployed to an existing Metl agent. See the Deploy section for details on creating agents and deploying flows.
To test the flow in the design mode, use the Run button and the flow will be auto-deployed to the design agent and a user interface will be presented for testing the web services.
