Metl User Guide

Version 3.8.8

Permission to use, copy, modify, and distribute this Metl User Guide for any purpose and without fee is hereby granted in perpetuity, provided that the above copyright notice and this paragraph appear in all copies.

Preface

This user guide introduces Metl, a simple, web-based integration platform that allows for several different styles of data integration including file based Extract/Transform/Load (ETL), messaging, and remote procedure invocation via Web Services. This guide is intended for users, developers, and administrators who want to install the software, configure integrations, and manage its operation. Thank you to all the members of the open source community whose feedback and contributions helped us build better software and documentation. This version of the guide was generated on 2022-01-21.

1. Introduction

Metl is a simple, web-based integration platform that allows for several different styles of data integration including messaging, file based Extract/Transform/Load (ETL), and remote procedure invocation via Web Services. Metl was built to solve fairly simple day to day integration tasks without the need for custom coding, heavy infrastructure, or high costs. It can be deployed in the cloud or in an internal data center, and was built to allow developers to extend it to fit their needs by writing their own components that can be included and leveraged by the existing Metl infrastructure.

1.1. Overview

Metl is a web application. It can be deployed as a web application archive (war) file to an application server (i.e. Tomcat, JBoss, Websphere) or can be deployed stand-alone as a war file deployed under a bundled Jetty Servlet container. See installation section for details.

Once installed, the web application is accessed via a web browser and is used to design, deploy and manage integrations.

Integrations can be grouped or packaged into one or more projects. Projects are a means to logically group integrations together in some logical way. As an example, a project might contain all of the integrations in or out of a specific system. See projects for additional details.

Projects contain Flows, Models and Resources.

Flows are the integrations themselves. That is, flows allow the definition of data be retrieved from source systems, transformed, and written to target systems. Flows are graphical in nature and allow the developer to configure data movement.

Flows are constructed of components connected by links that describe the path of data through the flow. There are various components available, and custom components can be built and integrated in to the existing Metl infrastructure.

See flows for additional details.

Models provide structure to your data. When dealing with structured data, the data can be defined by modeling. Metl Models allow you to describe your data. Models can either be relational or hierarchical in nature. See models for additional details.

Resources represent connections to physical endpoints where data is read from or written to. The following resource types are available in Metl:

Database - A JDBC connection to a JDBC compliant relational database
Directories - Connections to file systems. Local, FTP, SFTP and SMB are all supported
HTTP Resource - An HTTP connection to REST or SOAP based services
Mail Session - An SMTP connection that can be used to send and receive email
Subscribe - A connection to a JMS queue

See resources for additional details.

1.2. Use Cases

Flat File Data Integrations
RDBMS Data Integrations
Pub / Sub (Queue based) Data Integrations
API / Service Based Data Integrations

1.3. Why Metl?

Integrations can be configured in hours vs coding them in days or weeks
Common infrastructure code all taken care of (reading, writing, formatting, parsing, threading, etc.)
Easy to install. Simply download the .war file and run
Web based. No tools to install on each desktop. Deploy locally, in your data center or in the cloud
Doesn’t require heavy infrastructure. Any windows or linux box will do
Open. No black box. Download the source. Write your own components if desired
Proven in large integration scenarios
Supported. JumpMind, Inc. provides professional support and services for any need for both the open source and commercial versions of Metl

1.4. System Requirements

Metl is written in Java and requires a Java Runtime Environment (JRE) Standard Edition (SE) or Java Development Kit (JDK) Standard Edition (SE). Most major operating systems and databases are supported. The minimum operating system requirements are:

Java SE - Runtime Environment or Development Kit version 8 or above
Memory - 1 (GB) available
Disk - 256 (MB) available

Metl is accessed from a web console, which requires one of the following supported web browsers:

Google Chrome 23 or newer
Internet Explorer 8 or newer
Mozilla Firefox 17 or newer
Safari 6 or newer

1.5. License

METL is available in both an open source and commercial offering. The open source version is free software licensed under the GNU General Public License (GPL) version 3.0. The commercial version is licensed by JumpMind as part of a Support Subscription Agreement from JumpMind. See <<http://www.gnu.org/licenses/gpl.html,GPL License>> and <<http://www.jumpmind.com/products/metl/overview,Metl Commercial>> for details. This project includes software developed by JumpMind and a community of multiple contributors. METL is licensed to JumpMind as the copyright holder under one or more Contributor License Agreements. METL and the METL logos are trademarks of JumpMind. == Installation

Metl is a java, web based application. It comes in both an open source and commercial version. Regardless of the version being used, Metl is simple, easy and lightweight to install. The computer on which you are installing must have Java 8 or higher. Both versions of metl may be deployed with the embedded, stand-alone Jetty server, or deployed to an existing servlet container (i.e. Tomcat, JBoss, Websphere, Oracle Application Server, etc.)

For details on installing the open source version, click here.

For details on installing the commercial version, click here.

2. The Metl User Interface

Accessing the Metl User Interface

Once Metl has been started, it can be accessed via a web browser as follows:

http://myservername:myport/metl

If Metl is running on the default port (42000) on the local machine, the url would be as follows:

http://localhost:42000/metl

The following home screen will be displayed:

home

Design - Design and Develop Integrations
⭐ Release - Create release packages for versioning and deployment to a Metl run time agent
⭐ Deploy - Deploy integrations to a Metl run time agent
Manage - Manage integrations that have been run on a given agent including looking at past executions, etc.
Explore - Explore resources that are configured for this Metl instance
Admin - Administrative management of Metl including defining system users, managing logging levels, etc.

In addition to these five major navigation points, the navigation menu also allows for:

Help - Access this help documentation
Logout - Log the current user out of the Metl UI

3. Design

The Design screens are used to design and develop integrations. The main design screen is shown below.

design

The design screen is split into two main panels, the navigation pane and the content/editor pane. The navigation pane shown below allows for traversing projects, flows, models and resources.

navigation panel

When Metl is first installed, it comes installed with a sample project that includes sample flows, models and resources. These samples allow the user to immediately see Metl in action. In addition, these sample flows are referenced in the component documentation in order to better understand how individual components work.

The content/editor pane is where flows, resources and models are edited. The screenshot below shows a database resource being edited.

content editor panel

3.1. Projects

Projects are a means to group integrations together in some logical way. Typically, the boundaries of a project should equate to a deployable unit. That means the scope or context of the project should be those flows that must be bundled together when deployed. I.E. if two flows depend on each other and are so tightly coupled that changes to one dictate changes in the other, then those flows should be bundled together in a single project. The project is the level at which flows and their associated resources and models are versioned (see Versions) and imported/exported (see Import/Export).

A new project can be added via the menu.

projects

3.1.1. Versions

Projects can have multiple versions. When a project is first created an initial version is also created with name master. After the master project is released (deployed to production), future changes to the project can be completed in a new version such that each production migration of the project can be tracked. By making each production release of a project under a new project version visibility to differences between versions is also available. New versions of a project can be created from any subsequent version. I.E. if you have versions 1.0.0, 1.1.0, and 1.2.0, a new version can be created from ANY of the previous 3 versions. A patch release (1.1.1) version can be created from the 1.1.0 version while simultaneously making changes to the 1.2.0 version for some future release date. This is very similar to branching and merging concepts that are available in most source control systems.

To create a new version of a project from an existing version, first select the source version from which the new version should be created and then click File New Project Version as shown below.