Extend your MFT to exchange files with any cloud storage: A Step-by-Step Guide to build custom MFT connector

Build MFT custom connector

With the increasing pervasiveness of applications in the cloud, your file exchange platform is likely to face additional requirements to exchange with different cloud storage solutions. Keeping your file exchange platform current with ever-increasing and evolving file storage options is a significant challenge. Examples of the file storage ecosystem include Amazon Web Services (AWS) S3, Azure Blob, Axway’s Syncplicity, OneDrive, Box, Dropbox, Google Drive, and the list goes on. So how do you address this? First, choose a data exchange platform that provides the agility and flexibility to extend the protocol framework so you can build your own connectors to different cloud storage solutions.

Axway’s Managed File Transfer (MFT) gateway solution—AMPLIFY SecureTransport provides several connectors to exchange files over different protocols and can be accessed via the Axway Marketplace.

However, as hard as Axway’s R&D tries to keep current, we experience the pain of this being a constant moving target. To address this, AMPLIFY SecureTransport comes packaged with the SDK and developer guide to help our customers build their own pluggable transports in a simple and safe way. This provides a client community to create their own connectors and contribute to the MarketPlace as an Open-source. Read more about an Integration Use Case for SecureTransport + Syncplicity.

Recently, I was involved in a project where I needed to exchange files with Microsoft OneDrive. Unfortunately, Microsoft OneDrive was not one of the available transports in the marketplace. So, I built a OneDrive connector using the SecureTransport framework. In this blog, I am going to take you through my journey to build the connector so that you have a kick start whenever you need to build a connector.

Understanding of your Cloud Storage

Every cloud storage comes with a set of APIs. Using those APIs you can access your files, put a file in your storage, etc. Visit the cloud storage developer portal and explore the APIs. In this case, I have explored Microsoft graph APIs to understand how the APIs work.

Microsoft Graph is the gateway to data and intelligence in Microsoft 365. It provides a unified programmability model that you can use to access the data in Office 365, Windows 10 and Enterprise Mobility + Security. The Microsoft Graph API offers a single endpoint, https://graph.microsoft.com to provide access to rich people-centric data and insights exposed as resources of Microsoft 365 services. You can use REST APIs or SDKs to access the endpoint and build apps that support scenarios spanning across productivity, collaboration, education, security, identity, access, device management and much more.

Microsoft Graph exposes REST APIs and client libraries to access data on the following Microsoft 365 services:

  • Office 365 services: Delve, Excel, Microsoft Bookings, Microsoft Teams, OneDrive, OneNote, Outlook/Exchange, Planner and SharePoint
  • Enterprise Mobility and Security services: Advanced Threat Analytics, Advanced Threat Protection, Azure Active Directory, Identity Manager and Intune
  • Windows 10 services: activities, devices, notifications
  • Dynamics 365 Business Central
Microsoft Graph connector

To know more about Microsoft Graph, check out the link.

You can use Microsoft Graph Explorer to play with some popular Microsoft Graph API. You can log in to your own Microsoft account, obtain an access token and use the access token to make API call and inspect the request, response.

Identify the APIs

In this step, you have to identify the APIs which you are going to use in your connector. Here is the list of APIs I have identified for my OneDrive connector.

  1. An access token:  https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token
  2. Get Userid from email: https://graph.microsoft.com/v1.0/users
  3. Get DriveId : https://graph.microsoft.com/v1.0/users/{userId}/drive
  4. Get File Path: https://graph.microsoft.com/v1.0/drives/{driveId}/root:{downloadFolder}:/children
  5. Download the File: URL returned in the attribute “@microsoft.graph.downloadUrl” of the previous request.

What is needed to make the API call?

Typically, you need a client id and the secret to make the API call. In this case, I need the Azure tenant id. So, I registered a demo app in Microsoft Application Registration Portal and received my client id, secret and tenant id.

Demo

Write the code

Now you are all set to build the connector. Secure Transport connector is java based. I have used Eclipse as IDE for java code development. To start with an existing project, you can clone my connector project from GitHub and make changes as required for your own connector.

So, do a git clone:

git clone https://github.com/KrishanuMaity76/SecureTransportOneDriveConnector.git

Import the code in your IDE, as you see it is a very simple J2EE project. I am using Gradle to build the project. You can you Maven as well. Add your Gradle dependency in the “build.gradle” file.

First of all, you have to define the custom site’s java bean, so I created a OneDriveBean.java which contains all the attributes I needed to carry from user input in the UI.

public class OneDriveBean implements UIBean{

    private String onedrive_siteName;
    private String onedrive_networkZone;
    private String onedrive_downloadFolder;
    private boolean onedriveDownloadExpression;
    private String onedrive_patternType;
    private String onedrive_downloadPattern;
    private String onedrive_appId;
    private String onedrive_secret;
    private String onedrive_tenantId;
    private String onedrive_userEmail;
    private boolean listFileFolderAdvancedExpressionEnabled;
    private boolean listFilePatternAdvancedExpressionEnabled;
    private String listPatternType;

//Add the corresponding getter and setter methods.
}

Then write the custom site’s java class, for example, OneDriveSite.java which extends CustomSite.java. In this class, you have to override several methods for the connector to work.

@Override
public void getFile(DestinationFile file) throws IOException {
//This method is responsible to get the file from the remote site
}

public List<FileItem> list() throws IOException {
//This method is implemented to list the files at the reomte site.
}

@Override
     public void putFile(SourceFile file) throws IOException {
//This method is implemented to put the file in the onedrive upload directory.
}

So, when you are building a new connector you have to implement the above methods. Depending on your implementation These methods can call other private methods like below methods,

private String getUserID(String accessToken, String email) throws IOException{
//This method returns onedrive user is based on the user email.
}
private String getDriveID(String accessToken, String userId) throws IOException{
//This method returns user's onedrive id based on the user id.
}
private List<FileItem> getFilePaths(String downloadFolder, String driveId, String accessToken) throws IOException{
//This method is called to get files with file name, file path, etc.
}
private String getAccessToken(String clientId, String secret, String tenantId) throws TransferFailedException{
//This method calls onedrive oauth API to get the access token.
}

Now you have to design custom site’s HTML page which is the easy part. Refer OneDriveSite.html, it has a <table> with bunch of <tr>, keep adding <tr> as per your need.

The most important thing in the HTML is to include the JavaScript with the right plugin name

<script src="/coreadmin/servlet/pluginContent/com/axway/st/plugins/site/onedrive/js/OneDriveSite.js?pluginName=MicrosoftOneDrive" />

For example, I have added “pluginName=MicrosoftOneDrive” and I used the same plugin name in the Plugin-info.yaml.

Plugin-info.yaml is the property file where you have to mention the plugin name( which comes as a drop-down in the transfer site protocol list), connector site java class package, HTML path, classpath, etc. My Plugin-info.yaml looks like

Custom-Protocol: MicrosoftOneDrive
SPI-Version:
  '1.1':
    Custom-Site: com.axway.st.plugins.site.onedrive.OneDriveSite
    Protocol-Label: Microsoft OneDrive
    Custom-UI: com/axway/st/plugins/site/onedrive/html/OneDriveSite.html
    Class-Path: MicrosoftOneDrive/libs/jsch-0.1.55.jar MicrosoftOneDrive/libs/commons-net-3.6.jar MicrosoftOneDrive/libs/json-20140107.jar MicrosoftOneDrive/libs/okhttp-3.12.1.jar

After configuring the connector in Secure Transport, in the transfer site page the connector looks like below. You have to write HTML code only for the red dotted box. Everything else in the transfer site will magically appear.

User Account

Now, you have to work on the JavaScript. For example, I have created OneDriveSite.js. This JavaScript you have to implement two ajax call.

This ajax makes a call to Secure Transport API & retrieves transfer site details and populate the page. So for the first time, siteId will be null and no ajax call will be made. Page loads with no data.

function load(accountName, siteId) 
    if (siteId) {
        var selectedCertificateURL = null;    
        $.ajax({
                url : "/api/v1.4/sites/" + siteId,
                type: 'GET',
                dataType : 'json',
                cache: false,
                success : function(data) {
                    if (data && data['protocol'] === 'MicrosoftOneDrive') {
                    //MicrosoftOneDrive is the connector name used in the yaml.
   .
   .
   .

This ajax makes a call to Secure Transport API and saves the transfer site details.

function save(siteData, callback) {
        setCheckBox("onedriveDownloadExpression");
        if (siteData) {
         //populate input data to store as site data.
            var customProperties = {
                 "onedrive_appId": $('#onedrive_appId').val(),
                "onedrive_secret": $('#onedrive_secret').val(),
                "onedrive_tenantId": $('#onedrive_tenantId').val(),
                "onedrive_networkZone": $('#onedrive_networkZone').val(),
                "onedrive_downloadFolder": $('#onedrive_downloadFolder').val(),
                "onedriveDownloadExpression": $('#onedriveDownloadExpression').val(),
                "onedriveDownloadFilePattern": $('#onedriveDownloadFilePattern').val(),
                "onedriveDownloadType": $('input:radio[name=onedriveDownloadType]:checked').val(),
                "onedrive_userEmail": $('#onedrive_userEmail').val()
            },
            siteUrl = "/api/v1.4/sites",
            .
            .
            $.ajax({
                    url : siteUrl,
                    type: 'POST',
                    dataType : 'json',
                    contentType : 'application/json',
                    data: JSON.stringify(data),
                    success : function() {
                        if (callback) {
                            callback();
                        }
            .
            .
            .
}       

Build and Deploy

Open your command prompt, navigate to “gradlew.bat” and run the file. You should see build logs and “Build Successful” message in the command prompt. Navigate to the “\build\distributions” directory and you should see the connector ZIP file. Now, move the ZIP file to the plugin directory under the Secure Transport home directory.

  1. Stop Secure Transport
  2. Unzip the connector distributions
  3. Start Secure Transport

Concluding Comments

Building the pluggable transport (connector) took me less than five days. Granted, you need Java development skills to make this happen but Axway does make this simple for our customers. If you are not comfortable building transports yourself, you should contact your account executive or Axway support. We have a team within Axway who can help you build these transports and they are not tied to regular SecureTransport release cycles.

Building pluggable transports are safe to do and when you come to upgrade to newer editions of SecureTransport, the pluggable transports will not break.

If you experience issues with your pluggable transports after an upgrade, contact Axway support and they will help you resolve the issue.