The CA API Gateway allows you to stand up an SFTP endpoint for inbound messages to be processed. These messages are instantly processed by the gateway in memory, so throw away everything you knew about SFTP file uploads.

Understanding the conversation flow

When you connect to an SFTP server to upload a file, 5 things happen – note this reflects batch operations; interactive sessions may differ:

1. User/service connects to the SFTP server specifying credentials via the command line (i.e. username@hostname.domain.com – and likely using an SSH key)
2. Change directories (optional)
   1. cd /directory/name
   2. stat /directory/name
3. PUT filename
   1. put filename
   2. stat filename

What’s not obvious, or documented, is that each one of these functions (specifically SFTP command responses) must be explicitly implemented in your API policy.

First Things First – Create a Listener

SFTP uses the SSH2 protocol with a different shell on the gateway. The CA API Gateway implements its own SFTP handler to effectively intercept SFTP commands as requests, and allows these to be routed to policies on the gateway.

A listener must be created first, to allow this to occur. Once you create the listener, specifying the ports and supported SFTP commands, you have two options:

• Use Service Resolution to map requests to policies
  
• Select a policy that will handle all requests to the listener

In my testing, the first option never, ever worked. It’s easier to pretend CA never gave us the option and select a policy.

User Authentication

This is the easy one. By simply adding the “Require SSH Credentials” and “Require User Authentication” assertions, you can authenticate a specific user, such that the rest of the API policy will not function if you haven’t passed this simple entry requirement.

The gotcha? Well, if you connect to an SFTP listener without specifying credentials, you seemingly have a full SFTP connection. But without authenticating, it doesn’t do anything anyway (i.e. LIST/PUT commands will simply fail).

Handling the STAT command

This is the one that will give you the most agony. The STAT command is issued automatically by the SFTP client in the following scenarios

• After you change directory (cd)
  
• After you put a file (put)

The reason this is tricky is two fold; each time a command is issued, it generates a new request to your policy, which in turn means that the commands are stateless.

The purpose of the STAT command, at least according to the RFC959, is to return status of a file:

STATUS (STAT)

            This command shall cause a status response to be sent over
            the control connection in the form of a reply.  The command
            may be sent during a file transfer (along with the Telnet IP
            and Synch signals--see the Section on FTP Commands) in which
            case the server will respond with the status of the
            operation in progress, or it may be sent between file
            transfers.  In the latter case, the command may have an
            argument field.  If the argument is a pathname, the command
            is analogous to the "list" command except that data shall be
            transferred over the control connection.  If a partial
            pathname is given, the server may respond with a list of
            file names or attributes associated with that specification.
            If no argument is given, the server should return general
            status information about the server FTP process.  This
            should include current values of all transfer parameters and
            the status of connections.

When this is issued after the “cd” command, it is to verify that the folder is valid.

When this is issued after the “put” command, it is to verify the file was uploaded.

For file PUTs this obviously creates an issue whereby however you process the file upload, needs to be able to return a response to a second request, asking for the status of the first.

Given we are uploading files, the filename in the request can and should be returned in the response to allow the SFTP client to gracefully admit that its attempt was successful.

To obtain the status of files uploaded, it, therefore, makes sense to query the backend system for a list of files that have been previously uploaded; simply returning an XML formatted file list will do the trick (of course, CA’s undocumented format can be seen below). Responding to the STAT command when the directory is being changed, however, is where we came unstuck.

Three options here:

1. don’t use directories (instead, connect, and upload files directly to the root of the SFTP host, i.e. “/”)
   
2. handle the directory change with a hardcoded STAT response which simply returns the directory name being called for
3. handle the directory change by routing to your backend to actually return a response

Handling the PUT Command

This one almost doesn’t deserve it’s own heading. The file uploaded to the SFTP listener is gobbled up by the listener, mapped to the policy, and then you can do normal policy things with it.

Dealing with XML files? Just set the content type on the listener to text/xml and then you can route it via HTTP like you would any other RESTful API on the gateway. ASCII or Binary files can also be uploaded – just make sure you encode them properly within your policy, using Base64 or similar, before routing them to backend services.

One of my current projects follows a workflow where the file is uploaded, the file contents decrypted with a known key, scanned for viruses, base64 encoded, and then finally sent to a backend service to process the request.

Of course, this is followed by the SFTP client issuing the STAT command for the uploaded file, and an XML file list response is returned from a backend service, which the gateway translates into an RFC compliant STAT response to the SFTP client! Glorious!

Format of the STAT / LIST response

The response is in XML format, and the SFTP listener on the gateway will transform this XML policy response into a valid SFTP response to the client. Note that this can be dynamically generated by your backend system, OR, hardcoded within the policy for some specific use cases.

<?xml version="1.0" encoding="utf-16"?>
<files>
  <file file="false" lastModified="0" name="thisIsADirectory" permissions="755" size="0" />
  <file file="true" lastModified="0" name="thisIsAFile.xml" permissions="755" size="600" />
</files>