Orchestration is the ability to control operational flows and activities based on business rules, especially in multi-application systems complicated enough to require middleware such as ESB (“Enterprise Service Bus”) or the older MOM (“Message-Oriented Middleware”).
In the context of a file transfer system, orchestration often refers to the ability to apply automation such as triggers, schedules, explicit calls and chained calls to model or solve a business problem.
In the context of SOA (“Service Oriented Architecture”), orchestration typically refers to the ability of programmers to rapidly develop composite applications due to the fact that most available application APIs have been encapsulated and published in reliable directories that programmers’ applications can easily interpret and use.
BEST PRACTICE: Orchestration typically invokes an image of “drag and drop” business application development: a task easy enough for the average business analyst. Reality often requires more than that: shelling out to scripts, editing raw XML documents by hand and having to clean up “orchestrated code” after an incompatible interface is rolled out are still common issues.
Middleware is a software architecture concept that refers to integration of disparate applications to facilitate reliable communication. Middleware frequently relies on encapsulating inter-application communications in the concept of an “message”, and often has the ability to queue or perform optimized delivery or copying of messages to various applications.
Common types of middleware include EAI (“Enterprise Application Integration”) middleware such as ESB (“Enterprise Service Bus”) or the older MOM (“Message-Oriented Middleware”).
File transfer applications are themselves often used as middleware, helping to facilitate bulk data transfers between applications using standards such as FTP. Managed file transfers often include the ability to perform some intelligent routing of data and sensitivity to particular transmission windows set by the business.
In file transfer operations, a cut-off time is a specific time of day a processor must receive a batch or file by so processing can begin on that day. The processor, not the sender, decides the cut-off time.
For example, if a processer publishes a cut-off time of 5pm, then a file received at 4:59pm will be processed today, but a file received at 5:01pm may not. (Special dispensation is often granted to particular files on an operator-by-operator basis.)
See also: transmission window
The term “FTP with PGP” describes a workflow that combines the strong end-to-end encryption, integrity and signing of PGP with the FTP transfer protocol. While FTPS can and often should be used to protect your FTP credentials, the underlying protocol in FTP with PGP workflows is often just plain old FTP.
BEST PRACTICE: (If you like FTP with PGP.) FTP with PGP is fine as long as care is taken to protect the FTP username and password credentials while they are in transit. The easiest, most reliable and most interoperable way to protect FTP credentials is to use FTPS instead of non-secure FTP.
BEST PRACTICE: (If you want an alternative to FTP with PGP.) The AS1, AS2 and AS3 protocols all provide the same benefits of FTP over PGP, plus the benefit of a signed receipt to provide non-repudiation. Several vendors also have their own way to provide the same benefits of FTP with PGP without onerous key exchange, without a separate encrypt-in-transit step or with streaming encryption; ask your file transfer vendors what they offer as an alternative to FTP with PGP.
SLA is an abbreviation for “Service Level Agreement“, which is a specific contract between a customer and a provider that lays out exactly what each side can expect from the other. The minimum amount of work and minimum level of due care that a file transfer operations team is responsible for is often determined by the SLAs it must meet.
See “Service Level Agreement” for more information.
A transmission window is a window of time in which certain file transfers are expected or allowed to occur.
Transmission windows typically reoccur on a regular basis, such as every day, on all weekdays, on a particular day of the week, or on the first or last day of the month or quarter.
Most transmission windows are contiguous and set on hourly boundaries (e.g., from 3:00pm to 11:00pm) but can also contain breaks (e.g., 3-6pm and 7-9pm) and start/end on minute/second boundaries (e.g., from 3:05:30pm to 7:54:29).
Files received outside of transmission windows are not immediately processed or forwarded by file transfer systems. Instead, they are typically stored or queued, and are usually processed when the transmission window opens back up. (e.g., a file received at 7:58am for an 8am-2pm transmission window would be processed today at 8:00am, however a file received at 2:02pm would be processed tomorrow at 8:00am.)
When transmission windows are coupled with specific types of file transfers, service level agreements (SLAs) can be written to lock down expectations and reduce variability.
See also: cut-off time.
BEST PRACTICE: When possible, select file transfer scheduling technology that allows you to maintain transmission windows separate from your defined workflows. For example, if you want to change the transmission window for a particular type of file across 50 customer workflows, make sure you can do so by only changing one window definition, not 50 workflow definitions. Also look for technology that allows you to see and control the contents of queued files received outside of transmission windows. Your operators may want to allow, roll over or simply delete some of the files received outside any particular transmission window.
An operational level agreement (OLA) is a less stringent form of service level agreement (SLA) typically set up between two departments in the same organization, especially when an OLA is set up to help support a customer-facing SLA.
See “Service Level Agreement” for more information.
A file transfer service level agreement (SLA) establishes exactly what a particular customer should expect from a particular file transfer provider, and how that customer should seek relief for grievances.
A file transfer SLA will often contain the following kinds of service expectations:
Availability: This expresses how often the file transfer service is expected to be online. An availability SLA is often expressed as a percentage with a window of downtime. For example: “99.9% uptime except for scheduled downtime between 2:00am and 5:00am on the second Sunday of the month.”
Different availability SLAs may be in effect for different services or different customers. Availability SLAs are not unique to file transfer; most Internet-based services contain an availability SLA of some kind.
Round-Trip Response Time: This expresses how fast a complete response to a submitted file will be returned. A round-trip response time SLA is often expressed as a certain length of time. For example, “we promise to return a complete response for all files submitted within 20 minutes of a completed upload”. Sometimes a statistical percentage is also included, as in “on average, 90% of all files will receive a completed response within 5 minutes.”
The reference to “round-trip” response time rather than just “response time” indicates that the time counted against the SLA is the total time it takes for a customer to upload a file, for that file to be consumed and processed internally, and for any response files to be written and made available to customers. Simple “response time” could just indicate the amount of time it would take the system to acknowledge (but not process) the original upload.
Different round-trip response times SLAs may be in place for different customers, types of files or times of day. Round-trip response time SLAs are similar to agreements found in physical logistics: “if you place an order by this time the shipment will arrive by that time.”
Completed Body of Work: This expresses that a particular set of expected files will arrive in a particular window and will be appropriately handled, perhaps yielding a second set of files, within the same or extended window. For example, “we expect 3 data files and 1 control file between 4pm and 8pm everyday, and we expect 2 response files back at any time in that window but no later than 9pm”
Files in a body of work can be specified by name, path, size, contents or other pieces of metadata. There are typically two windows of time (“transmission windows“) associated with a body of work: the original submission window and a slightly larger window for responses.
SLAs can be set up between independent partners or between departments or divisions within an organization. A less stringent form of SLA known as an operating level agreement (OLA) when it is between two departments in the same organization, especially when an OLA is set up to help support a customer-facing SLA.
BEST PRACTICE: A good file transfer SLA will contain expectations around availability and either round-trip response times or expected work to be performed in a certain transfer window, as well as specific penalties for failing to meet expectations. File transfer vendors should provide adequate tools to monitor SLAs and allow people who use their solutions to detect SLA problems in advance and to compensate customers appropriately if SLAs are missed.
In file transfer, “metadata” usually refers to information about files moved through a file transfer system. Examples of metadata include usernames of original submitter, content types, paths taken through the system so far and affirmations of antivirus or DLP checks.
Metadata such as suggested next steps is often submitted to file transfer applications in control files. However, most metadata is typically collected during a file’s flow through a file transfer system. (All the metadata examples above are examples of passively collected metadata.)
File transfer applications often use metadata in their configured workflows to make runtime decisions. (e.g., A workflow engine may be configured to send files from two different users to two different destinations.)
Metadata is often stored in the status, workflow and log databases used by file transfer applications. When these data stores are proprietary or inaccessible integrating metadata from multiple applications can be challenging.
Explicit file attributes such as file size, file name, current location on disk and current permissions are not typically considered metadata. The reason these attributes are not considered metadata is because they are required by almost every operating system; by definition metadata is extra data used to provide additional context for each file.
A control file is a special file that is sent along with one or more data files to tell applications that handle the data files how to handle them. Control files are typically created by the same application that original sends files into a file transfer system.
The most common type of control file is a “trigger file“. Trigger files are used to initiate further processing or retransmission of files already uploaded to or present on a separate file transfer system.
Other types of control files are often used to communicate final file names, intermediate processing applications, conditions of transfer/processing success or other metadata.
There is no standard format for control files, but many control files are either simple “flat files” with one file configuration entry in each line or simple XML files.
There are three general expectations made on file transfer systems by applications that submit control files.
Send and Forget (No Control File): When applications send files into a file transfer system they rely on the file transfer system to correctly determine when files can safely be processed or retransmitted. This is often a safe assumption, especially when each file can be processed on its own and each file can be processed immediately.
Send and Commit: The use of a control file allows applications to commit a group of transmitted files to final acceptance, further processing or retransmission. The use of a control file is often used here when two or more files need to be submitted together to allow final acceptance, further processing or retransmission.
Send, Commit and Confirm: If a file transfer application returns a status code or file after acting on a submitted control file then it can confirm both the original files sent and the contents of the control file. The use of the extra confirmation step is often used when the submitting application wants to check up on the transfer status or history of its submitted files. (In this case, either the submitted control file or resulting confirmation file will contain an ID the submitting application can use to perform lookups after the initial confirmation.)
A “trigger file” is a common type of control file used to initiate further processing or retransmission. Trigger files are typically created by the same application that original sends files into a file transfer system.
The two most common types of trigger files are files with similar names to the files that need to be sent and files that contain the names of files that need to be sent. Examples of two of the most common types of trigger files are shown below.
Similar Name Trigger File Example: Two files named “textrequest_20110608.xml” and “textrequest_20110608.tiff” are sent into a file transfer system. A third trigger file called “textrequest_20110608.trg” is then uploaded to tell the system to process or send the two “xml” and “tiff” files bearing similar names.
List of Files Trigger File Example: Two files named “textrequest_20110608.xml” and “textrequest_20110608.tiff” are sent into a file transfer system. A third trigger file called “trigger24235.txt” containing the names of the “xml” and “tiff” files is then uploaded to tell the system to process those specific files.
See “control file” for more information.