Deploy Utilities and Reference Data using Fabric Archives

Example: Deploy a Utility in a FAR

Deploying a utility for use in a pipeline is a reasonably straightforward task. Gather the executable and any supporting files in a directory.

For example, consider deploying the Sequence Alignment/MAP (SAM) Tools. SAM Tools provide various utilities for manipulating alignments in the SAM format, including sorting, merging, indexing and generating genomic alignments. In this case, it is being deployed with a helper perl script.

To package the script and the executable, an XML file will need to be created that installs the tools on each worker in the fabric. Consider the following XML:

samtools-0_1_14.xml

• The far element is the base element and is provided with a description, display-name, name and version.
  • Only the name and version are required.
  • The name is a unique name (or ID) given to an item in its definition XML. It is an arbitrarily long, case sensitive sequence of one or more alphanumeric characters (upper or lower case), hyphens and underscores. Pattern:[A-Za-z0-9-_]+
  • The fabric uses the version value to ensure the most current version of a file is maintained. All version numbers are right justified and zero filled. When comparing version 1.2 with 1.19, the fabric actually compares 1.02 with 1.19. Version 1.19 is considered newer than version 1.2, but 1.20 is considered to be newer than 1.19. Also, when comparing version 1.0.0 with 1.0 and 1, 1.0.0 is considered newer than version 1.0 and 1.0 is considered newer than version 1. The version used above is a date and time for packaging, which guarantees the versions will always be incremented.
• The install element will describe what actions should be taken to put the utility in proper working order.
  • The exec element defines the executable to call to install the utility and any (0...n) arguments (arg) to the executable. The installation should not require any user interaction (i.e., it should be silent). In this case, the installation statement evaluates to chmod u+x samtools samtools.pl, making the files executable on the system they are installed on.
• If a special uninstall element were needed, that can be included as well. Otherwise, uninstalling the FAR will just delete all of the files inside the FAR's directory. This package doesn't require an uninstall script because nothing is written outside of the FAR directory.
• The support-files element identifies each of the files that will be packaged in the FAR.

To package the FAR, use the following fabric_pkg command on a system with Appistry development tools:

Use the Management Console to deploy the FAR.

Example: Deploy Reference Data in a FAR

Deploying reference data for use in a pipeline (for example, a database), is a fairly simple task.

There are two noteworthy ways to go about distributing the data to the workers. If the reference data is small (<8GB) and not updated very frequently, the file could be packaged up in the FAR as one of the support-files and modified as required during installation. If the file is greater than 8GB, an install script can be written to get the files during the installation, rather than deploying the file as part of the package.

For example, consider the following FAR which runs a shell script to wget files from an FTP server:

rdb-ecoli-genome-squashed.xml

• The far element is the base element and is provided with a description, display-name, name and version.
  • Only the name and version are required.
  • The name is a unique name (or ID) given to an item in its definition XML. It is an arbitrarily long, case sensitive sequence of one or more alphanumeric characters (upper or lower case), hyphens and underscores. Pattern:[A-Za-z0-9-_]+
  • The fabric uses the version value to ensure the most current version of a file is maintained. All version numbers are right justified and zero filled. When comparing version 1.2 with 1.19, the fabric actually compares 1.02 with 1.19. Version 1.19 is considered newer than version 1.2, but 1.20 is considered to be newer than 1.19. Also, when comparing version 1.0.0 with 1.0 and 1, 1.0.0 is considered newer than version 1.0 and 1.0 is considered newer than version 1. The version used above is a date and time for packaging, which guarantees the versions will always be incremented.
• The install element will describe what actions should be taken to put the utility in proper working order.
  • The exec element defines the executable to call to install the utility and any (0...n) arguments (arg) to the executable. The installation should not require any user interaction (i.e., it should be silent). In this case, the installation statement evaluates to sh install.sh, which runs a shell script that gets the required files. The timeout for the script is 10 minutes. If getting the files takes longer than the allotted time, the operation will fail.
• If a special uninstall element were needed, that can be included as well. Otherwise, uninstalling the FAR will just delete all of the files inside the FAR's directory. This package doesn't require an uninstall script because nothing is written outside of the FAR directory.
• The support-files element identifies each of the files that will be packaged in the FAR.

This script gets database files from an FTP server:

install.sh

To package the FAR, use the following fabric_pkg command on a system with Appistry development tools:

Use the Management Console to deploy the FAR.