Debian Science Project
Summary
Workflow
workflow management systems useful for scientific research

This task lists some packages providing workflow management systems useful for scientific research.

Description

For a better overview of the project's availability as a Debian package, each head row has a color code according to this scheme:

If you discover a project which looks like a good candidate for Debian Science to you, or if you have prepared an unofficial Debian package, please do not hesitate to send a description of that project to the Debian Science mailing list

Links to other tasks

Debian Science Workflow packages

Official Debian packages with high relevance

Coop-computing-tools
cooperative computing tools
Versions of package coop-computing-tools
ReleaseVersionArchitectures
buster4.0-1.1amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,ppc64el,s390x
wheezy3.5.1-2amd64,armel,armhf,i386,ia64,kfreebsd-amd64,kfreebsd-i386,mips,mipsel,powerpc,s390,s390x,sparc
sid3.5.1-2kfreebsd-amd64,kfreebsd-i386
sid4.0-1.1amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,powerpc,ppc64el,s390x
stretch4.0-1.1amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,ppc64el,s390x
upstream6.1
Debtags of package coop-computing-tools:
uitoolkitncurses
Popcon: 3 users (4 upd.)*
Newer upstream!
License: DFSG free
Git

This is a collection of software that help users to share resources in a complex, heterogeneous, and unreliable computing environment. This includes:

  • Chirp: A personal filesystem and I/O protocol that allows unprivileged users to share space securely, efficiently, and conveniently. When combined with Parrot, Chirp allows users to create custom wide-area distributed filesystems.
  • Parrot: A transparent user-level virtual filesystem that allows any ordinary program to be attached to a remote storage device such as an FTP server or a Chirp server.
  • Makeflow: A workflow system for parallel and distributed computing that uses a language very similar to Make.
  • Work Queue: A system and API for building master-worker style programs that scale up to thousands of processors.
  • All Pairs: A computational abstraction for running very large Cartesian products.
  • Wavefront: A computational abstraction for running very large dynamic programming problems.
  • The Fault Tolerant Shell: A high-level programming language that allows users to combine the ease of shell scripting, the power of distributed programming, and the precision of compiled languages. Basically, parallel programming and exception handling for scripts.
Cwltool
Common workflow language reference implementation
Versions of package cwltool
ReleaseVersionArchitectures
stretch1.0.20170114120503-1all
sid1.0.20170810192106-2all
upstream1.0.20170828135420
Popcon: 7 users (13 upd.)*
Newer upstream!
License: DFSG free
Git

This is the reference implementation of the Common Workflow Language. It is intended to be feature complete and provide comprehensive validation of CWL files as well as provide other tools related to working with CWL.

This is written and tested for Python 2.7.

The reference implementation consists of two packages. The "cwltool" package is the primary Python module containing the reference implementation in the "cwltool" module and console executable by the same name.

The "cwl-runner" package is optional and provides an additional entry point under the alias "cwl-runner", which is the implementation-agnostic name for the default CWL interpreter installed on a host.

Please cite: Peter Amstutz, Michael R. Crusoe, Nebojša Tijanić, Brad Chapman, John Chilton, Michael Heuer, Andrey Kartashov, Dan Leehr, Hervé Ménager, Maya Nedeljkovich, Matt Scales, Stian Soiland-Reyes and Luka Stojanovic: Common Workflow Language, v1.0. (2016)
Gearman
Distributed job queue
Versions of package gearman
ReleaseVersionArchitectures
jessie1.0.6-5all
squeeze0.13-1all
wheezy0.33-2all
stretch1.0.6-9all
buster1.1.17-2all
sid1.1.17-2all
Debtags of package gearman:
rolemetapackage
Popcon: 0 users (0 upd.)*
Versions and Archs
License: DFSG free
Git

Gearman is a system to farm out work to other machines, dispatching function calls to machines that are better suited to do work, to do work in parallel, to load balance lots of function calls, or to call functions between languages.

This package is an empty package that depends on both the client and the server.

Gearman-tools
Tools for the Gearman distributed job queue
Versions of package gearman-tools
ReleaseVersionArchitectures
wheezy0.33-2amd64,armel,armhf,i386,ia64,kfreebsd-amd64,kfreebsd-i386,mips,mipsel,powerpc,s390,s390x,sparc
buster1.1.17-2amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,ppc64el,s390x
squeeze0.13-1amd64,armel,i386,ia64,kfreebsd-amd64,kfreebsd-i386,mips,mipsel,powerpc,s390,sparc
sid1.1.17-2amd64,arm64,armel,armhf,hurd-i386,i386,kfreebsd-amd64,kfreebsd-i386,mips,mips64el,mipsel,powerpc,ppc64el,s390x
stretch1.0.6-9amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,ppc64el,s390x
jessie1.0.6-5amd64,arm64,armel,armhf,i386,mips,mipsel,powerpc,ppc64el,s390x
Popcon: 14 users (30 upd.)*
Versions and Archs
License: DFSG free
Git

Gearman is a system to farm out work to other machines, dispatching function calls to machines that are better suited to do work, to do work in parallel, to load balance lots of function calls, or to call functions between languages.

This package contains some command line tools for manipulating gearman jobs.

Make
utility for directing compilation
Versions of package make
ReleaseVersionArchitectures
squeeze3.81-8amd64,armel,i386,ia64,kfreebsd-amd64,kfreebsd-i386,mips,mipsel,powerpc,s390,sparc
sid4.1-9hurd-i386,kfreebsd-amd64,kfreebsd-i386,powerpc
jessie4.0-8.1amd64,arm64,armel,armhf,i386,mips,mipsel,powerpc,ppc64el,s390x
sid4.1-9.1amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,ppc64el,s390x
stretch4.1-9.1amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,ppc64el,s390x
buster4.1-9.1amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,ppc64el,s390x
wheezy3.81-8.2amd64,armel,armhf,i386,ia64,kfreebsd-amd64,kfreebsd-i386,mips,mipsel,powerpc,s390,s390x,sparc
upstream4.2.1
Debtags of package make:
develbuildtools
interfacecommandline
roleprogram
scopeutility
suitegnu
works-withsoftware:source
Popcon: 30614 users (7827 upd.)*
Newer upstream!
License: DFSG free
Git

GNU Make is a utility which controls the generation of executables and other target files of a program from the program's source files. It determines automatically which pieces of a large program need to be (re)created, and issues the commands to (re)create them. Make can be used to organize any task in which targets (files) are to be automatically updated based on input files whenever the corresponding input is newer --- it is not limited to building computer programs. Indeed, Make is a general purpose dependency solver.

The package is enhanced by the following packages: make-doc
Pegasus-wms
Scientific workflow management system for HTCondor
Versions of package pegasus-wms
ReleaseVersionArchitectures
sid4.4.0+dfsg-6amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,powerpc,ppc64el,s390x
wheezy4.0.1+dfsg-8amd64,armel,armhf,i386,ia64,mips,mipsel,powerpc,s390,s390x,sparc
jessie4.4.0+dfsg-4amd64,arm64,armel,armhf,i386,mips,mipsel,powerpc,ppc64el,s390x
stretch4.4.0+dfsg-6amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,ppc64el,s390x
buster4.4.0+dfsg-6amd64,arm64,armel,armhf,i386,mips,mips64el,mipsel,ppc64el,s390x
Popcon: 5 users (3 upd.)*
Versions and Archs
License: DFSG free

The Pegasus project encompasses a set of technologies the help workflow-based applications execute in a number of different environments including desktops, campus clusters, grids, and now clouds. Scientific workflows allow users to easily express multi-step computations, for example retrieve data from a database, reformat the data, and run an analysis. Once an application is formalized as a workflow the Pegasus Workflow Management Service can map it onto available compute resources and execute the steps in appropriate order.

Screenshots of package pegasus-wms
Python-nipype
Neuroimaging data analysis pipelines in Python
Versions of package python-nipype
ReleaseVersionArchitectures
sid0.12.1+git4-gbc3a0b5-1all
squeeze0.3.2-1all
wheezy0.5.3-2wheezy2all
experimental0.13.1-1all
upstream0.13.1
Popcon: 34 users (2 upd.)*
Newer upstream!
License: DFSG free
Git

Nipype interfaces Python to other neuroimaging packages and creates an API for specifying a full analysis pipeline in Python. Currently, it has interfaces for SPM, FSL, AFNI, Freesurfer, but could be extended for other packages (such as lipsia).

Please cite: SS Ghosh, C Burns, D Clark, K Gorgolewski, YO Halchenko, C Madison, R Tungaraza and KJ Millman: Nipype: Opensource platform for unified and replicable interaction with existing neuroimaging tools (eprint) 16th Annual Meeting of the Organization for Human Brain Mapping :106 (2010)
Snakemake
pythonic workflow management system
Versions of package snakemake
ReleaseVersionArchitectures
sid3.10.0-1all
buster3.10.0-1all
stretch3.10.0-1all
upstream4.0.0
Popcon: 4 users (3 upd.)*
Newer upstream!
License: DFSG free
Git

Build systems like GNU Make are frequently used to create complicated workflows, e.g. in bioinformatics. This project aims to reduce the complexity of creating workflows by providing a clean and modern domain specific language (DSL) in Python style, together with a fast and comfortable execution environment.

Please cite: Johannes Köster and Sven Rahmann: Snakemake-a scalable bioinformatics workflow engine. Bioinformatics (2012)

Packaging has started and developers might try the packaging code in VCS

Python-toil
cross-platform workflow engine (Python 2)
Versions of package python-toil
ReleaseVersionArchitectures
VCS3.5.0~alpha1.321-1all
Versions and Archs
License: Apache-2.0
Debian package not available
Git
Version: 3.5.0~alpha1.321-1

Toil is a scalable, efficient, cross-platform and easy-to-use workflow engine in pure Python. It works with several well established load balancers like Slurm or the Sun Grid Engine. Toil is compatible with the Common Workflow Language (CWL).

This package installs the library for Python 2.

*Popularitycontest results: number of people who use this package regularly (number of people who upgraded this package recently) out of 200581