Debian Science Data management packages

DataLad is a data management and distribution platform providing access to a wide range of data resources already available online. Using git-annex as its backend for data logistics it provides following facilities built-in or available through additional extensions

• command line and Python interfaces for manipulation of collections of datasets (install, uninstall, update, publish, save, etc.) and separate files/directories (add, get)

• extract, aggregate, and search through various sources of metadata (xmp, EXIF, etc; install datalad-neuroimaging for DICOM, BIDS, NIfTI support)

• crawl web sites to automatically prepare and update git-annex repositories with content from online websites, S3, etc (install datalad-crawler)

This extension enhances DataLad (http://datalad.org) for working with computational containers.

git-annex allows managing files with git, without checking the file contents into git. While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, time, or disk space.

It can store large files in many places, from local hard drives, to a large number of cloud storage services, including S3, WebDAV, and rsync, with a dozen cloud storage providers usable via plugins. Files can be stored encrypted with gpg, so that the cloud storage provider cannot see your data. git-annex keeps track of where each file is stored, so it knows how many copies are available, and has many facilities to ensure your data is preserved.

git-annex can also be used to keep a folder in sync between computers, noticing when files are changed, and automatically committing them to git and transferring them to other computers. The git-annex webapp makes it easy to set up and use git-annex this way.