Debian Science Data management packages

DataLad is a data management and distribution platform providing access to a wide range of data resources already available online. Using git-annex as its backend for data logistics it provides following facilities built-in or available through additional extensions

• command line and Python interfaces for manipulation of collections of datasets (install, uninstall, update, publish, save, etc.) and separate files/directories (add, get)

• extract, aggregate, and search through various sources of metadata (xmp, EXIF, etc; install datalad-neuroimaging for DICOM, BIDS, NIfTI support)

• crawl web sites to automatically prepare and update git-annex repositories with content from online websites, S3, etc (install datalad-crawler)

This extension enhances DataLad (http://datalad.org) for working with computational containers.

git-annex allows managing large files with git, without storing the file contents in git. It can sync, backup, and archive your data, offline and online. Checksums and encryption keep your data safe and secure. Bring the power and distributed nature of git to bear on your large files with git-annex.

It can store large files in many places, from local hard drives, to a large number of cloud storage services, including S3, WebDAV, and rsync, with dozens of cloud storage providers usable via plugins. Files can be stored encrypted with gpg, so that the cloud storage provider cannot see your data. git-annex keeps track of where each file is stored, so it knows how many copies are available, and has many facilities to ensure your data is preserved.

git-annex can also be used to keep a folder in sync between computers, noticing when files are changed, and automatically committing them to git and transferring them to other computers. The git-annex webapp makes it easy to set up and use git-annex this way.