DMTCP : Distributed MultiThreaded Checkpointing

DMTCP (Distributed MultiThreaded Checkpointing) transparently checkpoints a single-host or distributed computation in user-space -- with no modifications to user code or to the O/S. It works on most Linux applications, including Python, Matlab, R, GUI desktops, MPI, etc. It is robust and widely used (on Sourceforge since 2007).

Among the applications supported by DMTCP are MPI (various implementations), OpenMP, MATLAB, Python, Perl, R, and many programming languages and shell scripting languages. With the use of TightVNC, it can also checkpoint and restart X-Window applications. The OpenGL library for 3D graphics is supported through a special plugin. It also has strong support for HPC (High Performance Computing) environments, including MPI, SLURM, InfiniBand, and other components. See QUICK-START.md for further details.

DMTCP supports the commonly used OFED API for InfiniBand, as well as its integration with various implementations of MPI, and resource managers (e.g., SLURM). See contrib/infiniband/README for more details.

[2019-08-14]: Upcoming releases: 1. A totally revised DMTCP module for support of MPI is planned (based on MANA; MPI-Agnostic, Network-Agnostic; see DMTCP Publications). 2. Longer-term, a DMTCP version 3.0 is being prepared with new and better features (e.g., user-programmable barriers within a plugin). If you would like to try it now, see Downloads.

[2019-08-14]: DMTCP 2.6.0 released!

Newer flags for configure:

Rename --enable-debug to --enable-logging
Add --enable-debug: "-Wall -g3 -O0" (for debugging DMTCP)

Newer flags for dmtcp_restart:

Add --debug-restart-pause flag to dmtcp_restart

Bug fixes and enhancements:

Fixes for glibc versions greater than or equal to 2.24
Fix deadlock in system() wrapper when the child crashes
Fix deadlock when a process is forked in the resume phase (issue #691)
jsocket: Warn user if peer closes socket while draining (issue #701)
Fix epoll1 test (initialize addrlen for accept()) (#705)
Fix to correctly calculate Coordinator/Host IP: Affects some distributed applications
Allow restored stack to grow if needed.
Fix bug in POSIX timer: race condition manifested in test/timer.c/Ubuntu-18.04
Modified InfiniBand plugin for more robust support (primarily of interest for MPI)
The floating point environment (fegetenv()) is now restored on restart. (Formerly, only the rounding mode (fegetround()) was restored.)
The current resource limits (rlim_cur) for RLIMIT_NOFILE and RLIMIT_STACK are restored if possible.
Mutex ownership and robust mutexes are now supported if DMTCP is configured with --enable-mutex-wrappers. (However, this configuration can also add runtime overhead if mutex operations are called very frequently.) [Thanks to Johannes Stoelp, Laurent Buchard, Pankaj Mehta of Synopsys, Inc.]
Fix bug if stack grows a lot after a restart.
Improved support for pty's
util/gdbinit-example added for those who wish to debug DMTCP internals.
Many bug fixes

[2017-11-15]: DMTCP 2.5.2 released!

The small fixes and ehancements of this "point release" include:

All fixes in Release DMTCP-2.4.9 are incorporated in this release.
An incompatibility of DMTCP with Open MPI 1.10 when using orterun (mpirun) was discovered. This does not affect recent versions, such as Open MPI 2.x.
In some rare cases, open files were not properly restored due to a use-after-free bug. This is now fixed.
In some rare cases, one process had created a SysV shared memory object, and a different process was assigned to restore it on restart. This was not handled correctly, and is now fixed.
Correctly restore CPU affinities of threads
Virtualized SysV shared memory keys to avoid race condition on restart
Fixed logic for checking if relative path to file was a duplicate of another existing path
The NSCD area for name service caching daemon was not handled correctly in CentOS 6.8 and later correctly. Fixed now.
The Linux sched.h include file for scheduling of cores was added to satisfy some older Linux distros that needed it for compiling DMTCP.
Fixed a regression in which --enable-debug (for verbose debug logs) was not being properly written.
The DMTCP coordinator was displaying a spurious warning, "Failed to find coordinator IP address", because it did not check for a canoncial hostname. A related issue prevented DMTCP from working properly on some SUSE/openSUSE distros.

[2017-11-14]: DMTCP 2.4.9 released!

The small fixes and ehancements of this "point release" include:

Fixed a regression causing deleted NFS files to be handled incorrectly
Fixed handling of glibc for versions greater than glibc-2.24
Errors and warnings with gcc-7.x are fixed
A rare bug affecting pthread_cancel, etc., created incorrect pid on restart
man pages fixed: Description section was always describing dmtcp_command

[2017-09-05]: DMTCP 2.5.1 released!

This release mostly provides added robustness. Two notable items of added functionality are:

DMTCP_RESTART_PAUSE and DMTCP_RESTART_PAUSE0 environment variables for easier debugging upon initial restart
The --debug-logs flag was added to dmtcp_launch/dmtcp_restart. One can now turn on logging individually for separate plugins, instead of only turning it on globally.

An incompatibility of DMTCP with Open MPI 1.10 when using orterun (mpirun) was discovered. This may also affect some other versions of Open MPI 1.10. This bug will be fixed in a future release.

Other fixes and enhancements include:

Fixed an issue when starting multiple DMTCP coordinators on same host at approximately the same time
Fixed issue with PBS scheduler for HPC
Fixed issue when restarting on a different host with a larger limit on the number of open file descriptors
dmtcp_launch/dmtcp_restart now accept '--debug-logs' flag to specify which DMTCP plugins should produce logging information (It used to be all or nothing.)
Improved robustness for IB (InfiniBand) plugin
Fixed DMTCP_RESTART_PAUSE and DMTCP_RESTART_PAUSE0 environment variables for debugging upon restart
The brk() call was failing on restart on Debian due to overly strict assert
dmtcp_launch was hanging on some RHEL5 and RHEL6 due to deadlock with libc low-level locks. Fixed now.
Updated tls_pid_offset in DMTCP to handle newer GLIBc (versions > 2.24)
Fixed launch of 32-bit binary when forking/execing from a 64-bit executable
Fixed issue that can affect a parent holding a malloc-lock while forking
Fixed issue when a user thread calls 'dmtcp_get_coord_ckpt_dir()'

[2017-01-03]: DMTCP 2.5.0 released!

This release includes a few new plugins and several bug fixes for robustness. Some of the highlights include:

Support for InfiniBand UD (in addition to the more common InfiniBand RC).
Added support for CMA (Cross-Memory Attach): process_vm_readv and process_vm_writev
Improved multi-arch (mixed 32-/64- bit) support.
Re-added --enable-fast-restart.
Added a new commandline option --with-plugin-32 for dmtcp_launch to specify 32-bit plugins in a 64-bit environment.
Added --enable-pthread-mutex-wrappers configure flag to enable pthread_mutex_{lock,unlock} wrappers needed for Open MPI.
Added ability to specify environment file used in the modify-env plugin.
Allow dmtcp_restart to be invoked by root.
The following new plugins were added:
- pathvirt: to virtualize filesystem paths.
- delayresume: for finer-grained control over resuming of user threads during resume/restart.

More details are here.

[2017-02-13]: DMTCP 2.4.8 released!

If you are using a recent Linux kernel or a recent GLIBC library, and you need to continue to use the 2.4.x branch, then you are recommended to upgrade to this 2.4.8 release.

The newest kernels (approximately Linux version 4.0 and later map VDSO into memory slightly differently for 32-bit processes running in a 64-bit Linux. DMTCP was failing to restart for such 32-bit processes. This is now fixed.
dmtcp_dlsym() extended to provide more robust wrapper functions.
Corner case fixed: signal handlers now restored before plugins restart.
Minor bug fixes.

More details are here.

[2017-01-03]: DMTCP 2.4.7 released!

If you are using a recent Linux kernel or a recent GLIBC library, and you need to continue to use the 2.4.x branch, then you are recommended to upgrade to this 2.4.7 release.

The newest kernels (approximately Linux version 4.0 and later) map vDSO into memory slightly differently for 32-bit processes running in a 64-bit Linux. DMTCP was failing to restart for such 32-bit processes. This is now fixed.
In recent versions of GLIBC, libdl/dlerror calls libc/malloc, creating an infinite recursion when we interpose using libdl/dlsym. We now implement our own dlsym and use it in most cases, thus reducing our reliance on libdl.

More details are here.

[2016-12-31]: DMTCP 2.4.6 released!: Fixed JLOG to work with '--enable-debug' configure flag. A corner case in handling of InfiniBand was fixed, where two queue pairs were not connected. More details are here.

[2016-07-19]: DMTCP 2.4.5 released!: This now provides more robust handling of mutexes and InfiniBand. An '--enable-pthread-mutex-wrappers' configure flag was added to enable pthread_mutex_{lock,unlock} wrappers needed for Open MPI. A failure to build on SLES10 was fixed. Restart was fixed in a case where independent processes had orphaned roots of trees. The signals SIGCANCEL (signal #32) and SIGSETXID (signal #33) used by NPTL for threads were not being properly restored on restart. More details are here.

[2016-05-22]: DMTCP 2.5.0-rc2 released!: This "rc" version (release candidate version) is an a second, early version of a "feature release". This should be reasonably stable, and is the current recommended version. Back-ports of bug fixes continue to be supported for the previous DMTCP 2.4.x series. This version, like rc1, emphasizes more robust handling of InfiniBand, and also adds some minor enhancements. See the News page for further information.

[2016-10-29]: A DMTCP module for Slurm is coming.: There is now a private fork of Slurm that provides a DMTCP module for Slurm with tight integration. This will be submitted for inclusion in the main branch of Slurm. If someone is interested in testing an advance version, please contact us.

[2016-02-11]: DMTCP 2.5.0-rc1 released!: This "rc" version (release candidate version) is an early version of a "feature release". For those who don't need the newest features, they are still recommended to download the latest DMTP 2.4.x version. This version emphasizes more robust handling of InfiniBand with dmtcp_launch --infiniband. See the News page for further information.

[2016-01-14]: DMTCP 2.4.4 released!: DMTCP now supports InfiniBand UD. (Newer implementations of MPI are now often configured to use this transport mode.) Additionally, some issues were fixed concerning: a regression in our support for multi-arch installations (for a mixture of 32- and 64-bit target applications); a race appearing after many ckpt-restart cycles (restart of ckpt of restart of ...); some utilities for plugins from dmtcp.h marked "FOR EXPERTS ONLY" are now supported "out of the box"; the modify-env plugin was extended to support "dmtcp_env.txt" files of size up to 12 KB (with warnings for larger files); an improved error message for "dmtcp_launch --no-coordinator"; and some minor fixes. More details are here.

[2015-11-25]: DMTCP 2.4.3 released!: This is a bug fix release. Some issues were fixed concerning: added support for CMA (Cross-Memory Attach: process_vm_readv/process_vm_writev); fixed a regression affecting dmtcp_checkpoint() [applic-initiated ckpt API]; fixed a compilation error on RHEL-5.8; and some smaller bug fixes.

[2015-10-15]: DMTCP 2.4.2 released!: This is a bug fix release. Some issues were fixed concerning: ./configure --enable-debug; compiling under clang; interval checkpointing (dmtcp_launch --interval); having dmtcp_launch --quiet pass on the --quiet flag when implicitly starting a new coordinator; when an application mmap's /dev/zero; improved support for sched_setaffinity() and friends; a report on tclsh-8.6 and its use of pthread_atfork(); on_semget() and semctl(); and a compilation error for SLES10.

[2015-09-02]: DMTCP 2.4.1 released!: This is primarily a bug fix release. It fixes an issue with version 2.4.0 and deleted shared memory files. In particular, this can affect the use of DMTCP with MPI in some circumstances.

[2015-07-24]: DMTCP 2.4.0 released!: This release provides better support for MPI/SLURM integration for Intel MPI, MVAPICH-2, MPICH-2, and Open MPI; glibc 2.21; and shared-memory segments. Check the release notes for more details.

[2014-07-14]: DMTCP 2.3.1 released!: This is primarily a bug fix release.

[2014-07-03]: DMTCP 2.3 released!: This is primarily a bug fix release. However, if you are using DMTCP for the ARM v7 CPU, or if you are using DMTCP either with the InfiniBand network or with the SLURM batch system, then it is strongly recommended to upgrade. Check the release notes for more details.

[2014-03-20]: DMTCP 2.2.1 released!: This is a bug fix release. Users relying on --enable-unique-checkpoint-filenames configure flag are highly recommended to upgrade to this release. Check the release notes for more details.

[2014-03-14]: DMTCP 2.2 released!: In this release, the lowest layers have been re-organized and partially re-written for greater clarity of code and greater maintainability. Also, users relying on the use of DMTCP with MPI, InfiniBand or the Torque or SLURM batch queues are strongly advised to upgrade. Check the release notes for more details.

[2014-01-12]: DMTCP 2.1 released!: This release includes enhancement to the core feature set and some newly stable plugins. Check the release notes for more details.

[2013-10-03]: DMTCP 2.0 released!: This version 2.0 release represents the future of DMTCP. DMTCP version 2.0 has been re-designed around the concept of plugins. The older DMTCP version 1.2.x branch will continue to be maintained for bug fixes. Check the release notes for more details.

Older News

[2013-08-03]: DMTCP 1.2.8 released!: This is primarily a bug fix release. It is particularly recommended to upgrade if you are using DMTCP with the ARM CPU, or if you will compile DMTCP with a C++11 compiler (e.g. GNU flag -std=c++11). Check the release notes for more details.

[2013-03-13]: DMTCP 1.2.7 released!: This is primarily a bug fix release. Check the release notes for more details.

[2012-07-31]: DMTCP 1.2.6 released!: Previous release had issues with compilation on older kernels, this release fixes that. It also contains some changes needed for gcc 4.7. Check the release notes for more details.

[2012-05-27]: DMTCP 1.2.5 released!: Along with numerous bug fixes, this release features support for ARM processors along with support for epoll/eventfd/signalfd system calls. Check the release notes for more details.

[2012-01-23]: DMTCP 1.2.4 released!: Along with lot of bug fixes, this release focuses on robust treatment of processes that rapidly create and destroy threads. Users of DMTCP 1.2.3 are highly encouraged to upgrade to this release. Check the release notes for more details.

[2011-07-22]: DMTCP 1.2.3 released!: This is primarily a bug-fix release with lots of bug fixes that improve overall stability. Users of DMTCP 1.2.2 are highly encouraged to upgrade to this release. Check the release notes for more details.

[2011-06-22]: DMTCP 1.2.2 released!: Along with a lot of bug fixes, this release provides support for a module system allowing users to write their own extension to DMTCP, and removed dependency on libc.a for building. Check the release notes for more details.

[2011-03-12]: DMTCP 1.2.1 released!: Along with a lot of bug fixes, this release provides support for MPICH2 1.3.x (transparently checkpointing MPICH2 under DMTCP), and calling dmtcpaware API (dmtcpCheckpoint(), etc.) directly from inside a python session. Check the release notes for more details.

[2010-11-04]: DMTCP 1.2.0 released!: This is a semi-major release of DMTCP. The biggest change is the support for GNU screen sessions. It also fixes some instabilities in checkpointing Matlab under certain environments. Also includes numerous bug fixes were implemented as a part of review of DMTCP sub-systems.

DMTCP is currently maintained by Kapil Arya, Gene Cooperman, Rohan Garg, Jiajun Cao, and Artem Polyakov. The list of active developers continues to evolve.

The DMTCP project is partially supported by grants from Intel Corporation, and from the National Science Foundation under grants OCI-0960978, ACI-1440788, and OAC-1740218. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of Intel Corporation or of the National Science Foundation.

DMTCP: Distributed MultiThreaded CheckPointing

About DMTCP:

Announcement!

News

Authors

Acknowledgment