Scroll to navigation

CONDOR_SUBMIT_DAG(1) HTCondor Manual CONDOR_SUBMIT_DAG(1)

NAME

condor_submit_dag - HTCondor Manual

Place a node scheduler job with the local AP to manage job workflow described as a Direct Acyclic Graph (DAG).

SYNOPSIS

condor_submit_dag [-help | -version]

condor_submit_dag [OPTIONS] DAG_file [DAG_file...]

condor_submit_dag [-no_submit] [-v/-verbose] [-f/-force] [-dagman path] [-notification value] [-debug level] [-MaxIdle N] [-MaxJobs N] [-MaxPre N] [-MaxPost N] [-MaxHold N] [-suppress_notification | -dont_suppress_notification] [-SubmitMethod value] [-UseDagDir] [-outfile_dir path] [-config filename] [-Lockfile filename] [-insert_sub_file filename] [-a/-append command] [-batch-name name] [-AutoRescue <0|1>] [-DoRescueFrom N] [-load_save filename] [-DoRecovery] [-AllowVersionMismatch] [-no_recurse | -do_recurse] [-update_submit] [-import_env] [-include_env variable[,variable...]] [-insert_env key=value[;key=value...]] [-DumpRescue] [-valgrind] [-DontAlwaysRunPost | -AlwaysRunPost] [-priority N] [-r/-remote schedd_name] [-schedd-daemon-ad-file filename] [-schedd-address-file filename] [-RescueFile filename]

DESCRIPTION

Automatically produce and place a scheduler universe <href="../man-pages/condor_submit.html#universe"> job to execute HTCondor's DAGMan workflow manager on a list of specified Directed Acyclic Graphs (DAGs) of HTCondor jobs. Extensive documentation is in the HTCondor DAGMan Workflows Section.

OPTIONS

Display usage information.
Display version information.
Produce the HTCondor submit description file for DAGMan, but do not submit DAGMan as an HTCondor job.
Cause condor_submit_dag to give verbose error messages.
Don't fail to place DAGMan job to AP if previous execution files are discovered. Previous execution files (except the *.dagman.out) are cleaned up and old rescue files are renamed.
Specify a path to a condor_dagman executable to be executed rather than the installed one discovered in user's path.
Set the maximum number of idle jobs allowed before placing more jobs to the AP. If not specified then this is set to the value of DAGMAN_MAX_JOBS_IDLE <href="../admin-manual/configuration/dagman.html#DAGMAN_MAX_JOBS_IDLE">.

Note:

condor_dagman can place jobs beyond this threshold in a single job placement cycle, but won't place more jobs until the detected number of idle jobs drops below the specified threshold again.


Set the maximum number of nodes that have placed a list of jobs to the AP at any given moment. If not specified then this is set to the value of DAGMAN_MAX_JOBS_SUBMITTED <href="../admin-manual/configuration/dagman.html#DAGMAN_MAX_JOBS_SUBMITTED">.
Set the maximum number of PRE scripts being executed at any given moment. If not specified then this value is set to DAGMAN_MAX_PRE_SCRIPTS <href="../admin-manual/configuration/dagman.html#DAGMAN_MAX_PRE_SCRIPTS">.
Set the maximum number of POST scripts being executed at any given moment. If not specified then this value is set to DAGMAN_MAX_POST_SCRIPTS <href="../admin-manual/configuration/dagman.html#DAGMAN_MAX_POST_SCRIPTS">.
Set the maximum number of HOLD scripts being executed at any given moment. If not specified then this value is set to DAGMAN_MAX_HOLD_SCRIPTS <href="../admin-manual/configuration/dagman.html#DAGMAN_MAX_HOLD_SCRIPTS">.
Set the e-mail notification <href="../man-pages/condor_submit.html#notification"> for DAGMan itself.
Specify a condor_schedd on a remote machine to place the DAGMan job.

Note:

Since DAGMan expects all necessary files to be present in it's working directory, all files used by DAGMan need to be present on the remote machine via some method like a shared filesystem.


Set the DAGMan debug log level (see below for levels).
Inform DAGMan to execute each specified DAG from their respective directories.
Specify the path to a directory for DAGMan's *.dagman.out debug log to be written.
Specify an HTCondor configuration file to a specific DAGMan instance's execution.

Note:

Only one configuration file can be specified which will cause a failure if used in conjunction with the CONFIG <href="../automated-workflows/dagman-reference.html#CONFIG"> command.


Path to a file to write DAGMan's process information. This prevents other DAGMan processes executing the same DAG(s) from being executed in the same directory.
Specify a file containing JDL submit commands to be inserted in the produced DAGMan job submit description (*.condor.sub).

Note:

Only one extra submit description file can be specified. So, the specified file will override any file specified via DAGMAN_INSERT_SUB_FILE <href="../admin-manual/configuration/dagman.html#DAGMAN_INSERT_SUB_FILE">


Specify JDL commands to insert into the produced DAGMan job submit description (*.condor.sub). Can be used multiple times to add multiple JDL commands. Commands with spaces in them must be enclosed in double quotes.

Note:

JDL commands specified via -append take precedence over any commands added via -insert_sub_file or DAGMAN_INSERT_SUB_FILE <href="../admin-manual/configuration/dagman.html#DAGMAN_INSERT_SUB_FILE">.


Set the JobBatchName <href="../classad-attributes/job-classad-attributes.html#JobBatchName"> to name for DAGMan and all jobs managed by DAGMan.
Automatically detect rescue DAG files upon startup and rescue from the most recent (highest number) rescue file discovered. 0 is False. Default 1 is True.
Specify a specific rescue number to locate and restore state from.
Specify a specific rescue file by name to restore state from.
Specify a specific SAVE_POINT_FILE <href="../automated-workflows/dagman-reference.html#SAVE_POINT_FILE"> to restore state from. If provided a path DAGMan will attempt to read that file following that path. Otherwise, DAGMan will check for the file in the DAG's save_files sub-directory.
Inform DAGMan to startup in recovery mode (restore state from *.nodes.log).
Allow a version difference between condor_dagman itself and the *.condor.sub file produced by condor_submit_dag <href="../man-pages/condor_submit_dag.html">.

Warning:

This option should only be used if absolutely necessary because version mismatches can cause subtle problems when running DAGMan.


Don't recursively pre-produce SUBDAG <href="../automated-workflows/dagman-reference.html#SUBDAG">'s submit description files prior to placing root DAG to the AP.
Recursively pre-produce all SUBDAG <href="../automated-workflows/dagman-reference.html#SUBDAG">'s submit description files prior to placing the root DAG to the AP.
Don't treat an existing DAGMan submit description file (*.condor.sub) as an error; rather update the file while preserving the -maxjobs, -maxidle, -maxpre, and -maxpost options (if specified).
Inform condor_submit_dag to import the current shell environment into the produced DAGMan submit description file's environment <href="../man-pages/condor_submit.html#environment"> command.
Specify a comma separated list of environment variables to add to the produced DAGMan submit description file's getenv <href="../man-pages/condor_submit.html#getenv"> command.
Specify a delimited string of key=value pairs to explicitly set into the produced DAGMan submit description file's environment <href="../man-pages/condor_submit.html#environment"> command. If the same key is specified multiple times then the last occurrences value takes precedence.

Note:

The base delimiter is a semicolon that can be overridden by setting the first character in the string to a valid delimiting character. 

$ condor_submit_dag -insert_env |foo=0|bar=1|baz=2




Inform DAGMan to produce a full rescue DAG file and exit before executing the DAG.
Run DAGMan under valgrind.

Note:

This is option is intended for testing and development of DAGMan. The DAGMan execution speed is drastically reduced.


Warning:

Failure will occur if necessary valgrind is not installed. valgrind is only available of Linux OS.


Always execute POST scripts even upon failure of any PRE scripts.
Only execute POST scripts after a node's list of jobs have completed (Success or Failure). Default.
Set the minimum JobPrio <href="../classad-attributes/job-classad-attributes.html#JobPrio"> for jobs managed by DAGMan.
Specifies a full path to a daemon ad file for a specific condor_schedd to place the DAGMan job.
Specifies a full path to an address file for a specific condor_schedd to place the DAGMan job.
Suppress email notifications for all jobs managed by DAGMan.
Allow email notifications for any jobs managed by DAGMan that have notifications specified. Default.
Specify how DAGMan will place managed jobs to the AP (see values below).



GENERAL REMARKS

Note:

All command line flags are case insensitive.


Some of the command line options also have corresponding configuration values. The values specified via the command line will take precedence over any configured values.

Some of the command line options are passed down to DAGMan to use when executing SUBDAG <href="../automated-workflows/dagman-reference.html#SUBDAG">s.

Debug Level Values

level Details
0 Never produce output except for usage info.
1 Very quiet. Only output severe errors.
2 Normal output and error messages
3 (Default) Print warnings.
4 Internal debugging output.
5 Outer loop debugging.
6 Inner loop debugging.
7 Output parsed DAG input lines.

Submit Method Values

Value Method
0 Run condor_submit <href="../man-pages/condor_submit.html">
1 Direct place job(s) to local condor_schedd

EXIT STATUS

0 - Success

1 - Failure

EXAMPLES

Execute a single DAG:

$ condor_submit_dag sample.dag


Execute a single DAG that has successfully completed once already:

$ condor_submit_dag -force sample.dag


Execute a DAG with a max threshold of 10 idle jobs:

$ condor_submit_dag -maxidle 10 sample.dag


Execute a DAG with a max limit of 10 PRE scripts and 5 POST scripts executing concurrently:

$ condor_submit_dag -maxpre 10 -maxpost 5 sample.dag


Execute multiple DAGs:

$ condor_submit_dag first.dag second.dag


Execute multiple DAGs in their respective directories:

$ condor_submit_dag -usedagdir subdir1/first.dag subdir2/second.dag


Execute DAG and notify user once DAG is complete:

$ condor_submit_dag -notification complete sample.dag


Execute DAG with a custom batch name:

$ condor_submit_dag -batch-name my-awesome-dag sample.dag


Execute DAG and restore state from specific rescue file:

$ condor_submit_dag -RescueFile simple.dag.rescue008 simple.dag


Execute DAG and restore state from specific rescue number 8:

$ condor_submit_dag -dorescuefrom 8 sample.dag


Execute DAG and restore state from save file post-analysis.save:

$ condor_submit_dag -load_save post-analysis.save sample.dag


Execute DAG and suppress all job e-mail notifications:

$ condor_submit_dag -suppress_notification sample.dag


SEE ALSO

None

AVAILABILITY

Linux, MacOS, Windows

Author

HTCondor Team

Copyright

1990-2026, Center for High Throughput Computing, Computer Sciences Department, University of Wisconsin-Madison, Madison, WI, US. Licensed under the Apache License, Version 2.0.

March 16, 2026