A job is submitted for execution to Condor using the condor_submit command. condor_submit takes as an argument the name of a file called a submit description file. This file contains commands and keywords to direct the queuing of jobs. In the submit description file, Condor finds everything it needs to know about the job. Items such as the name of the executable to run, the initial working directory, and command-line arguments to the program all go into the submit description file. condor_submit creates a job ClassAd based upon the information, and Condor works toward running the job.
The contents of a submit file can save time for Condor users. It is easy to submit multiple runs of a program to Condor. To run the same program 500 times on 500 different input data sets, arrange your data files accordingly so that each run reads its own input, and each run writes its own output. Each individual run may have its own initial working directory, stdin, stdout, stderr, command-line arguments, and shell environment. A program that directly opens its own files will read the file names to use either from stdin or from the command line. A program that opens a static filename every time will need to use a separate subdirectory for the output of each run.
The condor_submit manual page
is on page ![[*]](crossref.png)
and
contains a complete and full description of how to use condor_submit.
Each submit description file describes one cluster of jobs to be placed in the Condor execution pool. All jobs in a cluster must share the same executable, but they may have different input and output files, and different program arguments. The submit description file is the only command-line argument to condor_submit. If the submit description file argument is omitted, condor_submit will read the submit description from standard input.
The submit description file must contain one executable command and at least one queue command. All of the other commands have default actions.
The commands which can appear in the submit description file are numerous. They are listed here in alphabetical order by category.
BASIC COMMANDS
There are two permissible formats for specifying arguments. The new syntax supports uniform quoting of spaces within arguments; the old syntax supports spaces in arguments only in special circumstances.
In the old syntax, arguments are delimited (separated) by space characters. Double-quotes must be escaped with a backslash (i.e. put a backslash in front of each double-quote).
Further interpretation of the argument string differs depending on the operating system. On Windows, your argument string is simply passed verbatim (other than the backslash in front of double-quotes) to the Windows application. Most Windows applications will allow you to put spaces within an argument value by surrounding the argument with double-quotes. In all other cases, there is no further interpretation of the arguments.
Example:
arguments = one \"two\" 'three'
Produces in Unix vanilla universe:
argument 1: one argument 2: "two" argument 3: 'three'
Here are the rules for using the new syntax:
Example:
arguments = "one ""two"" 'spacey ''quoted'' argument'"
Produces:
argument 1: one argument 2: "two" argument 3: spacey 'quoted' argument
Notice that in the new syntax, backslash has no special meaning. This is for the convenience of Windows users.
There are two different formats for specifying the environment variables: the old format and the new format. The old format is retained for backward-compatibility. It suffers from a platform-dependent syntax and the inability to insert some special characters into the environment.
The new syntax for specifying environment values:
<name>=<value>
Example:
environment = "one=1 two=""2"" three='spacey ''quoted'' value'"
Produces the following environment entries:
one=1 two="2" three=spacey 'quoted' value
Under the old syntax, there are no double quote marks surrounding the environment specification. Each environment entry remains of the form
<name>=<value>Under Unix, list multiple environment entries by separating them with a semicolon (;). Under Windows, separate multiple entries with a vertical bar (| ). There is no way to insert a literal semicolon under Unix or a literal vertical bar under Windows. Note that spaces are accepted, but rarely desired, characters within parameter names and values, because they are treated as literal characters, not separators or ignored white space. Place spaces within the parameter list only if required.
A Unix example:
environment = one=1;two=2;three="quotes have no 'special' meaning"
This produces the following:
one=1 two=2 three="quotes have no 'special' meaning"
If the environment is set with the environment command and getenv is also set to true, values specified with environment override values in the submittor's environment (regardless of the order of the environment and getenv commands).
If no path or a relative path is used, then the executable file is presumed to be relative to the current working directory of the user as the condor_submit command is issued.
If submitting into the standard universe, then the named executable must have been re-linked with the Condor libraries (such as via the condor_compile command). If submitting into the vanilla universe (the default), then the named executable need not be re-linked and can be any process which can run in the background (shell scripts work fine as well). If submitting into the Java universe, then the argument must be a compiled .class file.
If the environment is set with the environment command and getenv is also set to true, values specified with environment override values in the submittor's environment (regardless of the order of the environment and getenv commands).
Note that this command does not refer to the command-line arguments of the program. The command-line arguments are specified by the arguments command.
.
job-owner@UID_DOMAINwhere the configuration variable UID_DOMAIN is specified by the Condor site administrator. If UID_DOMAIN has not been specified, Condor sends the e-mail to:
job-owner@submit-machine-name
Note that if a program explicitly opens and writes to a file, that file should not be specified as the output file.
COMMANDS FOR MATCHMAKING
requirements = Memory > 60
rank = Memory
asks Condor to find all available machines with more than 60 megabytes of memory
and give to the job the machine with the most amount of memory.
See section 2.5.3
within the Condor Users
Manual for complete information on the syntax and available attributes
that can be used in the ClassAd expression.
requirements = Memory >= 64 && Mips > 45For scheduler and local universe jobs, the requirements expression is evaluated against the Scheduler ClassAd which represents the the condor_schedd daemon running on the submit machine, rather than a remote machine. Like all commands in the submit description file, if multiple requirements commands are present, all but the last one are ignored. By default, condor_submit appends the following clauses to the requirements expression:
. Also, see the Condor Users
Manual for complete information on the syntax and available attributes
that can be used in the ClassAd expression.
FILE TRANSFER COMMANDS
If defining should_transfer_files you must also
define when_to_transfer_output (described below).
For more information about this and other settings related to
transferring files, see section 2.5.5 on
page .
Note that should_transfer_files is not supported for jobs submitted to the grid universe.
Only the transfer of files is available; the transfer of subdirectories is not supported.
For more information about this and other settings related to
transferring files, see section 2.5.5 on
page .
For grid universe jobs, to have files other than standard output and standard error transferred from the execute machine back to the submit machine, do use transfer_output_files, listing all files to be transferred. These files are found on the execute machine in the working directory of the job.
For more information about this and other settings related to
transferring files, see section 2.5.5 on
page .
name describes an output file name produced by your job, and newname describes the file name it should be downloaded to. Multiple remaps can be specified by separating each with a semicolon. If you wish to remap file names that contain equals signs or semicolons, these special characters may be escaped with a backslash.
Setting when_to_transfer_output equal to ON_EXIT will cause Condor to transfer the job's output files back to the submitting machine only when the job completes (exits on its own).
The ON_EXIT_OR_EVICT option is intended for fault tolerant jobs which periodically save their own state and can restart where they left off. In this case, files are spooled to the submit machine any time the job leaves a remote site, either because it exited on its own, or was evicted by the Condor system for any reason prior to job completion. The files spooled back are placed in a directory defined by the value of the SPOOL configuration variable. Any output files transferred back to the submit machine are automatically sent back out again as input files if the job restarts.
For more information about this and other settings related to
transferring files, see section 2.5.5 on
page .
POLICY COMMANDS
For example: Suppose a job is known to run for a minimum of an hour. If the job exits after less than an hour, the job should be placed on hold and an e-mail notification sent, instead of being allowed to leave the queue.
on_exit_hold = (CurrentTime - JobStartDate) < (60 * $(MINUTE))
This expression places the job on hold if it exits for any reason before running for an hour. An e-mail will be sent to the user explaining that the job was placed on hold because this expression became True.
periodic_* expressions take precedence over on_exit_* expressions, and *_hold expressions take precedence over a *_remove expressions.
Only job ClassAd attributes will be defined for use by this ClassAd expression. This expression is available for the vanilla, java, parallel, grid, local and scheduler universes. It is additionally available, when submitted from a Unix machine, for the standard universe.
For example, suppose a job occasionally segfaults, but chances are that the job will finish successfully if the job is run again with the same data. The on_exit_remove expression can cause the job to run again with the following command. Assume that the signal identifier for the segmentation fault is 11 on the platform where the job will be running.
on_exit_remove = (ExitBySignal == False) || (ExitSignal != 11)This expression lets the job leave the queue if the job was not killed by a signal or if it was killed by a signal other than 11, representing segmentation fault in this example. So, if the exited due to signal 11, it will stay in the job queue. In any other case of the job exiting, the job will leave the queue as it normally would have done.
As another example, if the job should only leave the queue if it exited on its own with status 0, this on_exit_remove expression works well:
on_exit_remove = (ExitBySignal == False) && (ExitCode == 0)If the job was killed by a signal or exited with a non-zero exit status, Condor would leave the job in the queue to run again.
periodic_* expressions take precedence over on_exit_* expressions, and *_hold expressions take precedence over a *_remove expressions.
Only job ClassAd attributes will be defined for use by this ClassAd expression. This expression is available for the vanilla, java, parallel, grid, local and scheduler universes. It is additionally available, when submitted from a Unix machine, for the standard universe. Note that the condor_schedd daemon, by default, only checks these periodic expressions once every 300 seconds. The period of these evaluations can be adjusted by setting the PERIODIC_EXPR_INTERVAL configuration macro.
periodic_* expressions take precedence over on_exit_* expressions, and *_hold expressions take precedence over a *_remove expressions.
Only job ClassAd attributes will be defined for use by this ClassAd expression. This expression is available for the vanilla, java, parallel, grid, local and scheduler universes. It is additionally available, when submitted from a Unix machine, for the standard universe. Note that the condor_schedd daemon, by default, only checks these periodic expressions once every 300 seconds. The period of these evaluations can be adjusted by setting the PERIODIC_EXPR_INTERVAL configuration macro.
Only job ClassAd attributes will be defined for use by this ClassAd expression. This expression is available for the vanilla, java, parallel, grid, local and scheduler universes. It is additionally available, when submitted from a Unix machine, for the standard universe. Note that the condor_schedd daemon, by default, only checks periodic expressions once every 300 seconds. The period of these evaluations can be adjusted by setting the PERIODIC_EXPR_INTERVAL configuration macro.
See section 9, the Examples section of the condor_submit manual page, for an example of a periodic_remove expression.
periodic_* expressions take precedence over on_exit_* expressions, and *_hold expressions take precedence over a *_remove expressions. So, the periodic_remove expression takes precedent over the on_exit_remove expression, if the two describe conflicting actions.
Only job ClassAd attributes will be defined for use by this ClassAd expression. This expression is available for the vanilla, java, parallel, grid, local and scheduler universes. It is additionally available, when submitted from a Unix machine, for the standard universe. Note that the condor_schedd daemon, by default, only checks periodic expressions once every 300 seconds. The period of these evaluations can be adjusted by setting the PERIODIC_EXPR_INTERVAL configuration macro.
This command has been historically used to implement a form of job start throttling from the job submitter's perspective. It was effective for the case of multiple job submission where the transfer of extremely large input data sets to the execute machine caused machine performance to suffer. This command is no longer useful, as throttling should be accomplished through configuration of the condor_schedd daemon.
COMMANDS SPECIFIC TO THE STANDARD UNIVERSE
#! /bin/sh
# get the host name of the machine
$host=`uname -n`
# grab a standard universe executable designed specifically
# for this host
scp elsewhere@cs.wisc.edu:${host} executable
# The PID MUST stay the same, so exec the new standard universe process.
exec executable ${1+"$@"}
If this command is not present (defined), then the value
defaults to false.
If your job attempts to access a file mentioned in this list, Condor will force all writes to that file to be appended to the end. Furthermore, condor_submit will not truncate it. This list uses the same syntax as compress_files, shown above.
This option may yield some surprising results. If several jobs attempt to write to the same file, their output may be intermixed. If a job is evicted from one or more machines during the course of its lifetime, such an output file might contain several copies of the results. This option should be only be used when you wish a certain file to be treated as a running log instead of a precise result.
This option only applies to standard-universe jobs.
These options only apply to standard-universe jobs.
If needed, you may set the buffer controls individually for each file using the buffer_files option. For example, to set the buffer size to 1 Mbyte and the block size to 256 Kbytes for the file input.data, use this command:
buffer_files = "input.data=(1000000,256000)"
Alternatively, you may use these two options to set the default sizes for all files used by your job:
buffer_size = 1000000 buffer_block_size = 256000
If you do not set these, Condor will use the values given by these two configuration file macros:
DEFAULT_IO_BUFFER_SIZE = 1000000 DEFAULT_IO_BUFFER_BLOCK_SIZE = 256000
Finally, if no other settings are present, Condor will use a buffer of 512 Kbytes and a block size of 32 Kbytes.
If your job attempts to access any of the files mentioned in this list, Condor will automatically compress them (if writing) or decompress them (if reading). The compress format is the same as used by GNU gzip.
The files given in this list may be simple file names or complete paths and may include * as a wild card. For example, this list causes the file /tmp/data.gz, any file named event.gz, and any file ending in .gzip to be automatically compressed or decompressed as needed:
compress_files = /tmp/data.gz, event.gz, *.gzipDue to the nature of the compression format, compressed files must only be accessed sequentially. Random access reading is allowed but is very slow, while random access writing is simply not possible. This restriction may be avoided by using both compress_files and fetch_files at the same time. When this is done, a file is kept in the decompressed state at the execution machine, but is compressed for transfer to its original location.
This option only applies to standard universe jobs.
This option only applies to standard universe jobs.
Directs Condor to use a new file name in place of an old one. name
describes a file name that your job may attempt to open, and newname
describes the file name it should be replaced with.
newname may include an optional leading
access specifier, local: or remote:. If left unspecified,
the default access specifier is remote:. Multiple remaps can be
specified by separating each with a semicolon.
This option only applies to standard universe jobs.
If you wish to remap file names that contain equals signs or semicolons, these special characters may be escaped with a backslash.
file_remaps = "dataset.1=other.dataset"
file_remaps = "very.big = local:/bigdisk/bigfile"
file_remaps = "very.big = local:/bigdisk/bigfile ; dataset.1 = other.dataset"
If your job attempts to access a file mentioned in this list, Condor will cause it to be read or written at the execution machine. This is most useful for temporary files not used for input or output. This list uses the same syntax as compress_files, shown above.
local_files = /tmp/*
This option only applies to standard universe jobs.
COMMANDS FOR THE GRID
For a grid-type-string of amazon, no additional parameters are used. See section 5.3.7 for details.
For a grid-type-string of condor, the first parameter is the name of the remote condor_schedd daemon. The second parameter is the name of the pool to which the remote condor_schedd daemon belongs. See section 5.3.1 for details.
For a grid-type-string of cream, there are three parameters. The first parameter is the web services address of the CREAM server. The second parameter is the name of the batch system that sits behind the CREAM server. The third parameter identifies a site-specific queue within the batch system. See section 5.3.8 for details.
For a grid-type-string of gt2, the single parameter is the name of the pre-WS GRAM resource to be used. See section 5.3.2 for details.
For a grid-type-string of gt4, the first parameter is the name of the WS GRAM service to be used. The second parameter is the name of WS resource to be used (usually the name of the back-end scheduler). See section 5.3.2 for details.
For a grid-type-string of gt5, the single parameter is the name of the pre-WS GRAM resource to be used, which is the same as for the grid-type-string of gt2. See section 5.3.2 for details.
For a grid-type-string of lsf, no additional parameters are used. See section 5.3.6 for details.
For a grid-type-string of nordugrid, the single parameter is the name of the NorduGrid resource to be used. See section 5.3.3 for details.
For a grid-type-string of pbs, no additional parameters are used. See section 5.3.5 for details.
For a grid-type-string of unicore, the first parameter is the name of the Unicore Usite to be used. The second parameter is the name of the Unicore Vsite to be used. See section 5.3.4 for details.
For transferring files other than stdin, see transfer_input_files.
For transferring files other than stdout, see transfer_output_files.
x509userproxy is relevant when the universe is grid, and the type of grid system is one of gt2, gt4, or nordugrid.
COMMANDS FOR PARALLEL, JAVA, and SCHEDULER UNIVERSES
remove_kill_sig = SIGUSR1 remove_kill_sig = 10If this command is not present, the value of kill_sig is used.
COMMANDS FOR THE VM UNIVERSE
An example that specifies two disk files:
xen_disk = /myxen/diskfile.img:sda1:w,/myxen/swap.img:sda2:w
ADVANCED COMMANDS
:)
and the numerical value.
See section 3.13.12 for details on concurrency limits.
See section 2.12.1 for further details and examples.
See section 2.12.1 for further details.
See section 2.12.1 for further details and examples.
Due to implementation details, a deferral time may not be used for scheduler universe jobs.
See section 2.12.1 for further details and examples.
For vanilla universe jobs where there is a shared file system, it is the current working directory on the machine where the job is executed.
For vanilla or grid universe jobs where file transfer mechanisms are utilized (there is not a shared file system), it is the directory on the machine from which the job is submitted where the input files come from, and where the job's output files go to.
For standard universe jobs, it is the directory on the machine from which the job is submitted where the condor_shadow daemon runs; the current working directory for file input and output accomplished through remote system calls.
For scheduler universe jobs, it is the directory on the machine from which the job is submitted where the job runs; the current working directory for file input and output with respect to relative path names.
Note that the path to the executable is not relative to initialdir; if it is a relative path, it is relative to the directory in which the condor_submit command is run.
SIGTSTP which tells the Condor libraries to initiate a checkpoint
of the process. For jobs submitted to the vanilla universe,
the default
is SIGTERM which is the standard way to terminate a program in Unix.
LastMatchName0 = "most-recent-Name" LastMatchName1 = "next-most-recent-Name"
The value for each introduced ClassAd is given by the
value of the Name attribute
from the machine ClassAd of a previous execution (match).
As a job is matched, the definitions for these attributes
will roll,
with LastMatchName1 becoming LastMatchName2,
LastMatchName0 becoming LastMatchName1,
and LastMatchName0 being set by the most recent
value of the Name attribute.
An intended use of these job attributes is in the requirements expression. The requirements can allow a job to prefer a match with either the same or a different resource than a previous match.
When a resource claim is to be preempted, this expression in the submit file specifies the maximum run time of the job (in seconds, since the job started). This expression has no effect, if it is greater than the maximum retirement time provided by the machine policy. If the resource claim is not preempted, this expression and the machine retirement policy are irrelevant. If the resource claim is preempted and the job finishes sooner than the maximum time, the claim closes gracefully and all is well. If the resource claim is preempted and the job does not finish in time, the usual preemption procedure is followed (typically a soft kill signal, followed by some time to gracefully shut down, followed by a hard kill signal).
Standard universe jobs and any jobs running with nice_user priority have a default max_job_retirement_time of 0, so no retirement time is utilized by default. In all other cases, no default value is provided, so the maximum amount of retirement time is utilized by default.
Setting this expression does not affect the job's resource requirements or preferences. For a job to only run on a machine with a minimum , or to preferentially run on such machines, explicitly specify this in the requirements and/or rank expressions.
In addition to commands, the submit description file can contain macros and comments:
<macro_name> = <string>
Three pre-defined macros are supplied by the submit description file parser.
The third of the pre-defined macros is only relevant to MPI applications
under the parallel universe.
The
$(Cluster) macro supplies the value of the
ClusterId job
ClassAd attribute, and the
$(Process) macro supplies the value of the
ProcId job
ClassAd attribute.
These macros are
intended to aid in the specification of input/output files, arguments,
etc., for clusters with lots of jobs, and/or could be used to supply a
Condor process with its own cluster and process numbers on the command
line.
The
$(Node) macro is defined for MPI applications run
as parallel universe jobs.
It is a unique value assigned for the duration of the job
that essentially identifies the machine on which a program is
executing.
To use the dollar sign character ($) as a literal,
without macro expansion, use
$(DOLLAR)
In addition to the normal macro, there is also a special kind of macro called a substitution macro that allows the substitution of a ClassAd attribute value defined on the resource machine itself (gotten after a match to the machine has been made) into specific commands within the submit description file. The substitution macro is of the form:
$$(attribute)
A common use of this macro is for the heterogeneous submission of an executable:
executable = povray.$$(opsys).$$(arch)Values for the opsys and arch attributes are substituted at match time for any given resource. This allows Condor to automatically choose the correct executable for the matched machine.
An extension to the syntax of the substitution macro provides an alternative string to use if the machine attribute within the substitution macro is undefined. The syntax appears as:
$$(attribute:string_if_attribute_undefined)
An example using this extended syntax provides a path name to a required input file. Since the file can be placed in different locations on different machines, the file's path name is given as an argument to the program.
argument = $$(input_file_path:/usr/foo)On the machine, if the attribute input_file_path is not defined, then the path /usr/foo is used instead.
A further extension to the syntax of the substitution macro allows the evaluation of a ClassAd expression to define the value. As all substitution macros, the expression is evaluated after a match has been made. Therefore, the expression may refer to machine attributes by prefacing them with the scope resolution prefix TARGET., as specified in section 4.1.2. To place a ClassAd expression into the substitution macro, square brackets are added to delimit the expression. The syntax appears as:
$$([ClassAd expression])An example of a job that uses this syntax may be one that wants to know how much memory it can use. The application cannot detect this itself, as it would potentially use all of the memory on a multi-slot machine. So the job determines the memory per slot, reducing it by 10% to account for miscellaneous overhead, and passes this as a command line argument to the application. In the submit description file will be
arguments=--memory $$([TARGET.Memory * 0.9])
To insert two dollar sign characters ($$) as literals
into a ClassAd string, use
$$(DOLLARDOLLAR)
The environment macro, $ENV, allows the evaluation of an environment variable to be used in setting a submit description file command. The syntax used is
$ENV(variable)An example submit description file command that uses this functionality evaluates the submittor's home directory in order to set the path and file name of a log file:
log = $ENV(HOME)/jobs/logfileThe environment variable is evaluated when the submit description file is processed.
The $RANDOM_CHOICE macro allows a random choice to be made from a given list of parameters at submission time. For an expression, if some randomness needs to be generated, the macro may appear as
$RANDOM_CHOICE(0,1,2,3,4,5,6)
When evaluated, one of the parameters values will be chosen.
In addition to the examples of submit description files given in the condor_submit manual page, here are a few more.
Example 1 is one of the simplest submit description files possible. It queues up one copy of the program foo (which had been created by condor_compile) for execution by Condor. Since no platform is specified, Condor will use its default, which is to run the job on a machine which has the same architecture and operating system as the machine from which it was submitted. No input, output, and error commands are given in the submit description file, so the files stdin, stdout, and stderr will all refer to /dev/null. The program may produce output by explicitly opening a file and writing to it. A log file, foo.log, will also be produced that contains events the job had during its lifetime inside of Condor. When the job finishes, its exit conditions will be noted in the log file. It is recommended that you always have a log file so you know what happened to your jobs.
####################
#
# Example 1
# Simple condor job description file
#
####################
Executable = foo
Universe = standard
Log = foo.log
Queue
Example 2 queues two copies of the program mathematica. The first copy will run in directory run_1, and the second will run in directory run_2. For both queued copies, stdin will be test.data, stdout will be loop.out, and stderr will be loop.error. There will be two sets of files written, as the files are each written to their own directories. This is a convenient way to organize data if you have a large group of Condor jobs to run. The example file shows program submission of mathematica as a vanilla universe job. This may be necessary if the source and/or object code to program mathematica is not available.
####################
#
# Example 2: demonstrate use of multiple
# directories for data organization.
#
####################
Executable = mathematica
Universe = vanilla
input = test.data
output = loop.out
error = loop.error
Log = loop.log
Initialdir = run_1
Queue
Initialdir = run_2
Queue
The submit description file for Example 3 queues 150 runs of program foo which has been compiled and linked for Sun workstations running Solaris 8. This job requires Condor to run the program on machines which have greater than 32 megabytes of physical memory, and expresses a preference to run the program on machines with more than 64 megabytes, if such machines are available. It also advises Condor that it will use up to 28 megabytes of memory when running. Each of the 150 runs of the program is given its own process number, starting with process number 0. So, files stdin, stdout, and stderr will refer to in.0, out.0, and err.0 for the first run of the program, in.1, out.1, and err.1 for the second run of the program, and so forth. A log file containing entries about when and where Condor runs, checkpoints, and migrates processes for all the 150 queued programs will be written into the single file foo.log.
#################### # # Example 3: Show off some fancy features including # use of pre-defined macros and logging. # #################### Executable = foo Universe = standard Requirements = Memory >= 32 && OpSys == "SOLARIS28" && Arch =="SUN4u" Rank = Memory >= 64 Image_Size = 28 Meg Error = err.$(Process) Input = in.$(Process) Output = out.$(Process) Log = foo.log Queue 150
The requirements and rank commands in the submit description file are powerful and flexible. Using them effectively requires care, and this section presents those details.
Both requirements and rank need to be specified
as valid Condor ClassAd expressions, however, default values are set by the
condor_submit program if these are not defined in the submit description file.
From the condor_submit manual page and the above examples, you see
that writing ClassAd expressions is intuitive, especially if you
are familiar with the programming language C. There are some
pretty nifty expressions you can write with ClassAds.
A complete description of ClassAds and their expressions
can be found in section 4.1 on
page .
All of the commands in the submit description file are case insensitive, except for the ClassAd attribute string values. ClassAds attribute names are case insensitive, but ClassAd string values are case preserving.
Note that the comparison operators
(<, >, <=, >=, and ==)
compare strings
case insensitively. The special comparison operators
=?= and =!=
compare strings case sensitively.
A requirements or rank command in
the submit description file may utilize attributes
that appear in a machine or a job ClassAd.
Within the submit description file (for a job) the
prefix MY. (on a ClassAd attribute name)
causes a reference to the job ClassAd attribute,
and the prefix TARGET. causes a reference to
a potential machine or matched machine ClassAd attribute.
The condor_status command displays statistics about machines within the pool. The -l option displays the machine ClassAd attributes for all machines in the Condor pool. The job ClassAds, if there are jobs in the queue, can be seen with the condor_q -l command. This shows all the defined attributes for current jobs in the queue.
A list of defined ClassAd attributes for job ClassAds
is given in the unnumbered Appendix on
page .
A list of defined ClassAd attributes for machine ClassAds
is given in the unnumbered Appendix on
page .
When considering the match between a job and a machine, rank is used to choose a match from among all machines that satisfy the job's requirements and are available to the user, after accounting for the user's priority and the machine's rank of the job. The rank expressions, simple or complex, define a numerical value that expresses preferences.
The job's rank expression evaluates to one of three values. It can be UNDEFINED, ERROR, or a floating point value. If rank evaluates to a floating point value, the best match will be the one with the largest, positive value. If no rank is given in the submit description file, then Condor substitutes a default value of 0.0 when considering machines to match. If the job's rank of a given machine evaluates to UNDEFINED or ERROR, this same value of 0.0 is used. Therefore, the machine is still considered for a match, but has no rank above any other.
A boolean expression evaluates to the numerical value of 1.0 if true, and 0.0 if false.
The following rank expressions provide examples to follow.
For a job that desires the machine with the most available memory:
Rank = memory
For a job that prefers to run on a friend's machine on Saturdays and Sundays:
Rank = ( (clockday == 0) || (clockday == 6) )
&& (machine == "friend.cs.wisc.edu")
For a job that prefers to run on one of three specific machines:
Rank = (machine == "friend1.cs.wisc.edu") ||
(machine == "friend2.cs.wisc.edu") ||
(machine == "friend3.cs.wisc.edu")
For a job that wants the machine with the best floating point performance (on Linpack benchmarks):
Rank = kflopsThis particular example highlights a difficulty with rank expression evaluation as currently defined. While all machines have floating point processing ability, not all machines will have the kflops attribute defined. For machines where this attribute is not defined, Rank will evaluate to the value UNDEFINED, and Condor will use a default rank of the machine of 0.0. The rank attribute will only rank machines where the attribute is defined. Therefore, the machine with the highest floating point performance may not be the one given the highest rank.
So, it is wise when writing a rank expression to check if the expression's evaluation will lead to the expected resulting ranking of machines. This can be accomplished using the condor_status command with the -constraint argument. This allows the user to see a list of machines that fit a constraint. To see which machines in the pool have kflops defined, use
condor_status -constraint kflopsAlternatively, to see a list of machines where kflops is not defined, use
condor_status -constraint "kflops=?=undefined"
For a job that prefers specific machines in a specific order:
Rank = ((machine == "friend1.cs.wisc.edu")*3) +
((machine == "friend2.cs.wisc.edu")*2) +
(machine == "friend3.cs.wisc.edu")
If the machine being ranked is "friend1.cs.wisc.edu", then the
expression
(machine == "friend1.cs.wisc.edu")is true, and gives the value 1.0. The expressions
(machine == "friend2.cs.wisc.edu")and
(machine == "friend3.cs.wisc.edu")are false, and give the value 0.0. Therefore, rank evaluates to the value 3.0. In this way, machine "friend1.cs.wisc.edu" is ranked higher than machine "friend2.cs.wisc.edu", machine "friend2.cs.wisc.edu" is ranked higher than machine "friend3.cs.wisc.edu", and all three of these machines are ranked higher than others.
If vanilla, java, or parallel universe jobs are submitted without using the File Transfer mechanism, Condor must use a shared file system to access input and output files. In this case, the job must be able to access the data files from any machine on which it could potentially run.
As an example, suppose a job is submitted from blackbird.cs.wisc.edu, and the job requires a particular data file called /u/p/s/psilord/data.txt. If the job were to run on cardinal.cs.wisc.edu, the file /u/p/s/psilord/data.txt must be available through either NFS or AFS for the job to run correctly.
Condor allows users to ensure their jobs have access to the right shared files by using the FileSystemDomain and UidDomain machine ClassAd attributes. These attributes specify which machines have access to the same shared file systems. All machines that mount the same shared directories in the same locations are considered to belong to the same file system domain. Similarly, all machines that share the same user information (in particular, the same UID, which is important for file systems like NFS) are considered part of the same UID domain.
The default configuration for Condor places each machine in its own UID domain and file system domain, using the full host name of the machine as the name of the domains. So, if a pool does have access to a shared file system, the pool administrator must correctly configure Condor such that all the machines mounting the same files have the same FileSystemDomain configuration. Similarly, all machines that share common user information must be configured to have the same UidDomain configuration.
When a job relies on a shared file system, Condor uses the requirements expression to ensure that the job runs on a machine in the correct UidDomain and FileSystemDomain. In this case, the default requirements expression specifies that the job must run on a machine with the same UidDomain and FileSystemDomain as the machine from which the job is submitted. This default is almost always correct. However, in a pool spanning multiple UidDomains and/or FileSystemDomains, the user may need to specify a different requirements expression to have the job run on the correct machines.
For example, imagine a pool made up of both desktop workstations and a dedicated compute cluster. Most of the pool, including the compute cluster, has access to a shared file system, but some of the desktop machines do not. In this case, the administrators would probably define the FileSystemDomain to be cs.wisc.edu for all the machines that mounted the shared files, and to the full host name for each machine that did not. An example is jimi.cs.wisc.edu.
In this example, a user wants to submit vanilla universe jobs from her own desktop machine (jimi.cs.wisc.edu) which does not mount the shared file system (and is therefore in its own file system domain, in its own world). But, she wants the jobs to be able to run on more than just her own machine (in particular, the compute cluster), so she puts the program and input files onto the shared file system. When she submits the jobs, she needs to tell Condor to send them to machines that have access to that shared data, so she specifies a different requirements expression than the default:
Requirements = TARGET.UidDomain == "cs.wisc.edu" && \
TARGET.FileSystemDomain == "cs.wisc.edu"
WARNING: If there is no shared file system, or the Condor pool administrator does not configure the FileSystemDomain setting correctly (the default is that each machine in a pool is in its own file system and UID domain), a user submits a job that cannot use remote system calls (for example, a vanilla universe job), and the user does not enable Condor's File Transfer mechanism, the job will only run on the machine from which it was submitted.
Condor works well without a shared file system. The Condor file transfer mechanism is utilized by the user when the user submits jobs. Condor will transfer any files needed by a job from the machine where the job was submitted into a temporary working directory on the machine where the job is to be executed. Condor executes the job and transfers output back to the submitting machine. The user specifies which files to transfer, and at what point the output files should be copied back to the submitting machine. This specification is done within the job's submit description file.
The default behavior of the file transfer mechanism varies across the different Condor universes, and it differs between UNIX and Windows machines.
For jobs submitted under the standard universe, the existence of a shared file system is not relevant. Access to files (input and output) is handled through Condor's remote system call mechanism. The executable and checkpoint files are transferred automatically, when needed. Therefore, the user does not need to change the submit description file if there is no shared file system.
For the vanilla, java, and parallel universes, access to files (including the executable) through a shared file system is presumed as a default on UNIX machines. If there is no shared file system, then Condor's file transfer mechanism must be explicitly enabled. When submitting a job from a Windows machine, Condor presumes the opposite: no access to a shared file system. It instead enables the file transfer mechanism by default. Submission of a job might need to specify which files to transfer, and/or when to transfer the output files back.
For the grid universe, jobs are to be executed on remote machines, so there would never be a shared file system between machines. See section 5.3.2 for more details.
For the scheduler universe, Condor is only using the machine from which the job is submitted. Therefore, the existence of a shared file system is not relevant.
To enable the file transfer mechanism, two commands are placed in the job's submit description file: should_transfer_files and when_to_transfer_output. An example is:
should_transfer_files = YES when_to_transfer_output = ON_EXIT
The should_transfer_files command specifies whether Condor should transfer input files from the submit machine to the remote machine where the job executes. It also specifies whether the output files are transferred back to the submit machine. The command takes on one of three possible values:
YES: Condor always transfers both input and output files.
IF_NEEDED: Condor transfers files if the job is
matched with (and to be executed on) a machine in a
different FileSystemDomain than the
one the submit machine belongs to.
If the job is matched with a machine in the local FileSystemDomain,
Condor will not transfer files and relies
on a shared file system.
NO: Condor's file transfer mechanism is disabled.
The when_to_transfer_output command tells Condor when output files are to be transferred back to the submit machine after the job has executed on a remote machine. The command takes on one of two possible values:
ON_EXIT: Condor transfers output files back to the submit
machine only when the job exits on its own.
ON_EXIT_OR_EVICT:
Condor will always do the transfer,
whether the job completes on its own, is preempted by another job,
vacates the machine, or is killed.
As the job completes on its own, files are transferred back
to the directory where the job was submitted, as expected.
For the other cases, files are transferred back at eviction time.
These files are placed in
the directory defined by the configuration
variable SPOOL, not the directory from which the
job was submitted.
The transferred files are named using the
ClusterId and ProcId job ClassAd attributes.
The file name takes the form:
cluster<X>.proc<Y>.subproc0where
<X> is the value of ClusterId, and
<Y> is the value of ProcId.
As an example, job 735.0 may produce the file
$(SPOOL)/cluster735.proc0.subproc0
This is only useful if partial runs of the job are valuable. An example of valuable partial runs is when the application produces its own checkpoints.
There is no default value for when_to_transfer_output.
If using the file transfer mechanism,
this command must be defined.
If when_to_transfer_output is specified in the submit
description file,
but should_transfer_files is not, Condor assumes a
value of YES for should_transfer_files.
NOTE: The combination of:
should_transfer_files = IF_NEEDED when_to_transfer_output = ON_EXIT_OR_EVICTwould produce undefined file access semantics. Therefore, this combination is prohibited by condor_submit.
When submitting from a Unix platform,
the file transfer mechanism is unused by default.
If neither when_to_transfer_output or should_transfer_files
are defined, Condor assumes
should_transfer_files = NO.
When submitting from a Windows platform, Condor does not provide any way to use a shared file system for jobs. Therefore, if neither when_to_transfer_output or should_transfer_files are defined, the file transfer mechanism is enabled by default with the following values:
should_transfer_files = YES when_to_transfer_output = ON_EXIT
If the file transfer mechanism is enabled, Condor will transfer the following files before the job is run on a remote machine.
transfer_input_files = file1,file2
As a default, for jobs other than those submitted to the grid universe, any files that are modified or created by the job in the temporary directory at the remote site are transferred back to the machine from which the job was submitted. Most of the time, this is the best option. To restrict the files that are transferred, specify the exact list of files with transfer_output_files. Delimit these file names with a comma. When this list is defined, and any of the files do not exist as the job exits, Condor considers this an error, and re-runs the job.
WARNING: Do not specify transfer_output_files (for other than grid universe jobs) unless there is a really good reason - it is best to let Condor figure things out by itself based upon what output the job produces.
For grid universe jobs, files to be transferred (other than standard output and standard error) must be specified using transfer_output_files in the submit description file.
The file transfer mechanism specifies file names and/or paths on both the file system of the submit machine and on the file system of the execute machine. Care must be taken to know which machine (submit or execute) is utilizing the file name and/or path.
Files in the transfer_input_files command are specified as they are accessed on the submit machine. The program (as it executes) accesses files as they are found on the execute machine.
There are three ways to specify files and paths for transfer_input_files:
Before executing the program, Condor copies the executable, an input file as specified by the submit command input, along with any input files specified by transfer_input_files. All these files are placed into a temporary directory (on the execute machine) in which the program runs. Therefore, the executing program must access input files without paths. Because all transferred files are placed into a single, flat directory, input files must be uniquely named to avoid collision when transferred. A collision causes the last file in the list to overwrite the earlier one.
If the program creates output files during execution, it must create them within the temporary working directory. Condor transfers back all files within the temporary working directory that have been modified or created. To transfer back only a subset of these files, the submit command transfer_output_files is defined. Transfer of files that exist, but are not within the temporary working directory is not supported. Condor's behavior in this instance is undefined.
It is okay to create files outside the temporary working directory on the file system of the execute machine, (in a directory such as /tmp) if this directory is guaranteed to exist and be accessible on all possible execute machines. However, transferring such a file back after execution completes may not be done.
Here are several examples to illustrate the use of file transfer. The program executable is called my_program, and it uses three command-line arguments as it executes: two input file names and an output file name. The program executable and the submit description file for this job are located in directory /scratch/test.
The directory tree for all these examples:
/scratch/test (directory)
my_program.condor (the submit description file)
my_program (the executable)
files (directory)
logs2 (directory)
in1 (file)
in2 (file)
logs (directory)
This simple example explicitly transfers input files. These input files to be transferred are specified relative to the directory where the job is submitted. The single output file, out1, created when the job is executed will be transferred back into the directory /scratch/test, not the files directory.
# file name: my_program.condor # Condor submit description file for my_program Executable = my_program Universe = vanilla Error = logs/err.$(cluster) Output = logs/out.$(cluster) Log = logs/log.$(cluster) should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = files/in1, files/in2 Arguments = in1 in2 out1 Queue
This second example is identical to Example 1, except that absolute paths to the input files are specified, instead of relative paths to the input files.
# file name: my_program.condor # Condor submit description file for my_program Executable = my_program Universe = vanilla Error = logs/err.$(cluster) Output = logs/out.$(cluster) Log = logs/log.$(cluster) should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = /scratch/test/files/in1, /scratch/test/files/in2 Arguments = in1 in2 out1 Queue
This third example illustrates the use of the
submit command initialdir, and its effect
on the paths used for the various files.
The expected location of the
executable is not affected by the
initialdir command.
All other files
(specified by input,
output,
transfer_input_files,
as well as files modified or created by the job
and automatically transferred back)
are located relative to the specified initialdir.
Therefore, the output file, out1,
will be placed in the files directory.
Note that the logs2 directory
exists to make this example work correctly.
# file name: my_program.condor # Condor submit description file for my_program Executable = my_program Universe = vanilla Error = logs2/err.$(cluster) Output = logs2/out.$(cluster) Log = logs2/log.$(cluster) initialdir = files should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = in1, in2 Arguments = in1 in2 out1 Queue
This example illustrates a job that will fail. The files specified using the transfer_input_files command work correctly (see Example 1). However, relative paths to files in the arguments command cause the executing program to fail. The file system on the submission side may utilize relative paths to files, however those files are placed into a single, flat, temporary directory on the execute machine.
Note that this specification and submission will cause the job to fail and re-execute.
# file name: my_program.condor # Condor submit description file for my_program Executable = my_program Universe = vanilla Error = logs/err.$(cluster) Output = logs/out.$(cluster) Log = logs/log.$(cluster) should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = files/in1, files/in2 Arguments = files/in1 files/in2 files/out1 Queue
This example fails with the following error:
err: files/out1: No such file or directory.
As with Example 4, this example illustrates a job that will fail. The executing program's use of absolute paths cannot work.
# file name: my_program.condor # Condor submit description file for my_program Executable = my_program Universe = vanilla Error = logs/err.$(cluster) Output = logs/out.$(cluster) Log = logs/log.$(cluster) should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = /scratch/test/files/in1, /scratch/test/files/in2 Arguments = /scratch/test/files/in1 /scratch/test/files/in2 /scratch/test/files/out1 Queue
The job fails with the following error:
err: /scratch/test/files/out1: No such file or directory.
This example illustrates a failure case where the executing program creates an output file in a directory other than within the single, flat, temporary directory that the program executes within. The file creation may or may not cause an error, depending on the existence and permissions of the directories on the remote file system.
Further incorrect usage is seen during the attempt to transfer the output file back using the transfer_output_files command. The behavior of Condor for this case is undefined.
# file name: my_program.condor # Condor submit description file for my_program Executable = my_program Universe = vanilla Error = logs/err.$(cluster) Output = logs/out.$(cluster) Log = logs/log.$(cluster) should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = files/in1, files/in2 transfer_output_files = /tmp/out1 Arguments = in1 in2 /tmp/out1 Queue
The requirements expression for a job must depend
on the should_transfer_files command.
The job must specify the correct logic to ensure that the job is matched
with a resource that meets the file transfer needs.
If no requirements expression is in the submit description file,
or if the expression specified does not refer to the
attributes listed below, condor_submit adds an
appropriate clause to the requirements expression for the job.
condor_submit appends these clauses with a logical AND, &&,
to ensure that the proper conditions are met.
Here are the default clauses corresponding to the different values of
should_transfer_files:
should_transfer_files = YES results in the addition of
the clause (HasFileTransfer).
If the job is always going to transfer files, it is required to
match with a machine that has the capability to transfer files.
should_transfer_files = NO results in the addition of
(TARGET.FileSystemDomain == MY.FileSystemDomain).
In addition, Condor automatically adds the
FileSystemDomain attribute to the job ad, with whatever
string is defined for the condor_schedd to which the job is
submitted.
If the job is not using the file transfer mechanism, Condor assumes
it will need a shared file system, and therefore, a machine in the
same FileSystemDomain as the submit machine.
should_transfer_files = IF_NEEDED results in the addition of
(HasFileTransfer || (TARGET.FileSystemDomain == MY.FileSystemDomain))If Condor will optionally transfer files, it must require that the machine is either capable of transferring files or in the same file system domain.
To ensure that the job is matched to a machine with enough local disk space to hold all the transferred files, Condor automatically adds the DiskUsage job attribute. This attribute includes the total size of the job's executable and all input files to be transferred. Condor then adds an additional clause to the Requirements expression that states that the remote machine must have at least enough available disk space to hold all these files:
&& (Disk >= DiskUsage)
If should_transfer_files = IF_NEEDED and the job prefers
to run on a machine in the local file system domain
over transferring files,
(but are still willing to allow the job to run remotely and transfer
files), the rank expression works well. Use:
rank = (TARGET.FileSystemDomain == MY.FileSystemDomain)
The rank expression is a floating point number, so if other items are considered in ranking the possible machines this job may run on, add the items:
rank = kflops + (TARGET.FileSystemDomain == MY.FileSystemDomain)
The value of kflops can vary widely among machines, so this rank expression will likely not do as it intends. To place emphasis on the job running in the same file system domain, but still consider kflops among the machines in the file system domain, weight the part of the rank expression that is matching the file system domains. For example:
rank = kflops + (10000 * (TARGET.FileSystemDomain == MY.FileSystemDomain))
The environment under which a job executes often contains information that is potentially useful to the job. Condor allows a user to both set and reference environment variables for a job or job cluster.
Within a submit description file, the user may define environment variables for the job's environment by using the environment command. See the condor_submit manual page at section 9 for more details about this command.
The submittor's entire environment can be copied into the job ClassAd for the job at job submission. The getenv command within the submit description file does this. See the condor_submit manual page at section 9 for more details about this command.
If the environment is set with the environment command and getenv is also set to true, values specified with environment override values in the submittor's environment (regardless of the order of the environment and getenv commands).
Commands within the submit description file may reference the
environment variables of the submitter as a job is submitted.
Submit description file commands use $ENV(EnvironmentVariableName)
to reference the value of an environment variable.
Again,
see the condor_submit manual page at
section 9 for more details about this usage.
Condor sets several additional environment variables for each executing job that may be useful for the job to reference.
1, just like the SlotID attribute in the machine's
ClassAd.
This setting is available in all universes.
See section 3.13.7 for more details about SMP
machines and their configuration.
If executables are available for the different platforms of machines in the Condor pool, Condor can be allowed the choice of a larger number of machines when allocating a machine for a job. Modifications to the submit description file allow this choice of platforms.
A simplified example is a cross submission. An executable is available for one platform, but the submission is done from a different platform. Given the correct executable, the requirements command in the submit description file specifies the target architecture. For example, an executable compiled for a Sun 4, submitted from an Intel architecture running Linux would add the requirement
requirements = Arch == "SUN4x" && OpSys == "SOLARIS251"Without this requirement, condor_submit will assume that the program is to be executed on a machine with the same platform as the machine where the job is submitted.
Cross submission works for all universes except scheduler and local. See section 5.3.9 for how matchmaking works in the grid universe. The burden is on the user to both obtain and specify the correct executable for the target architecture. To list the architecture and operating systems of the machines in a pool, run condor_status.
A more complex example of a heterogeneous submission occurs when a job may be executed on many different architectures to gain full use of a diverse architecture and operating system pool. If the executables are available for the different architectures, then a modification to the submit description file will allow Condor to choose an executable after an available machine is chosen.
A special-purpose Machine Ad substitution macro can be used in string attributes in the submit description file. The macro has the form
$$(MachineAdAttribute)The $$() informs Condor to substitute the requested MachineAdAttribute from the machine where the job will be executed.
An example of the heterogeneous job submission has executables available for three platforms: LINUX Intel, Solaris26 Intel, and Solaris 8 Sun. This example uses povray to render images using a popular free rendering engine.
The substitution macro chooses a specific executable after a platform for running the job is chosen. These executables must therefore be named based on the machine attributes that describe a platform. The executables named
povray.LINUX.INTEL povray.SOLARIS26.INTEL povray.SOLARIS28.SUN4uwill work correctly for the macro
povray.$$(OpSys).$$(Arch)
The executables or links to executables with this name are placed into the initial working directory so that they may be found by Condor. A submit description file that queues three jobs for this example:
####################
#
# Example of heterogeneous submission
#
####################
universe = vanilla
Executable = povray.$$(OpSys).$$(Arch)
Log = povray.log
Output = povray.out.$(Process)
Error = povray.err.$(Process)
Requirements = (Arch == "INTEL" && OpSys == "LINUX") || \
(Arch == "INTEL" && OpSys =="SOLARIS26") || \
(Arch == "SUN4u" && OpSys == "SOLARIS28")
Arguments = +W1024 +H768 +Iimage1.pov
Queue
Arguments = +W1024 +H768 +Iimage2.pov
Queue
Arguments = +W1024 +H768 +Iimage3.pov
Queue
These jobs are submitted to the vanilla universe to assure that once a job is started on a specific platform, it will finish running on that platform. Switching platforms in the middle of job execution cannot work correctly.
There are two common errors made with the substitution macro. The first is the use of a non-existent MachineAdAttribute. If the specified MachineAdAttribute does not exist in the machine's ClassAd, then Condor will place the job in the held state until the problem is resolved.
The second common error occurs due to an incomplete job set up. For example, the submit description file given above specifies three available executables. If one is missing, Condor report back that an executable is missing when it happens to match the job with a resource that requires the missing binary.
Jobs submitted to the standard universe may produce checkpoints. A checkpoint can then be used to start up and continue execution of a partially completed job. For a partially completed job, the checkpoint and the job are specific to a platform. If migrated to a different machine, correct execution requires that the platform must remain the same.
In previous versions of Condor, the author of the heterogeneous submission file would need to write extra policy expressions in the requirements expression to force Condor to choose the same type of platform when continuing a checkpointed job. However, since it is needed in the common case, this additional policy is now automatically added to the requirements expression. The additional expression is added provided the user does not use CkptArch in the requirements expression. Condor will remain backward compatible for those users who have explicitly specified CkptRequirements-implying use of CkptArch, in their requirements expression.
The expression added when the attribute CkptArch is not specified will default to
# Added by Condor
CkptRequirements = ((CkptArch == Arch) || (CkptArch =?= UNDEFINED)) && \
((CkptOpSys == OpSys) || (CkptOpSys =?= UNDEFINED))
Requirements = (<user specified policy>) && $(CkptRequirements)
The behavior of the CkptRequirements expressions and its addition to
requirements is as follows.
The CkptRequirements expression guarantees correct operation
in the two possible cases for a job.
In the first case, the job has not produced a checkpoint.
The ClassAd attributes CkptArch and CkptOpSys
will be undefined, and therefore the meta operator (=?=)
evaluates to true.
In the second case, the job has produced a checkpoint.
The Machine ClassAd is restricted to require further execution
only on a machine of the same platform.
The attributes CkptArch and CkptOpSys
will be defined, ensuring that the platform chosen for further
execution will be the same as the one used just before the
checkpoint.
Note that this restriction of platforms also applies to platforms where the executables are binary compatible.
The complete submit description file for this example:
####################
#
# Example of heterogeneous submission
#
####################
universe = standard
Executable = povray.$$(OpSys).$$(Arch)
Log = povray.log
Output = povray.out.$(Process)
Error = povray.err.$(Process)
# Condor automatically adds the correct expressions to insure that the
# checkpointed jobs will restart on the correct platform types.
Requirements = ( (Arch == "INTEL" && OpSys == "LINUX") || \
(Arch == "INTEL" && OpSys =="SOLARIS26") || \
(Arch == "SUN4u" && OpSys == "SOLARIS28") )
Arguments = +W1024 +H768 +Iimage1.pov
Queue
Arguments = +W1024 +H768 +Iimage2.pov
Queue
Arguments = +W1024 +H768 +Iimage3.pov
Queue