8.3 Development Release Series 6.9

This is the development release series of Condor. The details of each version are described below.

Version 6.9.5

Release Notes:

• There are some known vulnerabilities in the Virtual Machine (VM) universe support included in this release that can allow VM universe jobs to read arbitrary files on the host machine. VM universe is disabled by default, so this potential vulnerability does not affect most sites. However, VM universe should not be enabled unless the job policy only allows users that you trust completely to run jobs on the machine.

• Condor is now licensed under the terms of the Apache License version 2.0.

• Dropped support for the following platforms:
  • Red Hat Linux 7.x systems on the x86 processor.
  • Digital Unix systems on the Alpha processor.
  • Yellow Dog Linux 3.0 systems on the PPC processor.
  • MacOS 10.3 systems on the PPC processor.
  Theses ports are still supported in the 6.8 series of Condor.

• Dropped support for OGSA GRAM (grid-type gt3) in the grid universe. This version of GRAM is not included in recent versions of the Globus Toolkit. This does not affect Condor's support for pre-WS GRAM (grid-type gt2) or WS GRAM (grid-type gt4).

• The suggested configuration value for SHADOW_RENICE_INCREMENT has been changed from 10 to 0. If using the value 10 in an existing configuration file, we recommend changing it. This improves the performance of Condor on busy submit nodes where other processes would cause low priority condor_ shadow daemons to become starved for CPU time.

• For grid-type gt2 grid universe jobs, job arguments are now handled as they are for all other job types. Previously, the user would have to manually escape characters that had special meaning to GRAM's RSL language.

• condor_ version now includes a specific build identification number for official builds of Condor.

New Features:

• condor_ q, when Quill is enabled, now displays the last time Quill updated the database. This permits seeing how fresh the database information is.

• condor_ history, when Quill is enabled, will now query the database for historical items even when the -constraint option is given. Previously, it would go to the history file in that case.

• condor_ submit can now write the ClassAds it generates to a file instead of sending them to the condor_ schedd daemon.

• When the condor_ master daemon sends an obituary e-mail, it prints the last few lines of the log file for that daemon, and the name of the file. This e-mail now contains the full path name of that log file, not just the file name. This is more convenient for sites which run multiple instances of the same daemon on one machine.

• Added new policy for parallel universe jobs to control how they exit. If the attribute ParallelShutdownPolicy is set to the string "WAIT_FOR_ALL", then Condor will wait until every node in the parallel job has completed to consider the job finished. If this attribute is not set, or is set to any other string, the default policy is in effect. This policy is needed for MPI jobs: when the first node exits, the whole job is considered done, and condor kills all other running nodes in that parallel job.

• Added new Windows specific ClassAd attributes:
  • WindowsMajorVersion
  • WindowsMinorVersion
  • WindowsBuildNumber
  For definitions, please see the unnumbered subsection labeled Machine ClassAd Attributes on page .

• Added new authorization levels to allow fine-grained control over the security settings that are used by the collector when receiving ClassAd updates by different types of daemons: ADVERTISE_MASTER, ADVERTISE_STARTD, and ADVERTISE_SCHEDD. An example of what you can do with this is to require that all condor_ startds that join the pool be authenticated with a pool password and exist within a restricted set of IP addresses, while schedds may join the pool from a broader set of IP addresses and must authenticate with X509 credentials.

• Added ability to throttle in Condor's file transfer mechanism the maximum number of simultaneous stage-outs and stage-ins for jobs submitted from the same condor_ schedd. The configuration variables are MAX_CONCURRENT_DOWNLOADS and MAX_CONCURRENT_UPLOADS. The default is 10 simultaneous uploads of input files and 10 simultaneous downloads of output files. These limits currently do not apply to grid universe jobs or standard universe jobs.

• Added SCHEDD_QUERY_WORKERS, which is 3 by default in Unix, and which is ignored in Windows. This specifies the maximum number of concurrent sub-processes that the condor_ schedd will spawn to handle queries.

• Condor-C now uses a more efficient protocol when querying the status of jobs from Condor 6.9.5 and newer condor_ schedd daemons.

• Added 4 new counters to the job ClassAd:
  • NumJobStarts
  • NumJobReconnects
  • NumShadowExceptions
  • NumShadowStarts
  For more information, please see their descriptions in section 9 on page .

• Added a new attribute, GridJobStatus, to the ClassAds of grid universe jobs. This string shows the job's status as reported by the remote job management system.

• condor_ q -analyze now shows the full hold reason for jobs that are on hold.

• Increased efficiency of condor_ preen when there are large numbers of jobs in the job queue. Without this, the condor_ schedd would become unresponsive for a long time (e.g. 10 minutes with 20,000 jobs in the queue) whenever condor_ preen was activated.

• A 6.9.5 condor_ q can now query an older condor_ quill daemon directly for job information.

• Reduced memory requirements of condor_ shadow.

• Added the ability to condor_ submit to list unused or unexpanded variables in submission file.

• Added the capability to assign priorities to DAG nodes. Ready nodes within a DAG are submitted in priority order by condor_ dagman.

• Added the capability to assign categories to DAG nodes, and throttle submission of node jobs by category.

• USE_CLONE_TO_CREATE_PROCESSES (which defaults to True) is now supported on ppc64, SUSE 9. This also fixes a bug in which the Yellow Dog Linux version of Condor installed on a ppc64 SUSE 9 machine would fail to start jobs.

• When the condor_ preen sends email about old files being found, it now includes the name of the machine and the number of files found in the subject of the message.

• The user log reading code is now able to handle global event log rotations correctly. The API is backwards compatible, but with several new method, it is able to invisibly handle rotated event log files.

• The user log writer code now generates a header record (as a ``generic'' event) with some meta information to the event log. This header is not written to the ``user log'', only to the global event log. Some of the information stored in this header is used by the enhanced log reader (see above) to more reliably detect rotated log files.

• The Grid Monitor now refrains from polling the status of jobs that it has learned are done.

• For grid-type condor jobs, the condor_ gridmanager is now more efficient when querying the status of jobs on the remote condor_ schedd when there are jobs with different X509 subjects.

• URLs can now be given for the input and output files of grid-type nordugrid jobs in the grid universe. URLs are forwarded to the NorduGrid server, which performs the transfers.

• The length of the paths to the job's initial working directory, user log, and input/output files are no longer limited to 255 characters. Previously, condor_ submit would refuse to accept jobs exceeding this POSIX_PATH_MAX limit. Now the only limit is whatever limit the operating system enforces on the system where the files are accessed.

• The condor_ startd now receives updates to the job ClassAd from the condor_ starter. The primary benefit of this is that DiskUsage is updated and can therefore be used in policy expressions, such as PREEMPT. The frequency of updates is determined by STARTER_UPDATE_INTERVAL.

• Several improvements have been made to Condor's ability to run using privilege separation on the execute side. See section 3.6.13 for details.

• Added support on Linux systems for reliably tracking all of a job's processes using a dedicated supplementary group ID. This has the advantage of working regardless of whether a job runs using a dedicated user account. See section 3.12.10 for details.

Configuration Variable Additions and Changes:

• The new variables MAX_CONCURRENT_DOWNLOADS and MAX_CONCURRENT_UPLOADS limit the number of simultaneous file transfers that may take place through Condor's file transfer mechanism for jobs submitted to the same condor_ schedd. The default is 10 simultaneous uploads of input files and 10 simultaneous downloads of output files. These limits currently do not apply to grid universe jobs or standard universe jobs. See page  for more information.

• The default for JOB_START_DELAY has been changed from 2 seconds to 0. This means the condor_ schedd will not limit the rate at which it starts up condor_ shadow processes by default. The delay between startup of jobs may now be controlled individually for jobs using the job attribute NextJobStartDelay, which defaults to 0 seconds and is at most MAX_NEXT_JOB_START_DELAY , which defaults to 10 minutes.

• The new variable SCHEDD_QUERY_WORKERS specifies the maximum number of concurrent sub-processes that the condor_ schedd will spawn to handle queries. This is ignored in Windows. In Unix, the default is 3. See page  for more details.

• The new variable WANT_UDP_COMMAND_SOCKET controls if Condor daemons should create a UDP command socket in addition to the TCP command socket (which is required). The default is True, but it is now possible to completely disable UDP command sockets by defining this to False. See section 3.3.3 on page  for more information.

• The new variable NEGOTIATOR_INFORM_STARTD controls if the condor_ negotiator should inform the condor_ startd when it has been matched with a job. The default is True. See section 3.3.18 on page  for more information.

• The new variable SHADOW_LAZY_QUEUE_UPDATE controls if the condor_ shadow should immediately update the job queue for certain attributes (for example, the new NumJobStarts and NumJobReconnects counters) or if it should wait and only update the job queue on the next periodic update. The default is True to do lazy, periodic updates. See section 3.3.12 on page  for more information.

• The new variable WARN_ON_UNUSED_SUBMIT_FILE_MACROS controls if condor_ submit should warn when there are unused or unexpanded variables in a submit file. The default is True to list unused or unexpanded variables.

• SCHEDD_ROUND_ATTR_xxxx can now take a value that is a percentage, such as 25%. This causes the value for the attribute <xxxx> to be rounded up to the specified percentage of its closest order of magnitude. For example, a setting of 25% will cause a value near 100 to be rounded up to the next multiple of 25 and a value near 1000 will be rounded up to the next multiple of 250. The purpose of this rounding is to be able to better group similar jobs together for negotiation purposes. The configuration variables SCHEDD_ROUND_ATTR_ImageSize, SCHEDD_ROUND_ATTR_ExecutableSize, and SCHEDD_ROUND_ATTR_DiskUsage now have a default value of 25% rather than 4. The result is that instead of rounding to 10MB multiples, the rounding scales at roughly 25% of the number being rounded.

• The default for STARTER_UPDATE_INTERVAL has been changed from 20 minutes to 5 minutes.

• The new parameters PRIVSEP_ENABLED and PRIVSEP_SWITCHBOARD are required when setting up execute-side Condor to use privilege separation. See section 3.6.13 for details.

• The new parameters USE_GID_PROCESS_TRACKING , MIN_TRACKING_GID , and MAX_TRACKING_GID are used when setting up a Linux machine to use process tracking based on dedicated supplementary group IDs. See section 3.12.10 for details.

Bugs Fixed:

• Updated the SOAP API's enum UniverseType to include all the supported universes.

• Missing files that Quill needed to function now appear in the downloadable release of Condor.

• When a condor_ starter discovered a missing username in the process of discovering the owner of an executing job, a cryptic and misleading error message was emitted to the daemon log. The error text has been cleaned up to be more meaningful.

• On Windows daylight saving is handled incorrectly by stat() and fstat(). According to the MSDN, they both return the UTC time of a file; however, if daylight saving is detected, the time is adjusted by one hour, which results in the condor_ master thinking that a different version of it has been installed. In which case it recycles itself, and it's child process twice a year: not exactly what one would expect given that UTC time is not intended to pay attention to these regional changes.

• When the master starts a collector on a central manager, the master now pauses for a short time before starting any other daemons. This helps the other daemons to appear in the collector more quickly.

• Patched the parallel universe scripts lamscript and mp1script so that they work with newer versions of the GNU textutils.

• Fixed a bad bug in the standard universe-introduced in Condor 6.9.4, which would cause corruption of any binary data being written to any fd the application opened. If an application only writes ASCII data to an fd, the application will not encounter this bug.

• Condor daemons will now print an error message in the logs when <SUBSYS>_ATTRS contains attributes that are not valid ClassAd values. Previously, this was a silent error. The most common reason for this problem is an unquoted string value.

• The condor_ negotiator now prints out the value of the configuration parameter PREEMPTION_REQUIREMENTS if it is set. Previously, it always logged that it was unset, even when it was.

• Fixed bug in the master that occurred if the collector was configured to use an ephemeral command port (i.e. by explicitly setting the port to 0). The collector is now more reliable in this situation.

• Standard universe jobs are no longer restricted in the length of file names that may be passed to system calls within the program. Previously, file names approaching 255 characters or more could cause the program to crash or behave incorrectly.

• Fixed a long-standing bug causing condor_ submit to fail when given a requirements expression longer than 2048 characters.

• Fixed a bug introduced in Condor 6.9.4 that caused grid universe jobs of type gt4 to not work when the Condor daemons were started as root and any file transfer was associated with the job.

• Fixed a bug introduced in Condor 6.9.4 that caused the condor_ gridmanager to exit immediately on startup when the Condor daemons were started as root and a condor username didn't exist.

• Removed race condition that was causing the condor_ schedd to core dump on Windows when the Condor service was stopped.

• When grid universe jobs of grid-type condor, lsf, or pbs are running, condor_ q will now show the correct accumulated runtime.

• When removing grid universe jobs of type gt2 that have just finished executing, the chance of encountering Globus GRAM error 31 (the job manager failed to cancel the job as requested) is now much reduced.

• Fixed a problem introduced in 6.9.4: the condor_ schedd would hang when given a constraint with condor_ hold that included jobs that the user did not have permission to modify.

• Fixed a problem from 6.9.4 in which the schedd would not relinquish a claimed startd after reconnecting to a disconnected job. After the job finished, the startd would remain in the claimed idle state until the claim lease expired (20 minutes by default).

• Applied the QUERY_TIMEOUT to fix problem where the schedd would block for a long time when doing negotiation with flocked or HAD negotiator, and one of the collectors was not routable (for instance, when the machine is powered off). Previously, there was no time-out and would result in a the schedd waiting for the connection attempt to fail, which may take a long time.

• If both EXECUTE_LOGIN_IS_DEDICATED and DEDICATED_EXECUTE_ACCOUNT_REGEXP are defined, the latter now takes precedence, whereas previously the reverse was true.

• Fixed a problem where if STARTD_RESOURCE_PREFIX was set to anything besides slot (the default), all jobs would run using the ``condor-reuse-slot1'' (Windows) or SLOT1_USER account, regardless of the actual slot used for execution. This problem existed in versions 6.9.3 and 6.9.4 of Condor.

• Undocumented ``DAGMan helper'' functionality has been removed due to disuse

• Reworked Condor's detection of CPUs and ``Hyper Threads'' under Linux. It now correctly detects these on all machines that we've been able to test against. No configuration changes are involved in this fix.

• When a standard universe job becomes held due to user job policy or a version mismatch, a hold reason is now set in the job ad.

• Invalid QUILL_DB_TYPE settings could result in a segmentation fault in condor_ q. Condor now ignores invalid settings and assumes PostgreSQL.

• In rare cases, condor_ reconfig could cause condor_ master and one of its children to become deadlocked. This problem was only possible with security negotiation enabled, and it has therefore existed in all versions of Condor since security negotiation was added.

• Fixed a potential crash in the condor_ starter if it's told to shutdown while it's disconnected from the condor_ shadow.

• Fixed the global event log rotation code. Previously, if two or more processes were concurrently writing to the event log, they didn't correctly detect that another writer process had rotated the file, and would do their own rotation, resulting in data loss.

• Fixed a bug in the condor_ schedd that could cause it to not match any jobs for long periods of time.

• Fixed potential crash when GCB was turned on.

• Removed spurious attempts to open file /home/condor/execute/dir_#####/userdir/externals/install/globus-4.0.5/cert.pem when SSL authentication is enabled.

• Fixed problem where local universe jobs could leave stray processes behind after termination.

• Fixed a memory leak that affected all daemons receiving ClassAds via the network if encryption were enabled. This bug existed in versions 6.9.3 and 6.9.4.

• On Windows, fixed a problem that could cause spurious failures with Condor-C or with streaming a job's standard output or error.

Known Bugs:

• Condor on MacOSX 10.4 on the PowerPC architecture. will report zero image size and resident set size for jobs. This is due to bugs in the MacOSX 10.4 kernel on the PowerPC.

• There are some known vulnerabilities in the Virtual Machine (VM) universe support included in this release that can allow VM universe jobs to read arbitrary files on the host machine. VM universe is disabled by default, so this potential vulnerability does not affect most sites. However, VM universe should not be enabled unless the job policy only allows users that you trust completely to run jobs on the machine.

• The condor_ startd will crash if ENABLE_BACKFILL is set to True. This was also the case in 6.9.4.

• The pool password authentication method fails to authenticate (and in fact will cause the client to crash).

• If condor_ dagman cannot execute a PRE or POST script (for example, if the script name is specified incorrectly), condor_ dagman will hang indefinitely. (Note that if the script is executed and fails, condor_ dagman deals with this correctly.) (This bug was fixed in version 6.8.7.)

Version 6.9.4

Release Notes:

• The default in standard universe for copy_to_spool is now true. In 6.9.3, it was changed to false for all universes for performance reasons, but this is deemed too risky for standard universe, because any modification of the executable is likely to make it impossible to resume execution using checkpoint files made from the original version of the executable.

• Version 1.5.0 of the Generic Connection Broker (GCB) is now used for building Condor. This version of GCB fixes a few critical bugs.
  • GCB was unable to pass information about sockets registered at a GCB broker to child processes due to a bug in the way a special environment variable was being set.
  • All sockets for outbound connections were being registered at the GCB broker, which was putting severe strain on the GCB broker even under relatively low load. Now, only sockets that are listening for inbound connections are registered at the broker.
  • The USE_CLONE_TO_CREATE_PROCESSES setting was causing havoc for applications linked with GCB. This configuration setting is now always disabled if GCB is enabled.
  • Fixed a race condition in GCB_connect() that would frequently cause connect() attempts to fail, especially non-blocking connections.
  • Fixed bugs in GCB_select() when GCB changes the direction of a connection from active to passive (for example, so that a Condor daemon running behind a firewall will use an outbound connection to communicate with a public client that had attempted to initiate contact via the GCB broker).
  • Also improved logging at the GCB broker.

  Additionally, there was a bug in how Condor was publishing the classified ads for GCB-enabled daemons. Condor used to be re-writing any attributes containing an IP address when a classified ad was sent over a network connection (in an effort to provide correct behavior for multi-homed machines). Now, this re-writing is disabled whenever GCB is enabled, since GCB already has logic to determine the correct IP addresses to advertise.

  For more information about GCB, see section 3.7.3 on page .

• The owner of the log file for the condor_ gridmanager has changed to the condor user. In Condor 6.9.3 and previous versions, it was owned by the user submitting the job. Therefore, the owner of and permissions on an existing log file are likely to be incorrect. Condor issues an error if the condor_ gridmanager is unable to read and write the existing file. To correct the problem, an administrator may modify file permissions such that the condor user may read and write the log file. Alternatively, an administrator may delete the file, and Condor will create a new file with the expected owner and permissions. In addition, the definition for GRIDMANAGER_LOG in the condor_config.generic file has changed for Condor 6.9.4.

New Features:

• Condor has been ported to Yellow Dog 5.0 Linux on the PPC architecture. This port of Condor will also run on the Sony Playstation 3 running said distribution of Linux.

• Enhanced the standard universe to utilize Condor's privilege separation mechanism.

• Implemented a completely new version of Quill. Quill can now record information about all the daemons into a relational database. See section 3.11 for details on Quill.

• Jobs in the mpi universe now can have $$ expanded in their ads in the same way as other universes.

• Added the vm universe, to facilitate running jobs under Xen or VMware virtual machines.

• Added the -subsystem command to condor_ status that queries all ClassAds of a given type.

• Improved the speed at which the condor_ schedd writes to its database file job_queue.log and the job history file. In benchmark tests, this roughly doubles the maximum throughput rate to approximately 20 jobs per second, although actual performance depends on the specific hardware used.

• The condor_ startd now records historical statistics about the total time (in seconds) that it spends in every state/activity pair. If a given slot spent more than 0 seconds in any of the possible pairs, the specifically-named ClassAd attribute for that pair is defined in the slot's ClassAd. The list of possible new machine attributes (alphabetically):

  TotalTimeBackfillBusy
  TotalTimeBackfillIdle
  TotalTimeBackfillKilling
  TotalTimeClaimedBusy
  TotalTimeClaimedIdle
  TotalTimeClaimedRetiring
  TotalTimeClaimedSuspended
  TotalTimeMatchedIdle
  TotalTimeOwnerIdle
  TotalTimePreemptingKilling
  TotalTimePreemptingVacating
  TotalTimeUnclaimedBenchmarking
  TotalTimeUnclaimedIdle
  

• The condor_ shadow now waits and retries after failing to commit the final update to the job ClassAd in the condor_ schedd's job queue, rather than immediately aborting and causing the job to be requeued to run again. See page  for the related configuration options.

• If the condor_ starter fails with a core dump on Unix, the core dump file is now put in the LOG directory. Previously, it was deleted by the condor_ startd.

• Added a small amount of randomization to the default values of PERIODIC_CHECKPOINT (in the example config file) and PASSWD_CACHE_REFRESH (in Condor's internal default) in order to decreases the chances of synchronized timing across many processes causing overloading of servers.

• Added the new submit command cron_window. It is an alias to deferral_window.

• Optimized the submission of grid-type gt4 grid universe jobs to the remote resource.Submission now takes one operation instead of three.

• Added new functionality for multi-homed machines (those with multiple network interfaces) to allow Condor to handle private networks in some cases without having to use the Generic Connection Broker (GCB). See the entries below that describe the new PRIVATE_NETWORK_NAME and PRIVATE_NETWORK_INTERFACE configuration variables.

Configuration Variable Additions and Changes:

• Added SLOTx_EXECUTE. This allows the execute directory to be configured independently for each batch slot. You could use this, for example, to have jobs on a multi-CPU machine using scratch space on different disks so that there is less chance of them interfering with each other. See page  for more details.

• The semantics of SLOT_TYPE_<N> has changed slightly. Previously, any resource shares left undefined would default to a fractional share equal to 1/NUM_CPUS. Now, the default is auto, which causes all remaining resources to be evenly divided. This is more convenient in cases where some slots are configured to take more or less than their ``fair'' share and the rest are desired to evenly split the remainder. The underlying reason for this change was to be able to better support the specification of disk partition shares in all the possible cases: The auto share takes into account how many slots are sharing the same disk partition.

• When set to True, the new configuration variable LOGS_USE_TIMESTAMP will cause Condor to print all daemon log messages using a Unix timestamp instead of a formatted date string. This feature is useful for debugging Condor Glideins that may be executing in different timezones. It should be noted that this does not affect job user logs. The default is False.

• The existing configuration variable LOG_ON_NFS_IS_ERROR has changed behavior. When set to False, condor_ submit does not emit a warning about user logs files being on NFS.

• The existing configuration variables DAEMON_LIST , DC_DAEMON_LIST , and MASTER_HA_LIST have changed behavior. Trailing commas are now ignored. Previously, trailing commas could cause the condor_ master to misbehave, including exiting with an error.

• The <SUBSYS>_DAEMON_AD_FILE was defined for the condor_ schedd. This setting was first made available in Condor 6.9.1 but was not used for any daemon. It appears in the configuration file as SCHEDD_DAEMON_AD_FILE and is set to the file .schedd_classad in the LOG directory. This setting is not necessary unless you are using the Quill functionality, and pools may upgrade to 6.9.4 without setting it if they are not using Quill.

• Added new configuration variables DBMSD , DBMSD_ARGS , and DBMSD_LOG , which define the location of the condor_ dbmsd daemon, the command line arguments to that daemon, and the location of the daemon's log. Default values are

  DBMSD = $(SBIN)/condor_dbmsd
  DBMSD_ARGS = -f
  DBMSD_LOG = $(LOG)/DbmsdLog
  

  These configuration variables are only necessary when using Quill, and then must be defined on only one machine in the Condor pool.

• Added new configuration variables PRIVATE_NETWORK_NAME and PRIVATE_NETWORK_INTERFACE , which allow Condor daemons to function more properly on multi-homed machines and in certain network configurations that involve private networks. There are no default values, both must be explicitly defined to have any effect. See section 3.3.6 on page  for more information about these two new settings.

• Added new configuration variables EVENT_LOG , MAX_EVENT_LOG , EVENT_LOG_USE_XML , and EVENT_LOG_JOB_AD_INFORMATION_ATTRS to specify the new event log which logs job user log events, but across all users. See section 3.3.4 on page  for definitions of these configuration variables.

Bugs Fixed:

• Trailing commas in lists of items in submit files and configuration files are now ignored. Previously, Condor would treat trailing commas in various surprising ways.

• Numerous bugs in GCB and the interaction between Condor and GCB. See the release notes above for details.

• The submit file entry ``coresize'' was not being honored properly on many universe. It is now honored on all universes except pvm and the grid universes (except where the grid type is Condor). For the java universe, it controls the core file size for the JVM itself.

• The condor_ configure installation script now allows Condor to be installed on hosts without a fully-qualified domain name.

• Fixed a bug in condor_ dagman: if a DAG run with a per-DAG configuration file specification generated a rescue DAG, the rescue DAG file did not contain the appropriate DAG configuration file line. (This bug was introduced when the per-DAG configuration file option was added in version 6.9.2.)

• Fixed a bug introduced in 6.9.3 when handling local universe jobs. The starter ignored failures in contacting the condor_ schedd in the final update to the job queue.

• When the condor_ schedd is issued a graceful shutdown command, any jobs that running with a job lease are allowed to keep running. When the condor_ schedd starts back up at a later time, it will spawn condor_ shadow to reconnect to the jobs if they are still executing. This mimics the same behavior as a fast shutdown. This also fixes a bug in 6.9.3 in which the condor_ schedd would fail to reconnect to jobs that were left running during a graceful shutdown.

• When the condor_ starter is gracefully shutting down and if it has become disconnected from the condor_ shadow, it will wait for the job lease time to expire before giving up on telling the condor_ shadow that the job was evicted. Previously, the condor_ starter would exit as soon as it was done evicting the job.

• Job ad attribute HoldReasonCode is now properly set when condor_ hold is called and when jobs are submitted on hold.

• If a job specified a job lease duration, and the condor_ schedd was killed or crashed, the condor_ shadow used to notice when the condor_ schedd was gone, and gracefully shutdown the job (evicting the job at the remote site). Now, the condor_ shadow honors the job lease duration, and if the lease has not yet expired, it simply exists without evicting the job, in the hopes that the condor_ schedd will be restarted in time to reconnect to the still-running job and resume computation.

• Fixed a bug from 6.9.3 in which condor_ q -format no longer worked when given an expression (as opposed to simple attribute reference). The expression was always treated as being undefined.

• When a condor daemon such as the condor_ schedd or condor_ negotiator tried to establish many new security sessions for UDP messages in a short span of time, it was possible for the daemon to run out of file descriptors, causing it to abort execution and be restarted by the condor_ master. A problem was found and fixed in the mechanism that protects against this.

• Improved error descriptions when Condor-C encounters failures when sending input files to the remote schedd.

• Rare failure conditions during stage in would cause Condor-C to put the state of the job in the remote schedd into an invalid state in which it would run but later fail during stage out. This now results in the job on the submit side going on hold with a staging failure.

• Fixed a bug which could cause condor_ store_cred to crash during common use.

• Fixed a bug where the vanilla universe condor_ starter would possibly crash when running a job not as the owner of the job.

• Fixed a bug which would cause a condor_ starter being used for the local universe to core dump.

• Fixed a bug which caused the condor_ schedd to core dump while processing a job's crontab entries in the submit description file.

• Fixed a privilege separation bug in the standard universe condor_ starter.

Known Bugs:

• standard universe jobs do not work when writing binary data. The behavior exhibited in this case may include the job crashing, or corrupt binary written.

• grid universe jobs for the gt4 grid type do not work, if Condor daemons are started as root and there is file transfer associated with or specified by the job. These jobs are placed on hold.

• The STARTD_RESOURCE_PREFIX setting on Windows results in broken behavior on both Condor 6.9.3 and 6.9.4. Specifically, when this setting is given a value other than its default (``slot''), all jobs will run using the ``condor-reuse-slot1'' user account, regardless of the actual slot used for execution.

Additions and Changes to the Manual:

• New documentation for the new vm universe in the User's Manual, section 2.12. Definitions of configuration variables for the vm universe are in section 3.3.27.

• New RDBMS schema tables added for Quill in section 3.11.4.

• ClassAd attribute definitions reside in a new appendix. In addition to machine and job attributes, DaemonMaster and Scheduler attributes are included.

Version 6.9.3

Release Notes:

• As of version 6.9.3, the entire Condor system has undergone a major terminology change. For almost 10 years, Condor has used the term virtual machine or vm to refer to each distinct resource that could run a Condor job (for example, each of the CPUs on an SMP machine). Back when we chose this terminology, it made sense, since each of these resource was like an independent machine in a pool, with its own state, ClassAd, claims, and so on. However, in recent years, the term virtual machine is now almost universally associated with the kinds of virtual machines created using tools such as VMware and Xen. Entire operating systems run inside a given process, usually emulating the underlying hardware on a host machine. So, to avoid confusion with these other kinds of virtual machines, the old virtual machine terminology has been replaced by the term slot.

  Numerous configuration settings, command-line arguments to Condor tools, ClassAd attribute names, and so on, have all been modified to reflect the new slot terminology. In general, the old settings and options will still work, but are now retired and may disappear in the future.

• The condor_ install installation script has been removed. All sites should use condor_ configure when setting up a new Condor installation.

• The SECONDARY_COLLECTOR_LIST configuration variable has been removed. Sites relying on this variable should instead use the configuration variable COLLECTOR_HOST . It may be used to define a list of condor_ collector daemon hosts.

• Cleaned up and improved help information for condor_ history.

New Features:

• Numerous scalability and performance improvements. Given enough memory, the schedd can now handle much larger job queues (e.g. 10s of thousands) without the severe degradation in performance that used to be the case.

• Added the START_LOCAL_UNIVERSE and START_SCHEDULER_UNIVERSE parameters for the condor_ schedd. This allows administrators to control whether a Local/Scheduler universe job will be started. This expression is evaluated against the job's ClassAd before the Requirements expression.

• All Local and Scheduler universe jobs now have their Requirements expressions evaluated before execution. If the expression evaluates to false, the job will not be allowed to begin running. In previous versions of Condor, Local and Scheduler universe jobs could begin execution without the condor_ schedd checking the validity of the Requirements.

• Added SCHEDD_INTERVAL_TIMESLICE and PERIODIC_EXPR_TIMESLICE. These indicate the maximum fraction of time that the schedd will spend on the respective activities. Previously, these activities were done on a fixed interval, so with very large job queue sizes, the fraction of time spent was increasing to unreasonable levels.

• Under Intel Linux, added USE_CLONE_TO_CREATE_PROCESSES . This defaults to true and results in scalability improvements for processes using large amounts of memory (e.g. a schedd with a lot of jobs in the queue).

• Jobs in the parallel universe now can have $$ expanded in their ads in the same way as other universes.

• Local universe jobs now support policy expression evaluation, which includes the ON_EXIT_REMOVE, ON_EXIT_HOLD, PERIODIC_REMOVE, PERIODIC_HOLD, and PERIODIC_RELEASE attributes. The periodic expressions are evaluated at intervals determined by the PERIODIC_EXPR_INTERVAL configuration macro.

• Jobs can be scheduled to executed periodically, similar to the crontab functionality found in Unix systems. The condor_ schedd calculates the next runtime for a job based on the new CRON_MINUTE, CRON_HOUR, CRON_DAY_OF_MONTH, CRON_MONTH, and CRON_DAY_OF_WEEK attributes. A preparation time defined by the CRON_PREP_TIME attribute allows a job to be submitted to the execution machine before the actual time the job is to begin execution. Jobs that would like to be run repeatedly will need to define the the ON_EXIT_REMOVE attribute properly so that they are re-queued after executing each time.

• Condor now looks for its configuration file in /usr/local/etc if the CONDOR_CONFIG environment variable is not set and there is no condor_config file located in /etc/condor. This allows a default Condor installation to be more compatible with Free BSD.

• If a user job requests streaming input or output in the submit file, the job can now run with job leases and the job will continue to run for the lease duration should the submit machine crash. Previously, jobs with streaming i/o would be evicted if the submit machine crashed. While the submit machine is down, if the job tried to issue a streaming read or write, the job will block until the submit machine returns or the job lease expires.

• Ever since version 6.7.19, condor_ submit has added a default job lease duration of 20 minutes to all jobs that support these leases. However, there was no way to disable this functionality if a user did not want job lease semantics. Now, a user can place job_lease_duration = 0 in their submit file to manually disable the job lease.

• Added new configuration knob STARTER_UPLOAD_TIMEOUT which sets the timeout for the starter to upload output files to the shadow on job exit. The default value is 200 seconds, which should be sufficient for serial jobs. For parallel jobs, this may need to be increased if many large output files are sent back to the shadow on job exit.

• condor_ dagman now aborts the DAG on ``scary'' submit events. These are submit events in which the Condor ID of the event does not match the expected value. Previously, condor_ dagman printed a warning, but continued. To restore Condor to the previous behavior, set the new DAGMAN_ABORT_ON_SCARY_SUBMIT configuration variable to False.

• When the condor_ master detects that its GCB broker is unavailable and there is a list of alternative brokers, it will restart immediately if MASTER_WAITS_FOR_GCB_BROKER is set to False instead of waiting for another broker to became available. condor_ glidein now sets MASTER_WAITS_FOR_GCB_BROKER to False in its configuration file.

• When using GCB and a list of brokers is available, the condor_ master will now pick a random broker rather than the least-loaded one.

• All Condor daemons now evaluate some ClassAd expressions whenever they are about to publish an update to the condor_ collector. Currently, the two supported expressions are:

  DAEMON_SHUTDOWN

  If True, the daemon will gracefully shut itself down and will not be restarted by the condor_ master (as if it sent itself a condor_ off command).

  DAEMON_SHUTDOWN_FAST

  If True, the daemon will quickly shut itself down and will not be restarted by the condor_ master (as if it sent itself a condor_ off command using the -fast option).

  For more information about these expressions, see section 3.3.5 on page .

• When the condor_ master sends email announcing that another daemon has died, exited, or been killed, it now notes the name of the machine, the daemon's name, and a summary of the situation in the Subject line.

• Anyplace in a Condor configuration or submit description file where wild cards may be used, you can now place wild cards at both the beginning and end of the string pattern (i.e. match strings that contain the text between the wild cards anywhere in the string). Previously, only one wild card could appear in the string pattern.

• Added optional configuration setting NEGOTIATOR_MATCH_EXPRS . This allows the negotiator to insert expressions into the matched ClassAd. See page  for more information.

• Increased speed of ClassAd parsing.

• Added DEDICATED_EXECUTE_ACCOUNT_REGEXP and deprecated the boolean setting EXECUTE_LOGIN_IS_DEDICATED , because the latter could not handle a policy where some jobs run as the job owner and some run as dedicated execution accounts. Also added support for STARTER_ALLOW_RUNAS_OWNER under Unix. See Section 3.3.7 and Section 3.6.12 for more information.

• All Condor daemons now publish a MyCurrentTime attribute which is the current local time at the time the update was generated and sent to the condor_ collector. This is in addition to the LastHeardFrom attribute which is inserted by the condor_ collector (the current local time at the collector when the update is received).

• condor_ history now accepts partial command line arguments. For example, -constraint can be abbreviated -const. This brings condor_ history in line with other Condor command line tools.

• condor_ history can now emit ClassAds formatted as XML with the new -xml option. This brings condor_ history more in line condor_ q.

• The $$ substitution macro syntax now supports the insertion of literal $$ characters through the use of $$(DOLLARDOLLAR). Also, $$ expansion is no longer recursive, so if the value being substituted in place of a $$ macro itself contains $$ characters, these are no longer interpreted as substitution macros but are instead inserted literally.

• When started as root on a Linux 64-bit x86 machine, Condor daemons will now leave core files in the log directory when they crash. This matches Condor's behavior on most other Unix-like operating systems, including 32-bit x86 versions of Linux.

• The _CONDOR_SLOT variable is now placed into the environment for jobs of all universes. This variable indicates what slot a given job is running on, and will have the same value as the SlotID from the machine classified ad where the job is running. The _CONDOR_SLOT variable replaces the deprecated CONDOR_VM environment variable, which was only defined for standard universe jobs.

• Added a USE_PROCD configuration parameter. If this parameter is set to true for a given daemon, the daemon will use the condor_ procd program to monitor process families. If set to false, the daemon will execute process family monitoring logic on its own. The condor_ procd is more scalable and is also an essential piece in the ongoing privilege separation effort. The disadvantage of using the ProcD is that it is newer, less-hardened code.

Configuration Variable Additions and Changes:

• The SECONDARY_COLLECTOR_LIST configuration variable has been removed. Sites relying on this variable should instead use the configuration variable COLLECTOR_HOST to define a list of condor_ collector daemon hosts.

• Added new configuration variables START_LOCAL_UNIVERSE and START_SCHEDULER_UNIVERSE for the condor_ schedd daemon. These boolean expressions default to True. START_LOCAL_UNIVERSE is relevant only to local universe jobs. START_SCHEDULER_UNIVERSE is relevant only to scheduler universe jobs. These new variables allow an administrator to define a START expression specific to these jobs. The expression is evaluated against the job's ClassAd before the Requirements expression.

• Added new configuration variables SCHEDD_INTERVAL_TIMESLICE and PERIODIC_EXPR_TIMESLICE . These configuration variables address a scalability issue for very large job queues. Previously, the condor_ schedd daemon handled an activity related to counting jobs, as well as the activity related to evaluating periodic expressions for jobs at the fixed time interval of 5 minutes. With large job queues, the fraction of the condor_ schedd daemon execution time devoted to these two activities became excessive, such that it could be doing little else. The fixed time interval is now gone, and Condor calculates the amount of time spent on the two activities, using these new configuration variables to calculate an appropriate time interval.

  Each is a floating point value within the range (noninclusive) 0.0 to 1.0. Each determines the maximum fraction of the time interval that the condor_ schedd daemon will spend on the respective activity. SCHEDD_INTERVAL_TIMESLICE defaults to the value 0.05, such that the calculated time interval will be 20 * the amount of time spent on the counting jobs activity. PERIODIC_EXPR_TIMESLICE defaults to the value 0.01, such that the calculated time interval will be 100 * the amount of time spent on the periodic expression evaluation activity.

• Added new configuration variable USE_CLONE_TO_CREATE_PROCESSES , relevant only to the Intel Linux platform. This boolean value defaults to True, and it results in scalability improvements for Condor processes using large amounts of memory. These processes may clone themselves instead of forking themselves. An example of the improvement occurs for a condor_ schedd daemon with a lot of jobs in the queue.

• Added new configuration variable STARTER_UPLOAD_TIMEOUT , which allows a configurable time (in seconds) for a timeout used by the condor_ starter. The default value of 200 seconds replaces the previously hard coded value of 20 seconds. This timeout before job failure is to upload output files to the condor_ shadow upon job exit. The default value should be sufficient for serial jobs. For parallel jobs, it may need to be increased if there are many large output files.

• Added new configuration variable DAGMAN_ABORT_ON_SCARY_SUBMIT . This boolean variable defaults to True, and causes condor_ dagman to abort the DAG on ``scary'' submit events. These are submit events in which the Condor ID of the event does not match the expected value. Previously, condor_ dagman printed a warning, but continued. To restore Condor to the previous behavior, set DAGMAN_ABORT_ON_SCARY_SUBMIT to False.

• Added new configuration variable NEGOTIATOR_MATCH_EXPRS . It causes the condor_ negotiator to insert expressions into the matched ClassAd. See page  for details.

• Added new configuration variable DEDICATED_EXECUTE_ACCOUNT_REGEXP to replace the retired EXECUTE_LOGIN_IS_DEDICATED , because EXECUTE_LOGIN_IS_DEDICATED could not handle a policy where some jobs run as the job owner and others run as dedicated execution accounts. Also added support for the existing configuration variable STARTER_ALLOW_RUNAS_OWNER under Unix. See Section 3.3.7 and Section 3.6.12 for more information.

• Added new configuration variable USE_PROCD . This boolean variable defaults to False for the condor_ master, and True for all other daemons. When True, the daemon will use the condor_ procd program to monitor process families. When False, a daemon will execute process family monitoring logic on its own. The condor_ procd is more scalable and is also an essential piece in the ongoing privilege separation effort. The disadvantage of using the condor_ procd is that it is newer, less-hardened code.

Bugs Fixed:

• On Unix systems, Condor can now handle file descriptors larger than FD_SETSIZE when using the select system call. Previously, file descriptors larger than FD_SETSIZE would cause memory corruption and crashes.

• When an update to the condor_ collector from the condor_ startd is lost, it is possible for multiple claims to the same resource to be handed out by the condor_ negotiator. This is still true. What is fixed is that these multiple claims will not result in mutual annihilation of the various attempts to use the resource. Instead, the first claim to be successfully requested will proceed and the others will be rejected.

• condor_ glidein was setting PREEN_INTERVAL =0 in the default configuration, but this is no longer a legal value, as of 6.9.2.

• condor_ glidein was not setting necessary configuration parameters for condor_ procd in the default glidein configuration.

• In 6.9.2, Condor daemons crashed after failing to authenticate a network connection.

• condor_ status will now accurately report the ActvtyTime (activity time) value in Condor pools where not all machines are in the same timezone, or if there is clock-skew between the hosts.

• Fixed the known issue in Condor 6.9.2 where using the EXECUTE_LOGIN_IS_DEDICATED setting on UNIX platforms would cause the condor_ procd to crash.

• Failure when activating a COD claim no longer will result in an opportunistic job running on the same condor_ startd being left suspended. This problem was most likely to be seen when using the GLEXEC_STARTER feature.

• In Condor 6.9.2 for Tru64 UNIX, the condor_ master would immediately fail if started as root. This problem has been fixed.

• Condor 6.9.2 introduced a problem where the condor_ master would fail if started as root with the UID part of the CONDOR_IDS parameter set to 0 (root). This issue has been fixed.

Known Bugs:

• The 6.9.3 condor_ schedd daemon incorrectly handles jobs with leases (true by default for vanilla, java, and parallel universe jobs) when shutting down gracefully. These jobs are allowed to continue running, but when the condor_ schedd daemon is started back up, it fails to reconnect to them. The result is that the orphaned jobs are left running for the duration of the job's lease time (a default time of 20 minutes). The state of the jobs in the restarted queue is independent of any orphaned running jobs, so these queued jobs may begin running on another machine while orphans are still running.

• condor_ q -format in 6.9.3 does not work with expressions. It behaves as if the expression evaluates to an undefined result.

Version 6.9.2

Release Notes:

• As part of ongoing security enhancements, Condor now has a new, required daemon: condor_ procd. This daemon is automatically started by the condor_ master, you do not need to add it to DAEMON_LIST . However, you must be certain to update the condor_ master if you update any of the other Condor daemons.

• Some configuration settings that previously accepted 0 no longer do so. Instead the daemon using the setting will exit with an error message listing the acceptable range to its log. For these settings 0 was equivalent to requesting the default. As this was undocumented and confusing behavior it is no longer present. To request a setting use its default, either comment it out, or set it to nothing (``EXAMPLE_SETTING=''). Settings impacted include but are not limited to: MASTER_BACKOFF_CONSTANT , MASTER_BACKOFF_CEILING , MASTER_RECOVER_FACTOR , MASTER_UPDATE_INTERVAL , MASTER_NEW_BINARY_DELAY , PREEN_INTERVAL , SHUTDOWN_FAST_TIMEOUT , SHUTDOWN_GRACEFUL_TIMEOUT , MASTER_<name>_BACKOFF_CONSTANT , MASTER_<name>_BACKOFF_CEILING ,

• Version 1.4.1 of the Generic Connection Broker (GCB) is now used for building Condor. This version of GCB fixes a timing bug where a client may incorrectly think a network connection has been established, and also guards against an unresponsive client from causing a denial of service by the broker. For more information about GCB, see section 3.7.3 on page .

New Features:

• On UNIX, an execute-side Condor installation can run without root privileges and still execute jobs as different users, properly clean up when a job exits, and correctly enforce policies specified by the Condor administrator and resource owners. Privileged functionality has been separated into a well-defined set of functions provided by a setuid helper program. This feature currently does not work for the standard or PVM universes.

• Added support for EmailAttributes in the parallel universe. Previously, it was only valid in the vanilla and standard universes.

• Added configuration parameter DEDICATED_SCHEDULER_USE_FIFO which defaults to true. When false, the dedicated scheduler will use a best-fit algorithm to schedule parallel jobs. This setting is not recommended, as it can cause starvation. When true, the dedicated scheduler will schedule jobs in a first-in, first-out manner.

• Added -dump to condor_ config_val which will print out all of the macros defined in any of the configuration files found by the program. condor_ config_val -dump -v will augment the output with exactly what line and in what file each configuration variable was found. NOTE: : The output format of the -dump option will most likely change in a future revision of Condor.

• Node names in condor_ dagman DAG files can now be DAG keywords, except for PARENT and CHILD.

• Improved the log message when OnExitRemove or OnExitHold evaluates to UNDEFINED.

• Added the DAGMAN_ON_EXIT_REMOVE configuration macro, which allows customization of the OnExitRemove expression generated by condor_ submit_dag.

• When using GCB, Condor can now be told to choose from a list of brokers. NET_REMAP_INAGENT is now a space and comma separated list of brokers. On start up, the condor_ master will query all of the brokers and pick the least-used one for it and its children to use. If none of the brokers are operational, then the condor_ master will wait until one is working. This waiting can be disabled by setting MASTER_WAITS_FOR_GCB_BROKER to FALSE in the configuration file. If the chosen broker fails and recovery is not possible or another broker is available, the condor_ master will restart all of the daemons.

• When using GCB, communications between parent and child Condor daemons on the same host no longer use the GCB broker. This improves scalability and also allows a single host to continue functioning if the GCB broker is unavailable.

• The condor_ schedd now uses non-blocking methods to send the ``alive'' message to the condor_ startd when renewing the job lease. This prevents the condor_ schedd from blocking for 20 seconds while trying to connect to a machine that has become disconnected from the network.

• condor_ advertise can read the classad to be advertised from standard input.

• Unix Condor daemons now reinitialize their DNS configuration (e.g. IP addresses of the name servers) on reconfig.

• A configuration file for condor_ dagman can now be specified in a DAG file or on the condor_ submit_dag command line.

• Added condor_ cod option -lease for creation of COD claims with a limited duration lease. This provides automatic cleanup of COD claims that are not renewed by the user. The default lease is infinitely long, so existing behavior is unchanged unless -lease is explicitly specified.

• Added condor_ cod command delegate_proxy which will delegate an x509 proxy to the requested COD claim. This is primarily useful for sites wishing to use glexec to spawn the condor_ starter used for COD jobs. The new command optionally takes an -x509proxy argument to specify the proxy file. If this argument is not present, condor_ cod will search for the proxy using the same logic as condor_ submit does.

• STARTD_DEBUG can now be empty, indicating a default, minimal log level. It now defaults to empty. Previously it had to be non-empty and defaulted to include D_COMMAND.

• The addition of the condor_ procd daemon means that all process family monitoring and control logic is no longer replicated in each Condor daemon that needs it. This improves Condor's scalability, particularly on machines with many processes.

Bugs Fixed:

• Under various circumstances, condor 6.9.1 daemons would abort with the message, ``ERROR: Unexpected pending status for fake message delivery.'' A specific example is when OnExitRemove or OnExitHold evaluated to UNDEFINED. This caused the condor_ schedd to abort.

• In Condor 6.9.1, the condor_ schedd would die during startup when trying to reconnect to running jobs for which the condor_ schedd could not find a startd ClassAd. This would happen shortly after logging the following message: ``Could not find machine ClassAds for one or more jobs. May be flocking, or machine may be down. Attempting to reconnect anyway.''

• Improved Condor's validity checking of configuration values. For example, in some cases where Condor was expecting an integer but was given an expression such as 12*60, it would silently interpret this as 12. Such cases now result in the condor daemon exiting after issuing an error message into the log file.

• When sending a WM_CLOSE message to a process on Windows, Condor daemons now invoke the helper program condor_ softkill to do so. This prevents the daemon from needing to temporarily switch away from its dedicated service Window Station and Desktop. It also fixes a bug where daemons would leak Window Station and Desktop handles. This was mainly a problem in the condor_ schedd when running many scheduler universe jobs.

Known Bugs:

• condor_ glidein generates a default config file that sets PREEN_INTERVAL to an invalid value (0). To fix this, remove the setting of PREEN_INTERVAL.

• There are a couple of known issues with Condor's GLEXEC_STARTER feature when used in conjunction with COD. First, the condor_ cod tool invoked with the delegate_proxy option will sometimes incorrectly report that the operation has failed. In addition, the GLEXEC_STARTER feature will not work properly with COD unless the UID that the each COD job runs as is different than the UID of the opportunistic job or any other COD jobs that are running on the execute machine when the COD claim is activated.

• The EXECUTE_LOGIN_IS_DEDICATED feature has been found to be broken on UNIX platforms. Its use will cause the condor_ procd to crash, bringing down the other Condor daemons with it.

Version 6.9.1

Release Notes:

• The 6.9.1 release contains all of the bug fixes and enhancements from the 6.8.x series up to and including version 6.8.3.

• Version 1.4.0 of the Generic Connection Broker (GCB) library is now used for building Condor, and it is the 1.4.0 versions of the gcb_broker and gcb_relay_server programs that are included in this release. This version of GCB includes enhancements used by Condor along with a new GCB-related command-line tool: gcb_broker_query. Condor 6.9.1 will not work properly with older versions of the gcb_broker or gcb_relay_server. For more information about GCB, see section 3.7.3 on page .

New Features:

• Improved the performance of the ClassAd matching algorithm, which speeds up the condor_ schedd and other daemons.

• Improved the scalability of the algorithm used by the condor_ schedd daemon to find runnable jobs. This makes a noticeable difference in condor_ schedd daemon performance, when there are on the order of thousands of jobs in the queue.

• the D_ COMMAND debugging level has been enhanced to log many more messages.

• Updated the version of DRMAA, which contains several great improvements regarding scalability and race conditions.

• Added the DAGMAN_SUBMIT_DEPTH_FIRST configuration macro, which causes condor_ dagman to submit ready nodes in more-or-less depth-first order, if set to True. The default behavior is to submit the ready nodes in breadth-first order.

• Added configuration parameter USE_PROCESS_GROUPS . If it is set to False, then Condor daemons on Unix machines will not create new sessions or process groups. This is intended for use with Glidein, as we have had reports that some batch systems cannot properly track jobs that create new process groups. The default value is True.

• The default value for the submit file command copy_to_spool has been changed to False, because copying the executable to the spool directory for each job (or job cluster) is almost never desired. Previously, the default was True in all cases, except for grid universe jobs and remote submissions.

• More types of file transfer errors now result in the job going on hold, with a specific error message about what went wrong. The new cases involve failures to write output files to disk on the submit side (for example, when the disk is full). As always, the specific error number is recorded in HoldReasonSubCode, so you can enforce an automated error handling policy using periodic_release or periodic_remove.

• Added the <SUBSYS>_DAEMON_AD_FILE configuration variable, which is similar to the <SUBSYS>_ADDRESS_FILE . This new variable will be used in future versions of Condor, but is not necessary for 6.9.1.

Bugs Fixed:

• Fixed a bug in the condor_ master so that it will now send obituary e-mails when it kills child processes that it considers hung.

• condor_ configure used to always make a personal Condor with -install even when -type called for only execute or submit types. Now, condor_ configure honors the -type argument, even when using -install. If -type is not specified, the default is to still install a full personal Condor with the following daemons: condor_ master, condor_ collector, condor_ negotiator, condor_ schedd, condor_ startd.

• While removing, putting on hold, or vacating a large number of jobs, it was possible for the condor_ schedd and the condor_ shadow to temporarily deadlock with each other. This has been fixed under Unix, but not yet under Windows.

• Communication from a condor_ schedd to a condor_ startd now occurs in a nonblocking manner. This fixes the problem of the condor_ schedd blocking when the claimed machine running the condor_ startd cannot be reached, for example because the machine is turned off.

Known Bugs:

• Under various circumstances, condor 6.9.1 daemons abort with the message, ``ERROR: Unexpected pending status for fake message delivery.'' A specific example is when OnExitRemove or OnExitHold evaluated to UNDEFINED, which causes the condor_ schedd to abort.

• In Condor 6.9.1, the condor_ schedd will die during startup when trying to reconnect to running jobs for which the condor_ schedd can not find a startd ClassAd. This happens shortly after logging the following message: ``Could not find machine ClassAds for one or more jobs. May be flocking, or machine may be down. Attempting to reconnect anyway.''

Version 6.9.0

Release Notes:

• The 6.9.0 release contains all of the bug fixes and enhancements from the 6.8.x series up to and including version 6.8.2.

New Features:

• Preliminary support for using glexec on execute machines has been added. This feature causes the condor_ startd to spawn the condor_ starter as the user that glexec determines based on the user's GSI credential.

• A ``per-job history files'' feature has been added to the condor_ schedd. When enabled, this will cause the condor_ schedd to write out a copy of each job's ClassAd when it leaves the job queue. The directory to place these files in is determined by the parameter PER_JOB_HISTORY_DIR . It is the responsibility of whatever external entity (for example, an accounting or monitoring system) is using these files to remove them as it completes its processing.

• condor_ chirp command now supports writing messages to the user log.

• condor_ chirp getattr and putattr now send all classad getattr and putattr commands to the proc 0 classad, which allows multiple proc parallel jobs to use proc 0 as a scratch pad.

• Parallel jobs now support an AllRemoteHosts attribute, which lists all the hosts across all procs in a cluster.

• The DAGMAN_ABORT_DUPLICATES configuration macro (which causes condor_ dagman to abort itself if it detects another condor_ dagman running on the same DAG) now defaults to True instead of False.

Bugs Fixed:

• None.

Known Bugs:

• None.