A hook is an external program or script invoked by Condor at various points during the life cycle of a job. Instead of putting all the code and logic directly into the Condor daemons to handle the variety of external systems from which it might fetch work, sites can write their own programs or scripts and allow Condor to invoke these hooks at the right moments to accomplish the desired outcome. This eliminates the expense of the matchmaking and scheduling provided by the the condor_schedd and the condor_negotiator, although at the price of the flexibility they offer. Therefore, the hooks allow Condor to more easily and directly interface with external scheduling systems.
The following sections describe the system of hooks, both for work that is fetched from an external system, and for hooks that behave as a Job Router.
In the past, Condor has always sent work to the execute machines by pushing jobs to the condor_startd daemon, either from the condor_schedd daemon or via condor_cod. Beginning with the Condor 7.1.0, the condor_startd daemon now has the ability to pull work by fetching jobs via a system of plug-ins or hooks. Any site can configure a set of hooks to fetch work completely outside of the usual Condor matchmaking system.
A projected use of the hook mechanism implements what might be termed a glide-in factory, especially where the factory is behind a firewall. Without using the hook mechanism to fetch work, a glide-in condor_startd daemon behind a firewall depends on GCB to help it listen and eventually receive work pushed from elsewhere. With the hook mechanism, a glide-in condor_startd daemon behind a firewall uses the hook to pull work. The hook needs only an outbound network connection to complete its task, thereby being able to operate from behind the firewall, without the intervention of GCB.
Periodically, each execution slot managed by a condor_startd will invoke a hook to see if there is any work that can be fetched. Whenever this hook returns a valid job, the condor_startd will evaluate the current state of the slot and decide if it should start executing the fetched work. If the slot is unclaimed and the Start expression evaluates to TRUE, a new claim will be created for the fetched job. If the slot is claimed, the condor_startd will evaluate the Rank expression relative to the fetched job and compare it to the value of the Rank for the currently running job and decide if the existing job should be preempted due to the fetched job having a higher rank. If the slot is unavailable for whatever reason, the condor_startd will refuse the fetched job and ignore it. Either way, once the condor_startd decides what it should do with the fetched job, it will invoke another hook to reply to the attempt to fetch work, so that the external system knows what happened to that work unit.
If the job is accepted, a claim is created for it and the slot moves into the Claimed state. As soon as this happens, the condor_startd will spawn a condor_starter to manage the execution of the job. At this point, from the perspective of the condor_startd, this claim is just like any other. The usual policy expressions are evaluated, and if the job needs to be suspended or evicted, it will be. If a higher-ranked job being managed by a condor_schedd is matched with the slot, that job will preempt the fetched work.
The condor_starter itself can optionally invoke additional hooks to help manage the execution of the specific job. There are hooks to prepare the execution environment for the job, periodically update information about the job as it runs, notify when the job exits, and to take special actions when the job is being evicted.
Assuming there are no interruptions, the job completes, and the condor_starter exits, the condor_startd will invoke the hook to fetch work again. If another job is available, the existing claim will be reused and a new condor_starter is spawned. If the hook returns that there is no more work to perform, the claim will be evicted, and the slot will return to the Owner state.
There are a handful of hooks invoked by Condor related to fetching work, some of which are called by the condor_startd and others by the condor_starter. Each hook will be described below, including when it is invoked, what task it is supposed to accomplish, what data is passed to the hook, and what output (and, when relevant) exit status is expected.
The hook defined by the configuration variable
HOOK_FETCH_WORK is invoked whenever the condor_startd
wants to see if there is any work to fetch.
There is a related configuration expression called
FetchWorkDelay which determines how long the condor_startd
will wait between attempts to fetch work, which is described in detail
in within section 4.4.1 on
page ![[*]](crossref.png)
.
HOOK_FETCH_WORK is the most important hook in the whole
system, and is the only hook that must be defined for any of the other
condor_startd hooks to operate.
The job ClassAd returned by the hook needs to contain enough
information for the condor_starter to eventually spawn the work.
The required and optional attributes in this ClassAd are identical to
the ones described for Computing on Demand (COD) jobs in
section 4.3.3
called COD Application Attributes on
page .
The hook defined by the configuration variable HOOK_REPLY_FETCH is invoked whenever HOOK_FETCH_WORK returns data and the the condor_startd decides if it is going to accept the fetched job or not.
The condor_startd will not wait for this hook to return before taking other actions, and ignores all output. The hook is simply advisory, and has no impact on the behavior of the condor_startd.
accept or reject.
----- and a new line).
The hook defined by the configuration variable HOOK_EVICT_CLAIM is invoked whenever the condor_startd needs to evict a claim representing fetched work.
The condor_startd will not wait for this hook to return before taking other actions, and ignores all output. The hook is simply advisory, and has no impact on the behavior of the condor_startd.
----- and a new line).
The hook defined by the configuration variable HOOK_PREPARE_JOB is invoked by the condor_starter before a job is going to be run. This hook provides a chance to execute commands to setup the job environment, for example to transfer input files.
The condor_starter waits until this hook returns before attempting to execute the job. If the hook returns a non-zero exit status, the condor_starter will assume an error was reached while attempting to set up the job environment and abort the job.
The hook defined by the configuration variable HOOK_UPDATE_JOB_INFO is invoked periodically during the life of the job to update information about the status of the job. When the job is first spawned, the condor_starter will invoke this hook after STARTER_INITIAL_UPDATE_INTERVAL seconds (defaults to 8). Thereafter, the condor_starter will invoke the hook every STARTER_UPDATE_INTERVAL seconds (defaults to 300, in other words, every 5 minutes).
The condor_starter will not wait for this hook to return before taking other actions, and ignores all output. The hook is simply advisory, and has no impact on the behavior of the condor_starter.
The additional attributes included inside the job ClassAd are:
The hook defined by the configuration variable HOOK_JOB_EXIT is invoked whenever a job exits, either on its own or when being evicted from an execution slot.
The condor_starter will wait for this hook to return before taking any other actions. In the case of jobs that are being managed by a condor_shadow, this hook is invoked before the condor_starter does its own optional file transfer back to the submission machine, writes to the local user log file, or notifies the condor_shadow that the job has exited.
exit The job exited or died with a signal on its own.
remove The job was removed with condor_rm or the
user job policy expressions (for example, PeriodicRemove).
hold The job was held with condor_hold or the
user job policy expressions (for example, PeriodicHold).
evict The job was evicted from the execution slot for
any other reason (PREEMPT evaluated to TRUE in the
condor_startd, condor_vacate, condor_off, etc).
The job ClassAd passed to this hook contains all of the extra attributes described above for HOOK_UPDATE_JOB_INFO , and the following additional attributes that are only present once a job exits:
Hooks are defined in the Condor configuration files by prefixing the name of the hook with a keyword. This way, a given machine can have multiple sets of hooks, each set identified by a specific keyword.
Each slot on the machine can define a separate keyword for the set of hooks that should be used (SLOTN_JOB_HOOK_KEYWORD ). Note that the N in SLOTN should be replaced with the slot identification number. For example, on slot1, the setting would be called SLOT1_JOB_HOOK_KEYWORD. If the slot-specific keyword is not defined, the condor_startd will use a global keyword (STARTD_JOB_HOOK_KEYWORD ).
Once a job is fetched via HOOK_FETCH_WORK , the condor_startd will insert the keyword used to fetch that job into the job ClassAd as HookKeyword. This way, the same keyword will be used to select the hooks invoked by the condor_starter during the actual execution of the job. However, the STARTER_JOB_HOOK_KEYWORD can be defined to force the condor_starter to always use a given keyword for its own hooks, instead of looking the job ClassAd for a HookKeyword attribute.
For example, the following configuration defines two sets of hooks, and on a machine with 4 slots, 3 of the slots use the global keyword for running work from a database-driven system, and one of the slots uses a custom keyword to handle work fetched from a web service.
# Most slots fetch and run work from the database system. STARTD_JOB_HOOK_KEYWORD = DATABASE # Slot4 fetches and runs work from a web service. SLOT4_JOB_HOOK_KEYWORD = WEB # The database system needs to both provide work and know the reply # for each attempted claim. DATABASE_HOOK_DIR = /usr/local/condor/fetch/database DATABASE_HOOK_FETCH_WORK = $(DATABASE_HOOK_DIR)/fetch_work.php DATABASE_HOOK_REPLY_FETCH = $(DATABASE_HOOK_DIR)/reply_fetch.php # The web system only needs to fetch work. WEB_HOOK_DIR = /usr/local/condor/fetch/web WEB_HOOK_FETCH_WORK = $(WEB_HOOK_DIR)/fetch_work.php
The keywords "DATABASE" and "WEB" are completely arbitrary, so each site is encouraged to use different (more specific) names as appropriate for their own needs.
There are two events that trigger the condor_startd to attempt to fetch new work:
Even if a given compute slot is already busy running other work, it is possible that if it fetched new work, the condor_startd would prefer this newly fetched work (via the Rank expression) over the work it is currently running. However, the condor_startd frequently evaluates its own state, especially when a slot is claimed. Therefore, administrators can define a configuration variable which controls how long the condor_startd will wait between attempts to fetch new work. This variable is called FetchWorkDelay .
The FetchWorkDelay expression must evaluate to an integer, which defines the number of seconds since the last fetch attempt completed before the condor_startd will attempt to fetch more work. However, as a ClassAd expression (evaluated in the context of the ClassAd of the slot considering if it should fetch more work, and the ClassAd of the currently running job, if any), the length of the delay can be based on the current state the slot and even the currently running job.
For example, a very common configuration would be to always wait 5 minutes (300 seconds) between attempts to fetch work, unless the slot is Claimed/Idle, in which case the condor_startd should fetch immediately:
FetchWorkDelay = ifThenElse(State == "Claimed" && Activity == "Idle", 0, 300)
If the condor_startd wants to fetch work, but the time since the last attempted fetch is shorter than the current value of the delay expression, the condor_startd will set a timer to fetch as soon as the delay expires.
If this expression is not defined, the condor_startd will default to a five minute (300 second) delay between all attempts to fetch work.
A common problem the Java Universe does not solve is when there are multiple Java virtual machines available to run user jobs. If the user knows which jvm their job needs, the administrator can use the Job hooks feature to enable the user to select one java vm. The main idea is to use the prepare job hook to overwrite the Cmd attribute of the job ad. This attribute is the name of the executable the starter will use to run the job. When the job hook overwrites Cmd, it effectively picks which executable will run. So, it is easy to put the execute machine in charge of selecting the specific java install. To do this, define a job hook on the execute machine by adding the following line to the config file.
JAVA5_HOOK_PREPARE_JOB=$(LIBEXEC)/java5_prepare_hook
With this line, if a user job asks for the JAVA5 job hook, by setting
+HookKeyword="JAVA"
in the condor_submit file, the starter will run the program java5_prepare_hook before running this job. Note that the double quote characters are required. Any output from this program will be considered updates to the job ad. So, a simple java5_prepare_hook script might look like this:
#!/bin/sh # Read and discard the job classad cat > /dev/null echo 'Cmd = "/usr/java/java5/bin/java"'
A complete job submit file in this case might look like the following:
universe = vanilla executable = /usr/bin/java arguments = Hello should_transfer_files = always when_to_transfer_output = on_exit transfer_input_files = Hello.class output = o error = e log = log +HookKeyword="JAVA5" queue
Job Router Hooks allow for an alternate transformation and/or monitoring engine for the Job Router. Routing is still controlled by the Job Router, but if the Job Router Hooks are configured, then these hooks will be used to transform and monitor the job instead of Condor's internal engine.
Job Router Hooks are similar in concept to Job Fetch Hooks, but they are limited in their scope to just the Job Router. As with Job Fetch Hooks, a hook is an external program or script invoked by Condor's Job Router at various points during the life cycle of a routed job.
The following sections describe how these hooks are used by the Job Router, what hooks are invoked by Condor at various stages of the job's life, and how to configure Condor to use Job Router Hooks.
The Job Router Hooks allow for replacement of the transformation engine used by Condor for routing a job. Since the external transformation engine is not controlled by Condor, additional hooks provide a means to update the job's status in Condor and clean up upon exit or failure cases. This allows a job to be transformed to just about any other type of job that Condor supports, as well as to use execution nodes not normally available to Condor.
It is important to note that if the Job Router Hooks are configured, then Condor will not ignore a failure in any hook execution. If a hook is configured, then Condor assumes it is required to execute and will not continue by falling back to a part of its internal engine. For example, if there is a problem transforming the job using the hooks, Condor will not fall back on its internal transformation engine to process the job.
There are 4 hooks that the Job Router can be configured to use. Each hook will be described below along with data passed to the hook and expected output. Again, all hooks must exit successfully.
The hook defined by the configuration variable HOOK_TRANSLATE is invoked when the Job Router has determined that a job meets the definition for a route. This hook is responsible for doing the transformation of the job and configuring any resources that are external to Condor if applicable.
"------" and a new line.
The hook defined by the configuration variable HOOK_UPDATE_JOB_INFO is invoked to provide status on the specified routed job when the Job Router polls the status of routed jobs at intervals set by JOB_ROUTER_POLLING_PERIOD .
The hook defined by the configuration variable HOOK_JOB_FINALIZE is invoked when the Job Router has found that the job has completed. Any output from the hook is treated as an update to the source job.
"------" and a new line.
The hook defined by the configuration variable HOOK_JOB_CLEANUP is invoked when the Job Router finishes managing the job. This hook will be invoked regardless of whether the job completes successfully or not, and must exit successfully.