LIGO Support Ticket 18376
Ticket Information
Number: admin 18376
User: stephen.fairhurst@astro.cf.ac.uk
Email:
Status: resolved
Assigned To: psilord
From: Stephen Fairhurst <stephen.fairhurst__AT__astro.cf.ac.uk>
To: condor-admin__AT__cs.wisc.edu, psilord__AT__cs.wisc.edu
Subject: LIGO: Use of DIR in DAG SPLICE
Date: Tue, 26 Aug 2008 11:53:52 +0100
X-Seen-BY: mailfromd 4.1 obsidian.cs.wisc.edu
--Apple-Mail-6-922664252
The DAG SPLICE capacity does not seem to be compatible with the DIR
command.
As an example, consider the very simple dag:
subdag.dag
JOB A job.sub
which is located in a directory subdag.
I want to include this into a larger dag (dag.dag). I know that it is
possible by:
1) running condor_submit_dag -no_submit subdag.dag
2) submitting the following dag:
dag.dag
JOB D subdag.dag.condor.sub DIR subdag
I have been unable to replicate this procedure using the SPLICE
command. In particular,
A) writing a dag:
dag2.dag
SPLICE D subdag.dag
Unsurprisingly fails with
ERROR: Failed to locate Condor job log files: Splice[D:subdag.dag]:
Unable to read file: subdag.dag; Failed to locate Stork job log files:
Splice[D:subdag.dag]: Unable to read file: subdag.dag
since subdag.dag is in a different directory.
B) writing a dag:
dag3.dag
SPLICE D subdag/subdag.dag
Fails to find the sub file as it is looking in the main directory:
MultiLogFiles::readFileToString: safe_fopen_wrapper(job.sub) failed
with errno 2 (No such file or directory)
MultiLogFiles: Unable to read file: job.sub
MultiLogFiles: No 'log =' value found in submit file job.sub for node A
ERROR: Failed to locate Condor job log files: Splice[D:subdag/
subdag.dag]: No 'log =' value found in submit file job.sub for node A
Aborting -- try again with the -AllowLogError flag if you *really*
think this shouldn't be a fatal error
Note that running
condor_submit_dag -usedagdir subdag.dag
doesn't seem to help here either.
C) writing a dag:
dag4.dag
SPLICE D subdag.dag DIR subdag
Also fails with:
ERROR: Failed to locate Condor job log files: Splice[D:subdag.dag]:
Unable to read file: subdag.dag; Failed to locate Stork job log files:
Splice[D:subdag.dag]: Unable to read file: subdag.dag
Method C above is the most similar to what is currently used to
include DAGs by adding their condor.sub files to the main DAG. I
think it would be preferable if that was also available for splices,
although any method which allows us to SPLICE sub dags which live in
different directories would be very useful.
Thanks,
Steve
Begin forwarded message:
> From: Stuart Anderson <anderson__AT__ligo.caltech.edu>
> Date: 14 August 2008 21:05:02 BST
> To: Peter Keller <psilord__AT__cs.wisc.edu>, Steve Fairhurst <fairhurst_s__AT__ligo.caltech.edu
> >
> Subject: Re: [CondorLIGO] [stephen.fairhurst__AT__astro.cf.ac.uk: Re: New
> DAG splicing feature]
>
> Steve,
> Please provide Pete with an example. I think it would be helpful
> to do so via a new problem ticket, i.e., send an email to,
> "condor-admin__AT__cs.wisc.edu, psilord__AT__cs.wisc.edu"
> with a subject line that starts with "LIGO:"
>
> Thanks.
>
>
> On Thu, Aug 14, 2008 at 01:50:58PM -0500, Peter Keller wrote:
>> Hello,
>>
>> Could I get a small example of a DAG from you which exhibits this
>> problem?
>>
>> -pete
>
> --
> Stuart Anderson anderson__AT__ligo.caltech.edu
> http://www.ligo.caltech.edu/~anderson
------------------------------------------------------------
Stephen Fairhurst
School of Physics & Astronomy
Cardiff University
The Parade
Cardiff, CF24 3AA, UK.
stephen.fairhurst__AT__astro.cf.ac.uk
Tel: +44 (0) 2920 870166
--Apple-Mail-6-922664252
<html><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webk=
it-line-break: after-white-space; ">The DAG SPLICE capacity does not seem t=
o be compatible with the DIR command. <div><br></div><div>As an examp=
le, consider the very simple dag:<div><br></div><div>subdag.dag</div><div><=
div>JOB A job.sub</div><div><br></div><div>which is located in a direc=
tory subdag.</div><div><br></div><div>I want to include this into a larger =
dag (dag.dag). I know that it is possible by:</div><div><br></div><di=
v>1) running condor_submit_dag -no_submit subdag.dag</div><div>2) submittin=
g the following dag:</div><div><br></div><div>dag.dag</div><div><div>JOB D =
subdag.dag.condor.sub DIR subdag</div><div><br></div><div>I have been unabl=
e to replicate this procedure using the SPLICE command. In particular=
, </div><div><br></div><div>A) writing a dag:</div><div><br></div><div=
>dag2.dag</div><div><div>SPLICE D subdag.dag</div><div><br></div><div>Unsur=
prisingly fails with </div><div><div><br></div><div>ERROR: Failed to l=
ocate Condor job log files: Splice[D:subdag.dag]: Unable to read file: subd=
ag.dag; Failed to locate Stork job log files: Splice[D:subdag.dag]: Unable =
to read file: subdag.dag</div><div><br></div></div></div></div><div>since s=
ubdag.dag is in a different directory.</div><div><br></div><div><br></div><=
div><div><div>B) writing a dag:</div><div><br></div><div><div>dag3.dag</div=
><div>SPLICE D subdag/subdag.dag </div><div><br></div><div>Fails to fi=
nd the sub file as it is looking in the main directory:</div><div><br></div=
><div><div>MultiLogFiles::readFileToString: safe_fopen_wrapper(job.sub) fai=
led with errno 2 (No such file or directory)</div><div>MultiLogFiles: Unabl=
e to read file: job.sub</div><div>MultiLogFiles: No 'log =3D' value found i=
n submit file job.sub for node A</div><div>ERROR: Failed to locate Condor j=
ob log files: Splice[D:subdag/subdag.dag]: No 'log =3D' value found in subm=
it file job.sub for node A</div><div>Aborting -- try again with the -AllowL=
ogError flag if you *really* think this shouldn't be a fatal error</div><di=
v><br></div><div>Note that running </div><div>condor_submit_dag -useda=
gdir subdag.dag </div><div>doesn't seem to help here either.</div><div=
><br></div><div><br></div><div><div>C) writing a dag:</div><div><br></div><=
div>dag4.dag</div><div><div>SPLICE D subdag.dag DIR subdag</div><div><br></=
div><div>Also fails with:</div><div><div>ERROR: Failed to locate Condor job=
log files: Splice[D:subdag.dag]: Unable to read file: subdag.dag; Failed t=
o locate Stork job log files: Splice[D:subdag.dag]: Unable to read file: su=
bdag.dag</div><div><br></div><div><br></div><div>Method C above is the most=
similar to what is currently used to include DAGs by adding their condor.s=
ub files to the main DAG. I think it would be preferable if that was =
also available for splices, although any method which allows us to SPLICE s=
ub dags which live in different directories would be very useful.</div><div=
><br></div><div>Thanks,</div><div><span class=3D"Apple-tab-span" style=3D"w=
hite-space:pre"> </span>Steve</div><div><br></div></div></div></div><div><b=
r></div></div></div></div><div>Begin forwarded message:</div><br class=3D"A=
pple-interchange-newline"><blockquote type=3D"cite"><div><div style=3D"marg=
in-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><fo=
nt face=3D"Helvetica" size=3D"3" color=3D"#000000" style=3D"font: 12.0px He=
lvetica; color: #000000"><b>From: </b></font><font face=3D"Helvetica" size=
=3D"3" style=3D"font: 12.0px Helvetica">Stuart Anderson <<a href=3D"mail=
to:anderson__AT__ligo.caltech.edu">anderson__AT__ligo.caltech.edu</a>></font></div><d=
iv style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-=
left: 0px; "><font face=3D"Helvetica" size=3D"3" color=3D"#000000" style=3D=
"font: 12.0px Helvetica; color: #000000"><b>Date: </b></font><font face=3D"=
Helvetica" size=3D"3" style=3D"font: 12.0px Helvetica">14 August 2008 21:05=
:02 BST</font></div><div style=3D"margin-top: 0px; margin-right: 0px; margi=
n-bottom: 0px; margin-left: 0px; "><font face=3D"Helvetica" size=3D"3" colo=
r=3D"#000000" style=3D"font: 12.0px Helvetica; color: #000000"><b>To: </b><=
/font><font face=3D"Helvetica" size=3D"3" style=3D"font: 12.0px Helvetica">=
Peter Keller <<a href=3D"mailto:psilord__AT__cs.wisc.edu">psilord__AT__cs.wisc.edu=
</a>>, Steve Fairhurst <<a href=3D"mailto:fairhurst_s__AT__ligo.caltech.edu">=
fairhurst_s__AT__ligo.caltech.edu</a>></font></div><div style=3D"margin-top: 0px=
; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face=3D"=
Helvetica" size=3D"3" color=3D"#000000" style=3D"font: 12.0px Helvetica; co=
lor: #000000"><b>Subject: </b></font><font face=3D"Helvetica" size=3D"3" st=
yle=3D"font: 12.0px Helvetica"><b>Re: [CondorLIGO] [<a href=3D"mailto:steph=
en.fairhurst__AT__astro.cf.ac.uk">stephen.fairhurst__AT__astro.cf.ac.uk</a>: Re: New =
DAG splicing feature]</b></font></div><div style=3D"margin-top: 0px; margin=
-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br>=
</div> </div><div>Steve,<br><span class=3D"Apple-tab-span" style=3D"white-s=
pace:pre"> </span>Please provide Pete with an example. I think it would be =
helpful<br>to do so via a new problem ticket, i.e., send an email to,<br>"<=
a href=3D"mailto:condor-admin__AT__cs.wisc.edu">condor-admin__AT__cs.wisc.edu</a>, <a=
href=3D"mailto:psilord__AT__cs.wisc.edu">psilord__AT__cs.wisc.edu</a>"<br>with a sub=
ject line that starts with "LIGO:"<br><br>Thanks.<br><br><br>On Thu, Aug 14=
, 2008 at 01:50:58PM -0500, Peter Keller wrote:<br><blockquote type=3D"cite=
">Hello,<br></blockquote><blockquote type=3D"cite"><br></blockquote><blockq=
uote type=3D"cite">Could I get a small example of a DAG from you which exhi=
bits this problem?<br></blockquote><blockquote type=3D"cite"><br></blockquo=
te><blockquote type=3D"cite">-pete<br></blockquote><br>-- <br>Stuart Anders=
on <a href=3D"mailto:anderson__AT__ligo.caltech.edu">anderson__AT__ligo.caltech=
.edu</a><br><a href=3D"http://www.ligo.caltech.edu/~anderson">http://www.li=
go.caltech.edu/~anderson</a><br></div></blockquote></div><br><div apple-con=
tent-edited=3D"true"> <span class=3D"Apple-style-span" style=3D"border-coll=
apse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12p=
x; font-style: normal; font-variant: normal; font-weight: normal; letter-sp=
acing: normal; line-height: normal; orphans: 2; text-align: auto; text-inde=
nt: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing=
: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spac=
ing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjus=
t: auto; -webkit-text-stroke-width: 0; "><div style=3D"word-wrap: break-wor=
d; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span=
class=3D"Apple-style-span" style=3D"border-collapse: separate; -webkit-bor=
der-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; color: r=
gb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; f=
ont-variant: normal; font-weight: normal; letter-spacing: normal; line-heig=
ht: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -we=
bkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space:=
normal; widows: 2; word-spacing: 0px; "><span class=3D"Apple-style-span" s=
tyle=3D"border-collapse: separate; -webkit-border-horizontal-spacing: 0px; =
-webkit-border-vertical-spacing: 0px; color: rgb(0, 0, 0); font-family: Hel=
vetica; font-size: 12px; font-style: normal; font-variant: normal; font-wei=
ght: normal; letter-spacing: normal; line-height: normal; -webkit-text-deco=
rations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; =
text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spac=
ing: 0px; "><span class=3D"Apple-style-span" style=3D"border-collapse: sepa=
rate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spaci=
ng: 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font=
-style: normal; font-variant: normal; font-weight: normal; letter-spacing: =
normal; line-height: normal; -webkit-text-decorations-in-effect: none; text=
-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans=
: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class=3D"Ap=
ple-style-span" style=3D"border-collapse: separate; -webkit-border-horizont=
al-spacing: 0px; -webkit-border-vertical-spacing: 0px; color: rgb(0, 0, 0);=
font-family: Helvetica; font-size: 12px; font-style: normal; font-variant:=
normal; font-weight: normal; letter-spacing: normal; line-height: normal; =
-webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-si=
ze-adjust: auto; text-transform: none; orphans: 2; white-space: normal; wid=
ows: 2; word-spacing: 0px; "><span class=3D"Apple-style-span" style=3D"bord=
er-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-bord=
er-vertical-spacing: 0px; color: rgb(0, 0, 0); font-family: Helvetica; font=
-size: 12px; font-style: normal; font-variant: normal; font-weight: normal;=
letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-e=
ffect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transfo=
rm: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; ">=
<div>------------------------------------------------------------</div><div=
>Stephen Fairhurst <div><div>School of Physics & Astronomy</div><d=
iv>Cardiff University</div><div>The Parade</div><div>Cardiff, CF24 3AA, UK.=
</div><div><br class=3D"khtml-block-placeholder"></div><div><a href=3D"mail=
to:stephen.fairhurst__AT__astro.cf.ac.uk">stephen.fairhurst__AT__astro.cf.ac.uk</a></=
div><div>Tel: +44 (0) 2920 870166</div><div><br></div></div></div><br class=
=3D"Apple-interchange-newline"></span></span></span></span></span></div></s=
pan><br class=3D"Apple-interchange-newline"> </div><br></div></div></body><=
/html>
--Apple-Mail-6-922664252--
===========================================================================
Date of creation: Tue Aug 26 5:54:12 2008 (1219748055)
Subject: Actions
Assigned to psilord by gthain
===========================================================================
Date of actions: Tue Aug 26 9:18:29 2008 (1219760309)
Date: Tue, 26 Aug 2008 14:38:57 -0500
From: Peter Keller <psilord__AT__cs.wisc.edu>
To: gthain <condor-admin__AT__cs.wisc.edu>
Subject: Re: [condor-admin #18376] LIGO: Use of DIR in DAG SPLICE
Hello,
Thanks for the detailed use case. I'll schedule this next to work on when
I finish the current thing I have to do.
Thank you.
-pete
===========================================================================
Date mail was appended: Tue Aug 26 14:39:00 2008 (1219779540)
Date: Mon, 22 Sep 2008 16:59:51 -0500
From: Peter Keller <psilord__AT__cs.wisc.edu>
To: gthain <condor-admin__AT__cs.wisc.edu>
Subject: Re: [condor-admin #18376] LIGO: Use of DIR in DAG SPLICE
Hello,
> From: Stephen Fairhurst <stephen.fairhurst__AT__astro.cf.ac.uk>
[snip]
> C) writing a dag:
>
> dag4.dag
> SPLICE D subdag.dag DIR subdag
>
> Also fails with:
> ERROR: Failed to locate Condor job log files: Splice[D:subdag.dag]:
> Unable to read file: subdag.dag; Failed to locate Stork job log files:
> Splice[D:subdag.dag]: Unable to read file: subdag.dag
>
>
> Method C above is the most similar to what is currently used to
> include DAGs by adding their condor.sub files to the main DAG. I
> think it would be preferable if that was also available for splices,
> although any method which allows us to SPLICE sub dags which live in
> different directories would be very useful.
I have this option implemented, but not fully tested. I hope to finish it
this week and get you some pre-release binaries before friday. We'll see
if the testing reveals any gotchas I may need to iron out, however.
Thank you.
-pete
===========================================================================
Date mail was appended: Mon Sep 22 16:59:53 2008 (1222120793)
From: Stephen Fairhurst <stephen.fairhurst__AT__astro.cf.ac.uk>
To: condor-admin__AT__cs.wisc.edu
Subject: Re: [condor-admin #18376] LIGO: Use of DIR in DAG SPLICE
Date: Tue, 23 Sep 2008 09:45:55 +0200
X-Cu-PHYSX-Virus-Scan: ClamAV did not find anything.
X-Mailfromd-RBL: IP Address 131.251.0.112 is listed on ix.dnsbl.manitu.net
X-Mailfromd: Total of 1 RBL listing (15 mins)
X-Mailfromd-Greylist-Time: May have had a total greylist delay of 15 minutes
X-Seen-BY: mailfromd 4.1 silica.cs.wisc.edu
Hi Pete,
On 22 Sep 2008, at 23:59, condor-admin response tracking system wrote:
> Hello,
>
>> From: Stephen Fairhurst <stephen.fairhurst__AT__astro.cf.ac.uk>
> [snip]
>
>> C) writing a dag:
>>
>> dag4.dag
>> SPLICE D subdag.dag DIR subdag
>>
>> Also fails with:
>> ERROR: Failed to locate Condor job log files: Splice[D:subdag.dag]:
>> Unable to read file: subdag.dag; Failed to locate Stork job log
>> files:
>> Splice[D:subdag.dag]: Unable to read file: subdag.dag
>>
>>
>> Method C above is the most similar to what is currently used to
>> include DAGs by adding their condor.sub files to the main DAG. I
>> think it would be preferable if that was also available for splices,
>> although any method which allows us to SPLICE sub dags which live in
>> different directories would be very useful.
>
> I have this option implemented, but not fully tested. I hope to
> finish it
> this week and get you some pre-release binaries before friday.
> We'll see
> if the testing reveals any gotchas I may need to iron out, however.
>
Great, thank you. Let us know when it is available and we can try to
do some testing.
Cheers,
Steve
> Thank you.
>
> -pete
>
>
> ========================================
> MESSAGE INFORMATION
> ========================================
> * From: Peter Keller <psilord__AT__cs.wisc.edu>
> * Ticket Email List: stephen.fairhurst__AT__astro.cf.ac.uk,
>
------------------------------------------------------------
Stephen Fairhurst
School of Physics & Astronomy
Cardiff University
The Parade
Cardiff, CF24 3AA, UK.
stephen.fairhurst__AT__astro.cf.ac.uk
Tel: +44 (0) 2920 870166
===========================================================================
Date mail was appended: Tue Sep 23 10:44:24 2008 (1222184665)
Date: Wed, 15 Oct 2008 16:16:52 -0500
From: Peter Keller <psilord__AT__cs.wisc.edu>
To: condor-admin response tracking system <condor-admin__AT__cs.wisc.edu>
Subject: Re: [condor-admin #18376] LIGO: Use of DIR in DAG SPLICE
Hello,
After some serious hacking, I've implemented splicing with subdirectories.
The formats allowed are:
1.
SPLICE A splice.dag DIR a/b/c
2.
SPLICE A /a/b/c/splice.dag
They work in very similar ways as to how DIR works with JOBs. It is
possible to create invalid dags by mixing the above forms. I recommend
only using form (1) in general for both SPLICEs and JOBs.
This feature should be available in the 7.1.4 series of Condor, and I
can give you a prerelease if need be in a day or two.
Thank you.
-pete
===========================================================================
Date mail was appended: Wed Oct 15 16:16:54 2008 (1224105415)
From: Stephen Fairhurst <stephen.fairhurst__AT__astro.cf.ac.uk>
To: condor-admin__AT__cs.wisc.edu
Subject: Re: [condor-admin #18376] LIGO: Use of DIR in DAG SPLICE
Date: Mon, 20 Oct 2008 12:58:55 +0100
X-Cu-PHYSX-Virus-Scan: ClamAV did not find anything.
X-Seen-BY: mailfromd 4.1 granite.cs.wisc.edu
Hi Pete,
Thank you for putting this feature in, what you describe sounds like a
good approach. I think it will be very useful. I'd be happy to try it
out if necessary, otherwise will wait for the next release.
Cheers,
Steve
On 15 Oct 2008, at 22:16, condor-admin response tracking system wrote:
> Hello,
>
> After some serious hacking, I've implemented splicing with
> subdirectories.
>
> The formats allowed are:
>
> 1.
> SPLICE A splice.dag DIR a/b/c
>
> 2.
> SPLICE A /a/b/c/splice.dag
>
> They work in very similar ways as to how DIR works with JOBs. It is
> possible to create invalid dags by mixing the above forms. I recommend
> only using form (1) in general for both SPLICEs and JOBs.
>
> This feature should be available in the 7.1.4 series of Condor, and I
> can give you a prerelease if need be in a day or two.
>
> Thank you.
>
> -pete
>
>
> ========================================
> MESSAGE INFORMATION
> ========================================
> * From: Peter Keller <psilord__AT__cs.wisc.edu>
> * Ticket Email List: stephen.fairhurst__AT__astro.cf.ac.uk,
>
------------------------------------------------------------
Stephen Fairhurst
School of Physics & Astronomy
Cardiff University
The Parade
Cardiff, CF24 3AA, UK.
stephen.fairhurst__AT__astro.cf.ac.uk
Tel: +44 (0) 2920 870166
===========================================================================
Date mail was appended: Mon Oct 20 6:59:21 2008 (1224503961)
Subject: Comments added
Stuart said via email on 12/11:
Pete,
Unless you think it is helpful to keep the original DIR+SPLICE ticket open (18376) I would like to suggest you resolve that ticket in favor of tracking this new stack dump problem.
Thanks.
So this ticket now "continues" in ticket 18817
Comments added by tannenba
===========================================================================
Date comments were added: Fri Dec 12 11:21:09 2008 (1229102469)
Subject: Actions
Ticket resolved by tannenba
===========================================================================
Date of actions: Fri Dec 12 11:22:59 2008 (1229102579)