LIGO Support Ticket 17209
Ticket Information
Number: admin 17209
User: anderson@ligo.caltech.edu
Email: skoranda__AT__gravity.phys.uwm.edu
Status: resolved
Assigned To: nleroy
Date: Tue, 13 Nov 2007 16:43:10 -0800
From: Stuart Anderson <anderson__AT__ligo.caltech.edu>
To: condor-admin__AT__cs.wisc.edu
CC: Scott Koranda <skoranda__AT__gravity.phys.uwm.edu>
Subject: LIGO: reduce cost of catalog building
X-Seen-BY: mailfromd 4.1 granite.cs.wisc.edu
It may be possible to significantly reduce the number of system calls made
by Condor to build a file catalog by changing the order of stat() and lstat()
when scanning a directory. This is not a priority for LIGO now that
condor_shadow no longer stat()'s any files to build a catalog, but if might
be useful on the execute machines for other projects, or possibly LIGO in
the long term.
In particular, in StatWrapper::DoStat why always call STAT_FUNC() followed
by LSTAT_FUNC(). If instead lstat() was called first it should only be
necessary to call stat() if the st_mode field from the first call indicates
that the path is a symbolic link. Assuming this is right and that most
directories are not dominated by the number of symbolic links in them,
this would significantly reduce the total number of system calls needed
to build a condor file catalog.
P.S. This was a side discussion in ticket 15795 but I probably did not state
the possibly improvement clearly enough there.
Thanks.
--
Stuart Anderson anderson__AT__ligo.caltech.edu
http://www.ligo.caltech.edu/~anderson
===========================================================================
Date of creation: Tue Nov 13 18:43:43 2007 (1195001027)
Subject: Actions
Assigned to nleroy by gthain
===========================================================================
Date of actions: Wed Nov 14 10:07:57 2007 (1195056477)
From: Nick LeRoy <nleroy__AT__cs.wisc.edu>
To: condor-admin__AT__cs.wisc.edu
Subject: Re: [condor-admin #17209] LIGO: reduce cost of catalog building
Date: Thu, 15 Nov 2007 08:51:28 -0600
Hello,
> It may be possible to significantly reduce the number of system calls made
> by Condor to build a file catalog by changing the order of stat() and
> lstat() when scanning a directory. This is not a priority for LIGO now
> that condor_shadow no longer stat()'s any files to build a catalog, but if
> might be useful on the execute machines for other projects, or possibly
> LIGO in the long term.
>
> In particular, in StatWrapper::DoStat why always call STAT_FUNC() followed
> by LSTAT_FUNC(). If instead lstat() was called first it should only be
> necessary to call stat() if the st_mode field from the first call indicates
> that the path is a symbolic link. Assuming this is right and that most
> directories are not dominated by the number of symbolic links in them,
> this would significantly reduce the total number of system calls needed
> to build a condor file catalog.
>
> P.S. This was a side discussion in ticket 15795 but I probably did not
> state the possibly improvement clearly enough there.
Yeah, this is my fault from some work I did a while ago. I've been aware of
this bad behavior for a while, but haven't taken the time to fix it. Because
of this ticket, I'm working on it, but, unfortunately, the fix isn't simple.
It *might* make it into 6.9.5 (and, thus, the next stable release), but it's
certainly not a given.
> Thanks.
-Nick
--
<<< Follow the white rabbit. >>>
/`-_ Nicholas R. LeRoy The Condor Project
{ }/ http://www.cs.wisc.edu/~nleroy http://www.cs.wisc.edu/condor
\ / nleroy__AT__cs.wisc.edu The University of Wisconsin
|_*_| 608-265-5761 Department of Computer Sciences
===========================================================================
Date mail was appended: Thu Nov 15 8:51:38 2007 (1195138298)
Subject: Actions
Ticket resolved by nleroy
===========================================================================
Date of actions: Thu Nov 15 8:51:38 2007 (1195138300)