LIGO Support Ticket 1816

Ticket Information
  Number:      support 1816
  User:        anderson@ligo.caltech.edu
  Email:       espinoza_e__AT__ligo.caltech.edu,ldas_admin_llo__AT__ligo.caltech.edu,igor__AT__ligo-la.caltech.edu
  Status:      resolved
  Assigned To: tannenba

Date: Sat, 20 Jan 2007 15:08:50 -0800
From: Stuart Anderson <anderson__AT__ligo.caltech.edu>
To: condor-support__AT__cs.wisc.edu
CC: Erik Espinoza <espinoza_e__AT__ligo.caltech.edu>,
 ldas_admin_llo__AT__ligo.caltech.edu
Subject: LIGO: schedd exit stat 4 due to log_transaction failure
X-MIME-Autoconverted: from quoted-printable to 8bit by chopin.cs.wisc.edu 
 id l0KN99ax014428

The LIGO LLO Condor pool running:

$ condor_version
$CondorVersion: 6.8.2 Oct 12 2006 $
$CondorPlatform: I386-LINUX_RHEL3 $

has recently had several restarts of condor_schedd due to condor_schedd exiting
with status 4 and reporting:

...
/vds/lib/puretls.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/mysql-connector-jav
a-3.0.11-stable-bin.jar:/ldcg/stow_pkg
 .mit.edu
/ldg4.3/software PACMAN_LOCATION=/ldcg/pacman G_BROKEN_FILENAMES=1"' to record
'04569965.-1' as it contains a newline, which is not allowed.
1/19 21:11:59 (pid:3944) ERROR "write inside a transaction failed, errno = 0" at
line 127 in file log_transaction.C

These appear to happen in bursts, i.e.,

$ grep STARTING SchedLog
1/19 18:17:42 (pid:28232) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
1/19 18:56:57 (pid:3944) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
1/19 21:13:03 (pid:28001) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
1/19 21:32:09 (pid:32438) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
1/19 21:33:23 (pid:333) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
1/19 21:34:08 (pid:410) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
1/19 21:35:12 (pid:477) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
1/19 21:37:41 (pid:981) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
1/19 21:43:33 (pid:2357) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
1/20 12:31:13 (pid:2685) ** condor_schedd (CONDOR_SCHEDD) STARTING UP


Here is a section of the SchedLog file before one of these restarts:

1/19 18:15:32 (pid:2543) DaemonCore: Command received via UDP from host <10.13.0.12:41912>
1/19 18:15:32 (pid:2543) DaemonCore: received command 421 (RESCHEDULE), calling handler (reschedule_negotiator)
1/19 18:15:32 (pid:2543) Called reschedule_negotiator()
1/19 18:15:39 (pid:2543) Sent ad to central manager for kleinewelle@ligo
1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for kleinewelle@ligo
1/19 18:15:39 (pid:2543) Sent ad to central manager for inspiralbns@ligo
1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for inspiralbns@ligo
1/19 18:15:39 (pid:2543) Sent ad to central manager for waveburst_test@ligo
1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for waveburst_test@ligo
1/19 18:15:39 (pid:2543) Sent ad to central manager for pulsar@ligo
1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for pulsar@ligo
1/19 18:15:39 (pid:2543) Sent ad to central manager for hoft@ligo
1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for hoft@ligo
1/19 18:15:39 (pid:2543) Sent ad to central manager for lindy@ligo
1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for lindy@ligo
1/19 18:15:39 (pid:2543) DaemonCore: Command received via UDP from host <10.13.0.12:41919>
1/19 18:15:39 (pid:2543) DaemonCore: received command 421 (RESCHEDULE), calling handler (reschedule_negotiator)
1/19 18:15:39 (pid:2543) Called reschedule_negotiator()
1/19 18:15:39 (pid:2543) Shadow pid 27707 for job 4567817.0 exited with status 100
1/19 18:15:41 (pid:2543) Shadow pid 27709 for job 4567818.0 exited with status 100
1/19 18:15:41 (pid:2543) Starting add_shadow_birthdate(4567844.0)
1/19 18:15:41 (pid:2543) Started shadow for job 4567844.0 on "<10.13.1.37:52609>", (shadow pid = 28158)
1/19 18:15:41 (pid:2543) Starting add_shadow_birthdate(4567845.0)
1/19 18:15:41 (pid:2543) Started shadow for job 4567845.0 on "<10.13.1.58:54972>", (shadow pid = 28160)
1/19 18:15:41 (pid:2543) Shadow pid 27713 for job 4567819.0 exited with status 100
1/19 18:15:42 (pid:2543) Starting add_shadow_birthdate(4567846.0)
1/19 18:15:42 (pid:2543) Started shadow for job 4567846.0 on "<10.13.1.121:53621>", (shadow pid = 28162)
1/19 18:15:42 (pid:2543) DaemonCore: Command received via UDP from host <10.13.0.12:41919>
1/19 18:15:42 (pid:2543) DaemonCore: received command 421 (RESCHEDULE), calling handler (reschedule_negotiator)
1/19 18:15:42 (pid:2543) Called reschedule_negotiator()
1/19 18:15:42 (pid:2543) Activity on stashed negotiator socket
1/19 18:15:42 (pid:2543) Negotiating for owner: inspiralbns@ligo
1/19 18:15:42 (pid:2543) Checking consistency running and runnable jobs
1/19 18:15:42 (pid:2543) Tables are consistent
1/19 18:15:42 (pid:2543) Out of servers - 0 jobs matched, 1 jobs idle, 1 jobs rejected
1/19 18:15:42 (pid:2543) Activity on stashed negotiator socket
1/19 18:15:42 (pid:2543) Negotiating for owner: inspiralbns@ligo
1/19 18:15:42 (pid:2543) Checking consistency running and runnable jobs
1/19 18:15:42 (pid:2543) Tables are consistent
1/19 18:15:42 (pid:2543) Out of jobs - 1 jobs matched, 0 jobs idle, flock level = 0
1/19 18:15:42 (pid:2543) Shadow pid 27721 for job 4567820.0 exited with status 100
1/19 18:15:43 (pid:2543) match (<10.13.1.130:59771>#1164737958#9096) out of jobs (cluster id 4567847); relinquishing
1/19 18:15:43 (pid:2543) Sent RELEASE_CLAIM to startd on <10.13.1.130:59771>
1/19 18:15:43 (pid:2543) Match record (<10.13.1.130:59771>, 4567847, 0) deleted
1/19 18:15:43 (pid:2543) Starting add_shadow_birthdate(4567847.0)
1/19 18:15:43 (pid:2543) Started shadow for job 4567847.0 on "<10.13.1.60:54490>", (shadow pid = 28164)
1/19 18:15:43 (pid:2543) DaemonCore: Command received via TCP from host <10.13.1.130:56651>
1/19 18:15:43 (pid:2543) DaemonCore: received command 443 (VACATE_SERVICE), calling handler (vacate_service)
1/19 18:15:43 (pid:2543) Got VACATE_SERVICE from <10.13.1.130:56651>
1/19 18:15:44 (pid:2543) Sent ad to central manager for kleinewelle@ligo
1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for kleinewelle@ligo
1/19 18:15:44 (pid:2543) Sent ad to central manager for inspiralbns@ligo
1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for inspiralbns@ligo
1/19 18:15:44 (pid:2543) Sent ad to central manager for waveburst_test@ligo
1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for waveburst_test@ligo
1/19 18:15:44 (pid:2543) Sent ad to central manager for pulsar@ligo
1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for pulsar@ligo
1/19 18:15:44 (pid:2543) Sent ad to central manager for hoft@ligo
1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for hoft@ligo
1/19 18:15:44 (pid:2543) Sent ad to central manager for lindy@ligo
1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for lindy@ligo
1/19 18:15:44 (pid:2543) Refusing attempt to add 'Environment' = '"EDITOR=emacs GRID_SECURITY_DIR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/etc MANPATH=/ldcg/condor/man:/opt/lscsoft/lalapps/share/man:/opt/lscsoft/lal/share/man:/opt/lscsoft/glue/man:/opt/lscsoft/libframe/man:/opt/lscsoft/libmetaio/man:/ldcg/condor//man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/man:/ldcg/condor//man:/ldcg/condor/man:/opt/lscsoft/lalapps/share/man:/opt/lscsoft/lal/share/man:/opt/lscsoft/glue/man:/opt/lscsoft/libframe/man:/opt/lscsoft/libmetaio/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/man:/ldcg/condor//man:::/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/share/man TERM=screen LAL_PREFIX=/opt/lscsoft/lal SASL_PATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib/sasl GSTAR_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar MYSQL_UNIX_PORT=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt-app-data/mysql/var/mysql.sock LSCSOFT_PREFIX=/opt/lscsoft HOSTNAME=ldas-grid SHELL=/bin/bash MATLABPATH=/ligotools/matlab WBONLINE=/archive/home/waveburst/S5_online VOMS_USERCONF=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/etc EDG_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg LDG_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg HISTSIZE=1000 GLOBUS_PATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus SSH_CLIENT=130.39.245.165' '37419' '22 PYLAL_LOCATION=/archive/home/ram/opt/pylal GLOBUS_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus X509_CADIR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/TRUSTED_CA X509_CERT_DIR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/TRUSTED_CA PERL5LIB=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/perl:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0/i686-linux-thread-multi::/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar/lib CVSROOT=:pserver:igor__AT__ldas-sw.ligo.caltech.edu:/ldcg_server/common/repository_gds PYTHONPATH=/opt/lscsoft/lalapps/lib/python2.4/site-packages:/opt/lscsoft/glue/lib/python:/opt/lscsoft/libframe/lib/python:/opt/lscsoft/libmetaio/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib/python2.2/site-packages:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib64/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib/python:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib/python2.2/site-packages:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib64/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib/python:/ldcg/pacman/src:/opt/lscsoft/lalapps/lib/python2.4/site-packages:/opt/lscsoft/glue/lib/python:/opt/lscsoft/libframe/lib/python:/opt/lscsoft/libmetaio/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib/python2.2/site-packages:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib64/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib/python:/ldcg/pacman/src: QTDIR=/usr/lib/qt-3.3 LAL_LOCATION=/opt/lscsoft/lalapps EXTRAS_LOCATION=/archive/home/ram/opt/extras SHLVL=3 SSH_TTY=/dev/pts/0 TZ=America/Chicago GLITE_LOCATION_LOG=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/log VDS_HOME=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds GLOBUS_TCP_PORT_RANGE=40000,45000 USER=waveburst GLUE_LOCATION=/archive/home/ram/opt/glue GLOBUS_ERROR_VERBOSE=true LALAPPS_LOCATION=/archive/home/ram/opt/lalapps LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35: LD_LIBRARY_PATH=/opt/lscsoft/lal/lib:/opt/lscsoft/glue/lib:/opt/lscsoft/libframe/lib:/opt/lscsoft/libmetaio/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/myodbc/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/lib/mysql:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/jre/lib/i386:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/jre/lib/i386/server:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/jre/lib/i386/client:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/berkeley-db/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib:/ligotools/lib ROOTSYS=/archive/home/igor/SOFT1/root GPT_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt LDG_INSTALL_LOG=/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/etc/ldg-install.log GLITE_LOCATION_TMP=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/tmp TERMCAP=SC|screen|VT' '100/ANSI' 'X3.64' 'virtual' 'terminal:\'
	':DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:UP=\E[%dA:bs:bt=\E[Z:\'
	':cd=\E[J:ce=\E[K:cl=\E[H\E[J:cm=\E[%i%d;%dH:ct=\E[3g:\'
	':do=^J:nd=\E[C:pt:rc=\E8:rs=\Ec:sc=\E7:st=\EH:up=\EM:\'
	':le=^H:bl=^G:cr=^M:it#8:ho=\E[H:nw=\EE:ta=^I:is=\E)0:\'
	':li#55:co#154:am:xn:xv:LP:sr=\EM:al=\E[L:AL=\E[%dL:\'
	':cs=\E[%i%d;%dr:dl=\E[M:DL=\E[%dM:dc=\E[P:DC=\E[%dP:\'
	':im=\E[4h:ei=\E[4l:mi:IC=\E[%d@:ks=\E[?1h\E=:\'
	':ke=\E[?1l\E>:vi=\E[?25l:ve=\E[34h\E[?25h:vs=\E[34l:\'
	':ti=\E[?1049h:te=\E[?1049l:us=\E[4m:ue=\E[24m:so=\E[3m:\'
	':se=\E[23m:mb=\E[5m:md=\E[1m:mr=\E[7m:me=\E[m:ms:\'
	':Co#8:pa#64:AF=\E[3%dm:AB=\E[4%dm:op=\E[39;49m:AX:\'
	':vb=\Eg:G0:as=\E(0:ae=\E(B:\'
	':ac=\140\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:\'
	':po=\E[5i:pf=\E[4i:Z0=\E[?3h:Z1=\E[?3l:k0=\E[10~:\'
	':k1=\EOP:k2=\EOQ:k3=\EOR:k4=\EOS:k5=\E[15~:k6=\E[17~:\'
	':k7=\E[18~:k8=\E[19~:k9=\E[20~:k;=\E[21~:F1=\E[23~:\'
	':F2=\E[24~:F3=\EO2P:F4=\EO2Q:F5=\EO2R:F6=\EO2S:\'
	':F7=\E[15;2~:F8=\E[17;2~:F9=\E[18;2~:FA=\E[19;2~:kb=:\'
	':K2=\EOE:kB=\E[Z:*4=\E[3;2~:*7=\E[1;2F:#2=\E[1;2H:\'
	':#3=\E[2;2~:#4=\E[1;2D:%c=\E[6;2~:%e=\E[5;2~:%i=\E[1;2C:\'
	':kh=\E[1~:@1=\E[1~:kH=\E[4~:@7=\E[4~:kN=\E[6~:kP=\E[5~:\'
	':kI=\E[2~:kD=\E[3~:ku=\EOA:kd=\EOB:kr=\EOC:kl=\EOD:km: DAGDBUPDATORLOCKFILE=/etc/onasys-dblockfile PWD=/archive/home/waveburst/COHERENT_ONLINE KDEDIR=/usr LIBPATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib:/ldcg/ldg/vdt/globus/lib:/usr/lib:/lib MAIL=/var/spool/mail/waveburst PATH=/ldcg/condor/bin:/ldcg/condor/sbin:/opt/lscsoft/lalapps/bin:/opt/lscsoft/lal/bin:/opt/lscsoft/glue/bin:/opt/lscsoft/libframe/bin:/opt/lscsoft/libmetaio/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/pyglobus-url-copy/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/ftsh/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/bin:/ldcg/condor//sbin:/ldcg/condor//bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/sbin:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/pyglobus-url-copy/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/ftsh/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/bin:/ldcg/condor//sbin:/ldcg/condor//bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt/sbin:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/bin:/ldcg/condor/bin:/ldcg/condor/sbin:/opt/lscsoft/lalapps/bin:/opt/lscsoft/lal/bin:/opt/lscsoft/glue/bin:/opt/lscsoft/libframe/bin:/opt/lscsoft/libmetaio/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/pyglobus-url-copy/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/ftsh/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/bin:/ldcg/condor//sbin:/ldcg/condor//bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt/sbin:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/bin:/usr/kerberos/bin:/usr/bin:/bin:/usr/sbin:/sbin:/ldcg/ldg/vdt/globus/bin:/usr/X11R6/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar/bin:/ligotools/bin:/ldcg/matlab_r2006a/bin:/archive/home/waveburst/bin:/archive/home/igor/SOFT1/root/bin:/archive/home/waveburst/bin:.:/ldcg/matlab_r2006a/bin SHLIB_PATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib:/ldcg/ldg/vdt/globus/lib STY=14485.pts-0.ldas-grid _=/ldcg/condor/bin/condor_submit JAVA_HOME=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4 CONDOR_LOCATION=/ldcg/condor CONDOR_CONFIG=/ldcg/condor/etc/condor_config VDT_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt ODBCINI=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/etc/odbc.ini LSC_SEGFIND_SERVER=ldas.ligo-la.caltech.edu LOGNAME=waveburst INPUTRC=/etc/inputrc X509_USER_CERT=/archive/home/waveburst/.certificates/ldas-grid.ligo-la.caltech.edu/waveburstcert.pem LANG=C HOME=/archive/home/waveburst LIGOTOOLS=/ligotools GLITE_LOCATION_VAR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/var X509_USER_PROXY=/tmp/x509up_p10457.file9bvIoT.1 BOSSDIR=/etc X509_USER_KEY=/archive/home/waveburst/.certificates/ldas-grid.ligo-la.caltech.edu/waveburstkey.pem VDS_JAVA_HEAPMAX=1024 DYLD_LIBRARY_PATH=/opt/lscsoft/lal/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib:/opt/lscsoft/lal/lib VDT_INSTALL_LOG=vdt-install.log LSC_DATAFIND_SERVER=ldas.ligo-la.caltech.edu WINDOW=0 CLASSPATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/commons-pool.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cog-jglobus.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/java-getopt-1.0.9.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cryptix-asn1.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cryptix.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cryptix32.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/exist-optional.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/exist.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/gvds.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/jakarta-oro.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/loggerservice-stub.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/jce-jdk13-117.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/jlinker.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/junit.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/log4j-1.2.8.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/resolver.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/puretls.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/mysql-connector-java-3.0.11-stable-bin.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/postgresql-8.1dev-400.jdbc3.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xercesImpl.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/rls.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xmlParserAPIs.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xmldb.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xmlrpc.jar GLOBUS_MYSQL_PATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql LSC_DATAGRID_SERVER_LOCATION=/ldcg/ldg SSH_CONNECTION=130.39.245.165' '37419' '130.39.245.243' '22 PKG_CONFIG_PATH=/opt/lscsoft/lal/lib/pkgconfig:/opt/lscsoft/glue/lib/pkgconfig:/opt/lscsoft/libframe/lib/pkgconfig:/opt/lscsoft/libmetaio/lib/pkgconfig:/opt/lscsoft/lal/lib/pkgconfig:/opt/lscsoft/glue/lib/pkgconfig:/opt/lscsoft/libframe/lib/pkgconfig:/opt/lscsoft/libmetaio/lib/pkgconfig: LDG_DIRECTORY=/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server LESSOPEN=|/usr/bin/lesspipe.sh' '%s VDT_POSTINSTALL_README=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/post-install/README DISPLAY=localhost:10.0 GLITE_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite LDG_SOFTWARE_LOCATION=http://www.ligo.mit.edu/ldg4.3/software PACMAN_LOCATION=/ldcg/pacman G_BROKEN_FILENAMES=1"' to record '04567848.-1' as it contains a newline, which is not allowed.
1/19 18:15:44 (pid:2543) ERROR "write inside a transaction failed, errno = 0" at line 127 in file log_transaction.C
1/19 18:17:42 (pid:28232) ******************************************************
1/19 18:17:42 (pid:28232) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
1/19 18:17:42 (pid:28232) ** /ldcg/stow_pkgs/condor-6.8.2/condor/sbin/condor_schedd
1/19 18:17:42 (pid:28232) ** $CondorVersion: 6.8.2 Oct 12 2006 $
1/19 18:17:42 (pid:28232) ** $CondorPlatform: I386-LINUX_RHEL3 $
1/19 18:17:42 (pid:28232) ** PID = 28232
1/19 18:17:42 (pid:28232) ** Log last touched 1/19 18:15:44
1/19 18:17:42 (pid:28232) ******************************************************
1/19 18:17:42 (pid:28232) Using config source: /usr1/condor/condor_config
1/19 18:17:42 (pid:28232) Using local config sources: 
1/19 18:17:42 (pid:28232)    /usr1/condor/condor_config.local
1/19 18:17:42 (pid:28232) DaemonCore: Command Socket at <10.13.0.12:33572>
1/19 18:17:42 (pid:28232) History file rotation is enabled.
1/19 18:17:42 (pid:28232)   Maximum history file size is: 1000000000 bytes
1/19 18:17:42 (pid:28232)   Number of rotated history files is: 100
1/19 18:17:45 (pid:28232) 4567427.0: JobLeaseDuration remaining: 1040
1/19 18:17:47 (pid:28232) Sent ad to central manager for kleinewelle@ligo
1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for kleinewelle@ligo
1/19 18:17:47 (pid:28232) Sent ad to central manager for inspiralbns@ligo
1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for inspiralbns@ligo
1/19 18:17:47 (pid:28232) Sent ad to central manager for waveburst_test@ligo
1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for waveburst_test@ligo
1/19 18:17:47 (pid:28232) Sent ad to central manager for hoft@ligo
1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for hoft@ligo
1/19 18:17:47 (pid:28232) Sent ad to central manager for pulsar@ligo
1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for pulsar@ligo
1/19 18:17:47 (pid:28232) Sent ad to central manager for lindy@ligo
1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for lindy@ligo
1/19 18:17:47 (pid:28232) Starting add_shadow_birthdate(4567427.0)
1/19 18:17:47 (pid:28232) Started shadow for job 4567427.0 on "<10.13.1.163:41265>", (shadow pid = 28238)
1/19 18:17:47 (pid:28232) Successfully created sched universe process
1/19 18:17:47 (pid:28232) Starting add_shadow_birthdate(4343359.0)


Igor,
	All of these restarts appear to be associated with the waveburst
account. Until the Condor team can explain/fix this please carefully
consider what you may have recently changed in the configuration of
this account. The simplest explanation is that you recently added a TERMCAP
environment variable setting that includes a newline which Condor apparently
does not allow.

Thanks.


-- 
Stuart Anderson  anderson__AT__ligo.caltech.edu
http://www.ligo.caltech.edu/~anderson


===========================================================================
Date of creation: Sat Jan 20 17:09:13 2007 (1169334556)

Subject: Actions

Assigned to gquinn by gquinn
===========================================================================
Date of actions: Mon Jan 22 10:03:27 2007 (1169481807)

Date: Mon, 22 Jan 2007 10:27:13 -0600
From: Greg Quinn <gquinn__AT__cs.wisc.edu>
To: condor-support__AT__cs.wisc.edu
Subject: Re: [condor-support #1816] LIGO: schedd exit stat 4 due to     
 log_transaction  failure

Stuart,

Indeed, the newlines in a job's environment are causing the SchedD to 
EXCEPT. I have been able to reproduce this problem locally, and we are 
working on a fix. Meanwhile, I think the only way to avoid this 
problem's continuing occurrence is to modify the offending jobs so they 
don't have newlines in any ClassAd attributes, or to keep them out of 
the queue.

Greg Quinn
Condor Team

gquinn wrote:
> ===========================================================================
> TICKET INFORMATION
> ===========================================================================
> Ticket Queue: condor-support
> Ticket Number: 1816
> Ticket Creation: Sat Jan 20 17:09:13 2007 (1169334556)
> Ticket Updated: Mon Jan 22 10:03:27 2007 (1169481807)
> Ticket Notification: 
> Ticket Category: user
> Ticket Subject: LIGO: schedd exit stat 4 due to log_transaction failure
> Ticket Type: active
> Ticket User(s): anderson__AT__ligo.caltech.edu
> Ticket Owner: gquinn
> Ticket Status: new
> Ticket Priority: normal
> Ticket ETA: 
> ===========================================================================
> Ticket LOGFILE
> ===========================================================================
> espinoza_e__AT__ligo.caltech.edu,ldas_admin_llo__AT__ligo.caltech.edupublicReceived: from shale.cs.wisc.edu (shale.cs.wisc.edu [128.105.6.25]) 	by
>  chopin.cs.wisc.edu (8.13.6/8.13.6) with ESMTP id l0KN99ax014428 	for
>  <condor-support__AT__chopin.cs.wisc.edu>; Sat, 20 Jan 2007 17:09:09 -0600
> Received: from obsidian.cs.wisc.edu (obsidian.cs.wisc.edu [128.105.6.13])
>  by shale.cs.wisc.edu (8.13.6/8.13.6) with ESMTP id l0KN99Zn016834 	for
>  <condor-support__AT__cs.wisc.edu>; Sat, 20 Jan 2007 17:09:09 -0600
> Received: from acrux.ligo.caltech.edu (acrux.ligo.caltech.edu
>  [131.215.115.14]) 	by obsidian.cs.wisc.edu (8.13.6/8.13.6) with ESMTP id
>  l0KN8v8O005576 	for <condor-support__AT__cs.wisc.edu>; Sat, 20 Jan 2007
>  17:09:02 -0600
> Received: from alphard.ligo.caltech.edu (alphard [131.215.114.160]) 	by
>  acrux.ligo.caltech.edu (8.12.11/8.12.11) with ESMTP id l0KN8tUg001444
>  (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT);
>  Sat, 20 Jan 2007 15:08:55 -0800 (PST)
> Received: from alphard.ligo.caltech.edu (localhost.localdomain
>  [127.0.0.1]) 	by alphard.ligo.caltech.edu (8.13.4/8.13.4) with ESMTP id
>  l0KN8o6W000656; 	Sat, 20 Jan 2007 15:08:50 -0800
> Received: (from anderson@localhost) 	by alphard.ligo.caltech.edu
>  (8.13.4/8.13.4/Submit) id l0KN8ogl000655; 	Sat, 20 Jan 2007 15:08:50 -0800
> Date: Sat, 20 Jan 2007 15:08:50 -0800
> From: Stuart Anderson <anderson__AT__ligo.caltech.edu>
> To: condor-support__AT__cs.wisc.edu
> CC: Erik Espinoza <espinoza_e__AT__ligo.caltech.edu>,
>  ldas_admin_llo__AT__ligo.caltech.edu
> Subject: LIGO: schedd exit stat 4 due to log_transaction failure
> Message-ID: <20070120230850.GA32685__AT__ligo.caltech.edu>
> MIME-Version: 1.0
> Content-Type: text/plain; charset=us-ascii
> Content-Disposition: inline
> User-Agent: Mutt/1.4.2.1i
> X-Spam-Score: undef - Domain Whitelisted (ligo.caltech.edu: )
> X-Canit-Stats-ID: 5852267 - 92b548b85f8e
> X-Scanned-BY: CanIt (www . roaringpenguin . com) on 131.215.115.14
> X-CSL-Mailscanner-Information: Please contact lab__AT__cs.wisc.edu for more  
>  information
> X-CSL-Mailscanner: Found to be clean
> Content-Transfer-Encoding: 8bit
> X-MIME-Autoconverted: from quoted-printable to 8bit by chopin.cs.wisc.edu 
>  id l0KN99ax014428
> 
> The LIGO LLO Condor pool running:
> 
> $ condor_version
> $CondorVersion: 6.8.2 Oct 12 2006 $
> $CondorPlatform: I386-LINUX_RHEL3 $
> 
> has recently had several restarts of condor_schedd due to condor_schedd exiting
> with status 4 and reporting:
> 
> ...
> /vds/lib/puretls.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/mysql-connector-jav
> a-3.0.11-stable-bin.jar:/ldcg/stow_pkg
>  .mit.edu
> /ldg4.3/software PACMAN_LOCATION=/ldcg/pacman G_BROKEN_FILENAMES=1"' to record
> '04569965.-1' as it contains a newline, which is not allowed.
> 1/19 21:11:59 (pid:3944) ERROR "write inside a transaction failed, errno = 0" at
> line 127 in file log_transaction.C
> 
> These appear to happen in bursts, i.e.,
> 
> $ grep STARTING SchedLog
> 1/19 18:17:42 (pid:28232) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 18:56:57 (pid:3944) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 21:13:03 (pid:28001) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 21:32:09 (pid:32438) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 21:33:23 (pid:333) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 21:34:08 (pid:410) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 21:35:12 (pid:477) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 21:37:41 (pid:981) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 21:43:33 (pid:2357) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/20 12:31:13 (pid:2685) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 
> 
> Here is a section of the SchedLog file before one of these restarts:
> 
> 1/19 18:15:32 (pid:2543) DaemonCore: Command received via UDP from host <10.13.0.12:41912>
> 1/19 18:15:32 (pid:2543) DaemonCore: received command 421 (RESCHEDULE), calling handler (reschedule_negotiator)
> 1/19 18:15:32 (pid:2543) Called reschedule_negotiator()
> 1/19 18:15:39 (pid:2543) Sent ad to central manager for kleinewelle@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for kleinewelle@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to central manager for inspiralbns@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for inspiralbns@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to central manager for waveburst_test@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for waveburst_test@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to central manager for pulsar@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for pulsar@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to central manager for hoft@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for hoft@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to central manager for lindy@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for lindy@ligo
> 1/19 18:15:39 (pid:2543) DaemonCore: Command received via UDP from host <10.13.0.12:41919>
> 1/19 18:15:39 (pid:2543) DaemonCore: received command 421 (RESCHEDULE), calling handler (reschedule_negotiator)
> 1/19 18:15:39 (pid:2543) Called reschedule_negotiator()
> 1/19 18:15:39 (pid:2543) Shadow pid 27707 for job 4567817.0 exited with status 100
> 1/19 18:15:41 (pid:2543) Shadow pid 27709 for job 4567818.0 exited with status 100
> 1/19 18:15:41 (pid:2543) Starting add_shadow_birthdate(4567844.0)
> 1/19 18:15:41 (pid:2543) Started shadow for job 4567844.0 on "<10.13.1.37:52609>", (shadow pid = 28158)
> 1/19 18:15:41 (pid:2543) Starting add_shadow_birthdate(4567845.0)
> 1/19 18:15:41 (pid:2543) Started shadow for job 4567845.0 on "<10.13.1.58:54972>", (shadow pid = 28160)
> 1/19 18:15:41 (pid:2543) Shadow pid 27713 for job 4567819.0 exited with status 100
> 1/19 18:15:42 (pid:2543) Starting add_shadow_birthdate(4567846.0)
> 1/19 18:15:42 (pid:2543) Started shadow for job 4567846.0 on "<10.13.1.121:53621>", (shadow pid = 28162)
> 1/19 18:15:42 (pid:2543) DaemonCore: Command received via UDP from host <10.13.0.12:41919>
> 1/19 18:15:42 (pid:2543) DaemonCore: received command 421 (RESCHEDULE), calling handler (reschedule_negotiator)
> 1/19 18:15:42 (pid:2543) Called reschedule_negotiator()
> 1/19 18:15:42 (pid:2543) Activity on stashed negotiator socket
> 1/19 18:15:42 (pid:2543) Negotiating for owner: inspiralbns@ligo
> 1/19 18:15:42 (pid:2543) Checking consistency running and runnable jobs
> 1/19 18:15:42 (pid:2543) Tables are consistent
> 1/19 18:15:42 (pid:2543) Out of servers - 0 jobs matched, 1 jobs idle, 1 jobs rejected
> 1/19 18:15:42 (pid:2543) Activity on stashed negotiator socket
> 1/19 18:15:42 (pid:2543) Negotiating for owner: inspiralbns@ligo
> 1/19 18:15:42 (pid:2543) Checking consistency running and runnable jobs
> 1/19 18:15:42 (pid:2543) Tables are consistent
> 1/19 18:15:42 (pid:2543) Out of jobs - 1 jobs matched, 0 jobs idle, flock level = 0
> 1/19 18:15:42 (pid:2543) Shadow pid 27721 for job 4567820.0 exited with status 100
> 1/19 18:15:43 (pid:2543) match (<10.13.1.130:59771>#1164737958#9096) out of jobs (cluster id 4567847); relinquishing
> 1/19 18:15:43 (pid:2543) Sent RELEASE_CLAIM to startd on <10.13.1.130:59771>
> 1/19 18:15:43 (pid:2543) Match record (<10.13.1.130:59771>, 4567847, 0) deleted
> 1/19 18:15:43 (pid:2543) Starting add_shadow_birthdate(4567847.0)
> 1/19 18:15:43 (pid:2543) Started shadow for job 4567847.0 on "<10.13.1.60:54490>", (shadow pid = 28164)
> 1/19 18:15:43 (pid:2543) DaemonCore: Command received via TCP from host <10.13.1.130:56651>
> 1/19 18:15:43 (pid:2543) DaemonCore: received command 443 (VACATE_SERVICE), calling handler (vacate_service)
> 1/19 18:15:43 (pid:2543) Got VACATE_SERVICE from <10.13.1.130:56651>
> 1/19 18:15:44 (pid:2543) Sent ad to central manager for kleinewelle@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for kleinewelle@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to central manager for inspiralbns@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for inspiralbns@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to central manager for waveburst_test@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for waveburst_test@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to central manager for pulsar@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for pulsar@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to central manager for hoft@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for hoft@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to central manager for lindy@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for lindy@ligo
> 1/19 18:15:44 (pid:2543) Refusing attempt to add 'Environment' = '"EDITOR=emacs GRID_SECURITY_DIR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/etc MANPATH=/ldcg/condor/man:/opt/lscsoft/lalapps/share/man:/opt/lscsoft/lal/share/man:/opt/lscsoft/glue/man:/opt/lscsoft/libframe/man:/opt/lscsoft/libmetaio/man:/ldcg/condor//man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/man:/ldcg/condor//man:/ldcg/condor/man:/opt/lscsoft/lalapps/share/man:/opt/lscsoft/lal/share/man:/opt/lscsoft/glue/man:/opt/lscsoft/libframe/man:/opt/lscsoft/libmetaio/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/man:/ldcg/condor//man:::/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/
ma!
>  n:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/share/man TERM=screen LAL_PREFIX=/opt/lscsoft/lal SASL_PATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib/sasl GSTAR_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar MYSQL_UNIX_PORT=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt-app-data/mysql/var/mysql.sock LSCSOFT_PREFIX=/opt/lscsoft HOSTNAME=ldas-grid SHELL=/bin/bash MATLABPATH=/ligotools/matlab WBONLINE=/archive/home/waveburst/S5_online VOMS_USERCONF=/ldcg/stow_pkgs
/l!
>  dg-4.3/ldg/vdt/glite/etc EDG_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/
> edg LDG_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg HISTSIZE=1000 GLOBUS_PATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus SSH_CLIENT=130.39.245.165' '37419' '22 PYLAL_LOCATION=/archive/home/ram/opt/pylal GLOBUS_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus X509_CADIR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/TRUSTED_CA X509_CERT_DIR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/TRUSTED_CA PERL5LIB=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkg
s/!
>  ldg-4.3/ldg/vdt/vds/lib/perl:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0/i686-linux-thread-multi::/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar/lib CVSROOT=:pserver:igor__AT__ldas-sw.ligo.caltech.edu:/ldcg_server/common/repository_gds PYTHONPATH=/opt/lscsoft/lalapps/lib/python2.4/site-packages:/opt/lscsoft/glue/lib/python:/opt/lscsoft/libframe/lib/python:/opt/lscsoft/libmetaio/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib/python2.2/site-packages:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib64/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib/python:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib/python2.2/site-packages:/ldcg/sto
w_!
>  pkgs/ldg-4.3/ldg/vdt/globus/lib64/python:/ldcg/stow_pkgs/ldg-4.3/ldg/v
> dt/globus/lib/python:/ldcg/pacman/src:/opt/lscsoft/lalapps/lib/python2.4/site-packages:/opt/lscsoft/glue/lib/python:/opt/lscsoft/libframe/lib/python:/opt/lscsoft/libmetaio/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib/python2.2/site-packages:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib64/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib/python:/ldcg/pacman/src: QTDIR=/usr/lib/qt-3.3 LAL_LOCATION=/opt/lscsoft/lalapps EXTRAS_LOCATION=/archive/home/ram/opt/extras SHLVL=3 SSH_TTY=/dev/pts/0 TZ=America/Chicago GLITE_LOCATION_LOG=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/log VDS_HOME=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds GLOBUS_TCP_PORT_RANGE=40000,45000 USER=waveburst GLUE_LOCATION=/archive/home/ram/opt/glue GLOBUS_ERROR_VERBOSE=true LALAPPS_LOCATION=/archive/home/ram/opt/lalapps LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.c
om!
>  =01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35: LD_LIBRARY_PATH=/opt/lscsoft/lal/lib:/opt/lscsoft/glue/lib:/opt/lscsoft/libframe/lib:/opt/lscsoft/libmetaio/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/myodbc/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/lib/mysql:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/jre/lib/i386:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/jre/lib/i386/server:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/jre/lib/i386/client:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/berkeley-db/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib:/ligotools/lib ROOTSYS=/archive/home/igor/SOFT
1/!
>  root GPT_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt LDG_INSTALL_LOG=
> /ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/etc/ldg-install.log GLITE_LOCATION_TMP=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/tmp TERMCAP=SC|screen|VT' '100/ANSI' 'X3.64' 'virtual' 'terminal:\'
> 	':DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:UP=\E[%dA:bs:bt=\E[Z:\'
> 	':cd=\E[J:ce=\E[K:cl=\E[H\E[J:cm=\E[%i%d;%dH:ct=\E[3g:\'
> 	':do=^J:nd=\E[C:pt:rc=\E8:rs=\Ec:sc=\E7:st=\EH:up=\EM:\'
> 	':le=^H:bl=^G:cr=^M:it#8:ho=\E[H:nw=\EE:ta=^I:is=\E)0:\'
> 	':li#55:co#154:am:xn:xv:LP:sr=\EM:al=\E[L:AL=\E[%dL:\'
> 	':cs=\E[%i%d;%dr:dl=\E[M:DL=\E[%dM:dc=\E[P:DC=\E[%dP:\'
> 	':im=\E[4h:ei=\E[4l:mi:IC=\E[%d@:ks=\E[?1h\E=:\'
> 	':ke=\E[?1l\E>:vi=\E[?25l:ve=\E[34h\E[?25h:vs=\E[34l:\'
> 	':ti=\E[?1049h:te=\E[?1049l:us=\E[4m:ue=\E[24m:so=\E[3m:\'
> 	':se=\E[23m:mb=\E[5m:md=\E[1m:mr=\E[7m:me=\E[m:ms:\'
> 	':Co#8:pa#64:AF=\E[3%dm:AB=\E[4%dm:op=\E[39;49m:AX:\'
> 	':vb=\Eg:G0:as=\E(0:ae=\E(B:\'
> 	':ac=\140\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:\'
> 	':po=\E[5i:pf=\E[4i:Z0=\E[?3h:Z1=\E[?3l:k0=\E[10~:\'
> 	':k1=\EOP:k2=\EOQ:k3=\EOR:k4=\EOS:k5=\E[15~:k6=\E[17~:\'
> 	':k7=\E[18~:k8=\E[19~:k9=\E[20~:k;=\E[21~:F1=\E[23~:\'
> 	':F2=\E[24~:F3=\EO2P:F4=\EO2Q:F5=\EO2R:F6=\EO2S:\'
> 	':F7=\E[15;2~:F8=\E[17;2~:F9=\E[18;2~:FA=\E[19;2~:kb=:\'
> 	':K2=\EOE:kB=\E[Z:*4=\E[3;2~:*7=\E[1;2F:#2=\E[1;2H:\'
> 	':#3=\E[2;2~:#4=\E[1;2D:%c=\E[6;2~:%e=\E[5;2~:%i=\E[1;2C:\'
> 	':kh=\E[1~:@1=\E[1~:kH=\E[4~:@7=\E[4~:kN=\E[6~:kP=\E[5~:\'
> 	':kI=\E[2~:kD=\E[3~:ku=\EOA:kd=\EOB:kr=\EOC:kl=\EOD:km: DAGDBUPDATORLOCKFILE=/etc/onasys-dblockfile PWD=/archive/home/waveburst/COHERENT_ONLINE KDEDIR=/usr LIBPATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib:/ldcg/ldg/vdt/globus/lib:/usr/lib:/lib MAIL=/var/spool/mail/waveburst PATH=/ldcg/condor/bin:/ldcg/condor/sbin:/opt/lscsoft/lalapps/bin:/opt/lscsoft/lal/bin:/opt/lscsoft/glue/bin:/opt/lscsoft/libframe/bin:/opt/lscsoft/libmetaio/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/pyglobus-url-copy/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/ftsh/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/bin:/ldcg/condor//sbin:/ldcg/condor//bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt/sbin:/ldcg/stow_pkgs/ldg-4.3/ld
g/!
>  vdt/globus/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/sbin:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/pyglobus-url-copy/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/ftsh/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/bin:/ldcg/condor//sbin:/ldcg/condor//bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt/sbin:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/bin:/ldcg/condor/bin:/ldcg/condor/sbin:/opt/lscsoft/lalapps/bin:/opt/lscsoft/lal/bin:/opt/lscsoft/glue/bin:/opt/lscsoft/libfram
e/!
>  bin:/opt/lscsoft/libmetaio/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/s
> bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/pyglobus-url-copy/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/ftsh/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/bin:/ldcg/condor//sbin:/ldcg/condor//bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt/sbin:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/bin:/usr/kerberos/bin:/usr/bin:/bin:/usr/sbin:/sbin:/ldcg/ldg/vdt/globus/bin:/usr/X11R6/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar/bin:/ligotools/bin:/ldcg/matlab_r2006a/bin:/archive/home/waveburst/bin:/archive/home/igor/SOFT1/root/bin:/archive/home/waveburst/bin:.:/ldcg/matlab_r2006a/bin SHLIB_PATH=/ldcg/stow_pkgs
/l!
>  dg-4.3/ldg/vdt/globus/lib:/ldcg/ldg/vdt/globus/lib STY=14485.pts-0.ldas-grid _=/ldcg/condor/bin/condor_submit JAVA_HOME=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4 CONDOR_LOCATION=/ldcg/condor CONDOR_CONFIG=/ldcg/condor/etc/condor_config VDT_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt ODBCINI=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/etc/odbc.ini LSC_SEGFIND_SERVER=ldas.ligo-la.caltech.edu LOGNAME=waveburst INPUTRC=/etc/inputrc X509_USER_CERT=/archive/home/waveburst/.certificates/ldas-grid.ligo-la.caltech.edu/waveburstcert.pem LANG=C HOME=/archive/home/waveburst LIGOTOOLS=/ligotools GLITE_LOCATION_VAR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/var X509_USER_PROXY=/tmp/x509up_p10457.file9bvIoT.1 BOSSDIR=/etc X509_USER_KEY=/archive/home/waveburst/.certificates/ldas-grid.ligo-la.caltech.edu/waveburstkey.pem VDS_JAVA_HEAPMAX=1024 DYLD_LIBRARY_PATH=/opt/lscsoft/lal/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib:/opt/lscsoft/lal/lib VDT_INSTALL_LOG=vdt-install.log LSC_DATAFIND_SERVER=ldas.li
go!
>  -la.caltech.edu WINDOW=0 CLASSPATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds
> /lib/commons-pool.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cog-jglobus.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/java-getopt-1.0.9.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cryptix-asn1.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cryptix.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cryptix32.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/exist-optional.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/exist.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/gvds.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/jakarta-oro.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/loggerservice-stub.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/jce-jdk13-117.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/jlinker.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/junit.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/log4j-1.2.8.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/resolver.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/puretls.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/mysql-connector-java-3.0.11-stable-bin.jar:/ld
cg!
>  /stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/postgresql-8.1dev-400.jdbc3.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xercesImpl.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/rls.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xmlParserAPIs.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xmldb.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xmlrpc.jar GLOBUS_MYSQL_PATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql LSC_DATAGRID_SERVER_LOCATION=/ldcg/ldg SSH_CONNECTION=130.39.245.165' '37419' '130.39.245.243' '22 PKG_CONFIG_PATH=/opt/lscsoft/lal/lib/pkgconfig:/opt/lscsoft/glue/lib/pkgconfig:/opt/lscsoft/libframe/lib/pkgconfig:/opt/lscsoft/libmetaio/lib/pkgconfig:/opt/lscsoft/lal/lib/pkgconfig:/opt/lscsoft/glue/lib/pkgconfig:/opt/lscsoft/libframe/lib/pkgconfig:/opt/lscsoft/libmetaio/lib/pkgconfig: LDG_DIRECTORY=/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server LESSOPEN=|/usr/bin/lesspipe.sh' '%s VDT_POSTINSTALL_README=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/post-install/README DISPLAY=localhost:10.0 GLITE_LOCATION=/ldcg/stow_
pk!
>  gs/ldg-4.3/ldg/vdt/glite LDG_SOFTWARE_LOCATION=http://www.ligo.mit.edu
> /ldg4.3/software PACMAN_LOCATION=/ldcg/pacman G_BROKEN_FILENAMES=1"' to record '04567848.-1' as it contains a newline, which is not allowed.
> 1/19 18:15:44 (pid:2543) ERROR "write inside a transaction failed, errno = 0" at line 127 in file log_transaction.C
> 1/19 18:17:42 (pid:28232) ******************************************************
> 1/19 18:17:42 (pid:28232) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 18:17:42 (pid:28232) ** /ldcg/stow_pkgs/condor-6.8.2/condor/sbin/condor_schedd
> 1/19 18:17:42 (pid:28232) ** $CondorVersion: 6.8.2 Oct 12 2006 $
> 1/19 18:17:42 (pid:28232) ** $CondorPlatform: I386-LINUX_RHEL3 $
> 1/19 18:17:42 (pid:28232) ** PID = 28232
> 1/19 18:17:42 (pid:28232) ** Log last touched 1/19 18:15:44
> 1/19 18:17:42 (pid:28232) ******************************************************
> 1/19 18:17:42 (pid:28232) Using config source: /usr1/condor/condor_config
> 1/19 18:17:42 (pid:28232) Using local config sources: 
> 1/19 18:17:42 (pid:28232)    /usr1/condor/condor_config.local
> 1/19 18:17:42 (pid:28232) DaemonCore: Command Socket at <10.13.0.12:33572>
> 1/19 18:17:42 (pid:28232) History file rotation is enabled.
> 1/19 18:17:42 (pid:28232)   Maximum history file size is: 1000000000 bytes
> 1/19 18:17:42 (pid:28232)   Number of rotated history files is: 100
> 1/19 18:17:45 (pid:28232) 4567427.0: JobLeaseDuration remaining: 1040
> 1/19 18:17:47 (pid:28232) Sent ad to central manager for kleinewelle@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for kleinewelle@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to central manager for inspiralbns@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for inspiralbns@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to central manager for waveburst_test@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for waveburst_test@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to central manager for hoft@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for hoft@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to central manager for pulsar@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for pulsar@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to central manager for lindy@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for lindy@ligo
> 1/19 18:17:47 (pid:28232) Starting add_shadow_birthdate(4567427.0)
> 1/19 18:17:47 (pid:28232) Started shadow for job 4567427.0 on "<10.13.1.163:41265>", (shadow pid = 28238)
> 1/19 18:17:47 (pid:28232) Successfully created sched universe process
> 1/19 18:17:47 (pid:28232) Starting add_shadow_birthdate(4343359.0)
> 
> 
> Igor,
> 	All of these restarts appear to be associated with the waveburst
> account. Until the Condor team can explain/fix this please carefully
> consider what you may have recently changed in the configuration of
> this account. The simplest explanation is that you recently added a TERMCAP
> environment variable setting that includes a newline which Condor apparently
> does not allow.
> 
> Thanks.
> 
> 


===========================================================================
Date mail was appended: Mon Jan 22 10:27:18 2007 (1169483238)

Date: Tue, 23 Jan 2007 08:58:23 -0600
From: Greg Quinn <gquinn__AT__cs.wisc.edu>
To: condor-support__AT__cs.wisc.edu
Subject: Re: [condor-support #1816] LIGO: schedd exit stat 4 due to     
 log_transaction  failure

Stuart,

I am wondering if you know what method of job submission is being 
attempted for these jobs that cause the SchedD to crash. The reason I 
ask is that we try to detect early if a job's ClassAd attributes may 
cause problems, and it appears that this is a case we have missed. (Of 
course, we must also fix the problem in the SchedD itself so that a job 
ClassAd with newlines in it can't bring down the SchedD - but we'd like 
to detect problems early as often as we can.)

I can think of a couple scenarios where a newline in an environment 
variable may go unnoticed until its too late:
   1) condor_submit with getenv = true
   2) Condor-C
   3) jobs submitted via our SOAP interface

Could you please let us know which (if any?) of these scenarios is 
ultimately leading to SchedD crashes in your pool?

Thank you,

Greg Quinn
Condor Team

gquinn wrote:
> ===========================================================================
> TICKET INFORMATION
> ===========================================================================
> Ticket Queue: condor-support
> Ticket Number: 1816
> Ticket Creation: Sat Jan 20 17:09:13 2007 (1169334556)
> Ticket Updated: Mon Jan 22 10:03:27 2007 (1169481807)
> Ticket Notification: 
> Ticket Category: user
> Ticket Subject: LIGO: schedd exit stat 4 due to log_transaction failure
> Ticket Type: active
> Ticket User(s): anderson__AT__ligo.caltech.edu
> Ticket Owner: gquinn
> Ticket Status: new
> Ticket Priority: normal
> Ticket ETA: 
> ===========================================================================
> Ticket LOGFILE
> ===========================================================================
> espinoza_e__AT__ligo.caltech.edu,ldas_admin_llo__AT__ligo.caltech.edupublicReceived: from shale.cs.wisc.edu (shale.cs.wisc.edu [128.105.6.25]) 	by
>  chopin.cs.wisc.edu (8.13.6/8.13.6) with ESMTP id l0KN99ax014428 	for
>  <condor-support__AT__chopin.cs.wisc.edu>; Sat, 20 Jan 2007 17:09:09 -0600
> Received: from obsidian.cs.wisc.edu (obsidian.cs.wisc.edu [128.105.6.13])
>  by shale.cs.wisc.edu (8.13.6/8.13.6) with ESMTP id l0KN99Zn016834 	for
>  <condor-support__AT__cs.wisc.edu>; Sat, 20 Jan 2007 17:09:09 -0600
> Received: from acrux.ligo.caltech.edu (acrux.ligo.caltech.edu
>  [131.215.115.14]) 	by obsidian.cs.wisc.edu (8.13.6/8.13.6) with ESMTP id
>  l0KN8v8O005576 	for <condor-support__AT__cs.wisc.edu>; Sat, 20 Jan 2007
>  17:09:02 -0600
> Received: from alphard.ligo.caltech.edu (alphard [131.215.114.160]) 	by
>  acrux.ligo.caltech.edu (8.12.11/8.12.11) with ESMTP id l0KN8tUg001444
>  (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT);
>  Sat, 20 Jan 2007 15:08:55 -0800 (PST)
> Received: from alphard.ligo.caltech.edu (localhost.localdomain
>  [127.0.0.1]) 	by alphard.ligo.caltech.edu (8.13.4/8.13.4) with ESMTP id
>  l0KN8o6W000656; 	Sat, 20 Jan 2007 15:08:50 -0800
> Received: (from anderson@localhost) 	by alphard.ligo.caltech.edu
>  (8.13.4/8.13.4/Submit) id l0KN8ogl000655; 	Sat, 20 Jan 2007 15:08:50 -0800
> Date: Sat, 20 Jan 2007 15:08:50 -0800
> From: Stuart Anderson <anderson__AT__ligo.caltech.edu>
> To: condor-support__AT__cs.wisc.edu
> CC: Erik Espinoza <espinoza_e__AT__ligo.caltech.edu>,
>  ldas_admin_llo__AT__ligo.caltech.edu
> Subject: LIGO: schedd exit stat 4 due to log_transaction failure
> Message-ID: <20070120230850.GA32685__AT__ligo.caltech.edu>
> MIME-Version: 1.0
> Content-Type: text/plain; charset=us-ascii
> Content-Disposition: inline
> User-Agent: Mutt/1.4.2.1i
> X-Spam-Score: undef - Domain Whitelisted (ligo.caltech.edu: )
> X-Canit-Stats-ID: 5852267 - 92b548b85f8e
> X-Scanned-BY: CanIt (www . roaringpenguin . com) on 131.215.115.14
> X-CSL-Mailscanner-Information: Please contact lab__AT__cs.wisc.edu for more  
>  information
> X-CSL-Mailscanner: Found to be clean
> Content-Transfer-Encoding: 8bit
> X-MIME-Autoconverted: from quoted-printable to 8bit by chopin.cs.wisc.edu 
>  id l0KN99ax014428
> 
> The LIGO LLO Condor pool running:
> 
> $ condor_version
> $CondorVersion: 6.8.2 Oct 12 2006 $
> $CondorPlatform: I386-LINUX_RHEL3 $
> 
> has recently had several restarts of condor_schedd due to condor_schedd exiting
> with status 4 and reporting:
> 
> ...
> /vds/lib/puretls.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/mysql-connector-jav
> a-3.0.11-stable-bin.jar:/ldcg/stow_pkg
>  .mit.edu
> /ldg4.3/software PACMAN_LOCATION=/ldcg/pacman G_BROKEN_FILENAMES=1"' to record
> '04569965.-1' as it contains a newline, which is not allowed.
> 1/19 21:11:59 (pid:3944) ERROR "write inside a transaction failed, errno = 0" at
> line 127 in file log_transaction.C
> 
> These appear to happen in bursts, i.e.,
> 
> $ grep STARTING SchedLog
> 1/19 18:17:42 (pid:28232) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 18:56:57 (pid:3944) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 21:13:03 (pid:28001) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 21:32:09 (pid:32438) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 21:33:23 (pid:333) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 21:34:08 (pid:410) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 21:35:12 (pid:477) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 21:37:41 (pid:981) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 21:43:33 (pid:2357) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/20 12:31:13 (pid:2685) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 
> 
> Here is a section of the SchedLog file before one of these restarts:
> 
> 1/19 18:15:32 (pid:2543) DaemonCore: Command received via UDP from host <10.13.0.12:41912>
> 1/19 18:15:32 (pid:2543) DaemonCore: received command 421 (RESCHEDULE), calling handler (reschedule_negotiator)
> 1/19 18:15:32 (pid:2543) Called reschedule_negotiator()
> 1/19 18:15:39 (pid:2543) Sent ad to central manager for kleinewelle@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for kleinewelle@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to central manager for inspiralbns@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for inspiralbns@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to central manager for waveburst_test@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for waveburst_test@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to central manager for pulsar@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for pulsar@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to central manager for hoft@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for hoft@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to central manager for lindy@ligo
> 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for lindy@ligo
> 1/19 18:15:39 (pid:2543) DaemonCore: Command received via UDP from host <10.13.0.12:41919>
> 1/19 18:15:39 (pid:2543) DaemonCore: received command 421 (RESCHEDULE), calling handler (reschedule_negotiator)
> 1/19 18:15:39 (pid:2543) Called reschedule_negotiator()
> 1/19 18:15:39 (pid:2543) Shadow pid 27707 for job 4567817.0 exited with status 100
> 1/19 18:15:41 (pid:2543) Shadow pid 27709 for job 4567818.0 exited with status 100
> 1/19 18:15:41 (pid:2543) Starting add_shadow_birthdate(4567844.0)
> 1/19 18:15:41 (pid:2543) Started shadow for job 4567844.0 on "<10.13.1.37:52609>", (shadow pid = 28158)
> 1/19 18:15:41 (pid:2543) Starting add_shadow_birthdate(4567845.0)
> 1/19 18:15:41 (pid:2543) Started shadow for job 4567845.0 on "<10.13.1.58:54972>", (shadow pid = 28160)
> 1/19 18:15:41 (pid:2543) Shadow pid 27713 for job 4567819.0 exited with status 100
> 1/19 18:15:42 (pid:2543) Starting add_shadow_birthdate(4567846.0)
> 1/19 18:15:42 (pid:2543) Started shadow for job 4567846.0 on "<10.13.1.121:53621>", (shadow pid = 28162)
> 1/19 18:15:42 (pid:2543) DaemonCore: Command received via UDP from host <10.13.0.12:41919>
> 1/19 18:15:42 (pid:2543) DaemonCore: received command 421 (RESCHEDULE), calling handler (reschedule_negotiator)
> 1/19 18:15:42 (pid:2543) Called reschedule_negotiator()
> 1/19 18:15:42 (pid:2543) Activity on stashed negotiator socket
> 1/19 18:15:42 (pid:2543) Negotiating for owner: inspiralbns@ligo
> 1/19 18:15:42 (pid:2543) Checking consistency running and runnable jobs
> 1/19 18:15:42 (pid:2543) Tables are consistent
> 1/19 18:15:42 (pid:2543) Out of servers - 0 jobs matched, 1 jobs idle, 1 jobs rejected
> 1/19 18:15:42 (pid:2543) Activity on stashed negotiator socket
> 1/19 18:15:42 (pid:2543) Negotiating for owner: inspiralbns@ligo
> 1/19 18:15:42 (pid:2543) Checking consistency running and runnable jobs
> 1/19 18:15:42 (pid:2543) Tables are consistent
> 1/19 18:15:42 (pid:2543) Out of jobs - 1 jobs matched, 0 jobs idle, flock level = 0
> 1/19 18:15:42 (pid:2543) Shadow pid 27721 for job 4567820.0 exited with status 100
> 1/19 18:15:43 (pid:2543) match (<10.13.1.130:59771>#1164737958#9096) out of jobs (cluster id 4567847); relinquishing
> 1/19 18:15:43 (pid:2543) Sent RELEASE_CLAIM to startd on <10.13.1.130:59771>
> 1/19 18:15:43 (pid:2543) Match record (<10.13.1.130:59771>, 4567847, 0) deleted
> 1/19 18:15:43 (pid:2543) Starting add_shadow_birthdate(4567847.0)
> 1/19 18:15:43 (pid:2543) Started shadow for job 4567847.0 on "<10.13.1.60:54490>", (shadow pid = 28164)
> 1/19 18:15:43 (pid:2543) DaemonCore: Command received via TCP from host <10.13.1.130:56651>
> 1/19 18:15:43 (pid:2543) DaemonCore: received command 443 (VACATE_SERVICE), calling handler (vacate_service)
> 1/19 18:15:43 (pid:2543) Got VACATE_SERVICE from <10.13.1.130:56651>
> 1/19 18:15:44 (pid:2543) Sent ad to central manager for kleinewelle@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for kleinewelle@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to central manager for inspiralbns@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for inspiralbns@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to central manager for waveburst_test@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for waveburst_test@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to central manager for pulsar@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for pulsar@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to central manager for hoft@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for hoft@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to central manager for lindy@ligo
> 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for lindy@ligo
> 1/19 18:15:44 (pid:2543) Refusing attempt to add 'Environment' = '"EDITOR=emacs GRID_SECURITY_DIR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/etc MANPATH=/ldcg/condor/man:/opt/lscsoft/lalapps/share/man:/opt/lscsoft/lal/share/man:/opt/lscsoft/glue/man:/opt/lscsoft/libframe/man:/opt/lscsoft/libmetaio/man:/ldcg/condor//man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/man:/ldcg/condor//man:/ldcg/condor/man:/opt/lscsoft/lalapps/share/man:/opt/lscsoft/lal/share/man:/opt/lscsoft/glue/man:/opt/lscsoft/libframe/man:/opt/lscsoft/libmetaio/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/man:/ldcg/condor//man:::/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/
ma!
>  n:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/share/man TERM=screen LAL_PREFIX=/opt/lscsoft/lal SASL_PATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib/sasl GSTAR_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar MYSQL_UNIX_PORT=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt-app-data/mysql/var/mysql.sock LSCSOFT_PREFIX=/opt/lscsoft HOSTNAME=ldas-grid SHELL=/bin/bash MATLABPATH=/ligotools/matlab WBONLINE=/archive/home/waveburst/S5_online VOMS_USERCONF=/ldcg/stow_pkgs
/l!
>  dg-4.3/ldg/vdt/glite/etc EDG_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/
> edg LDG_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg HISTSIZE=1000 GLOBUS_PATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus SSH_CLIENT=130.39.245.165' '37419' '22 PYLAL_LOCATION=/archive/home/ram/opt/pylal GLOBUS_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus X509_CADIR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/TRUSTED_CA X509_CERT_DIR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/TRUSTED_CA PERL5LIB=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkg
s/!
>  ldg-4.3/ldg/vdt/vds/lib/perl:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0/i686-linux-thread-multi::/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar/lib CVSROOT=:pserver:igor__AT__ldas-sw.ligo.caltech.edu:/ldcg_server/common/repository_gds PYTHONPATH=/opt/lscsoft/lalapps/lib/python2.4/site-packages:/opt/lscsoft/glue/lib/python:/opt/lscsoft/libframe/lib/python:/opt/lscsoft/libmetaio/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib/python2.2/site-packages:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib64/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib/python:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib/python2.2/site-packages:/ldcg/sto
w_!
>  pkgs/ldg-4.3/ldg/vdt/globus/lib64/python:/ldcg/stow_pkgs/ldg-4.3/ldg/v
> dt/globus/lib/python:/ldcg/pacman/src:/opt/lscsoft/lalapps/lib/python2.4/site-packages:/opt/lscsoft/glue/lib/python:/opt/lscsoft/libframe/lib/python:/opt/lscsoft/libmetaio/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib/python2.2/site-packages:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib64/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib/python:/ldcg/pacman/src: QTDIR=/usr/lib/qt-3.3 LAL_LOCATION=/opt/lscsoft/lalapps EXTRAS_LOCATION=/archive/home/ram/opt/extras SHLVL=3 SSH_TTY=/dev/pts/0 TZ=America/Chicago GLITE_LOCATION_LOG=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/log VDS_HOME=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds GLOBUS_TCP_PORT_RANGE=40000,45000 USER=waveburst GLUE_LOCATION=/archive/home/ram/opt/glue GLOBUS_ERROR_VERBOSE=true LALAPPS_LOCATION=/archive/home/ram/opt/lalapps LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.c
om!
>  =01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35: LD_LIBRARY_PATH=/opt/lscsoft/lal/lib:/opt/lscsoft/glue/lib:/opt/lscsoft/libframe/lib:/opt/lscsoft/libmetaio/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/myodbc/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/lib/mysql:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/jre/lib/i386:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/jre/lib/i386/server:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/jre/lib/i386/client:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/berkeley-db/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib:/ligotools/lib ROOTSYS=/archive/home/igor/SOFT
1/!
>  root GPT_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt LDG_INSTALL_LOG=
> /ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/etc/ldg-install.log GLITE_LOCATION_TMP=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/tmp TERMCAP=SC|screen|VT' '100/ANSI' 'X3.64' 'virtual' 'terminal:\'
> 	':DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:UP=\E[%dA:bs:bt=\E[Z:\'
> 	':cd=\E[J:ce=\E[K:cl=\E[H\E[J:cm=\E[%i%d;%dH:ct=\E[3g:\'
> 	':do=^J:nd=\E[C:pt:rc=\E8:rs=\Ec:sc=\E7:st=\EH:up=\EM:\'
> 	':le=^H:bl=^G:cr=^M:it#8:ho=\E[H:nw=\EE:ta=^I:is=\E)0:\'
> 	':li#55:co#154:am:xn:xv:LP:sr=\EM:al=\E[L:AL=\E[%dL:\'
> 	':cs=\E[%i%d;%dr:dl=\E[M:DL=\E[%dM:dc=\E[P:DC=\E[%dP:\'
> 	':im=\E[4h:ei=\E[4l:mi:IC=\E[%d@:ks=\E[?1h\E=:\'
> 	':ke=\E[?1l\E>:vi=\E[?25l:ve=\E[34h\E[?25h:vs=\E[34l:\'
> 	':ti=\E[?1049h:te=\E[?1049l:us=\E[4m:ue=\E[24m:so=\E[3m:\'
> 	':se=\E[23m:mb=\E[5m:md=\E[1m:mr=\E[7m:me=\E[m:ms:\'
> 	':Co#8:pa#64:AF=\E[3%dm:AB=\E[4%dm:op=\E[39;49m:AX:\'
> 	':vb=\Eg:G0:as=\E(0:ae=\E(B:\'
> 	':ac=\140\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:\'
> 	':po=\E[5i:pf=\E[4i:Z0=\E[?3h:Z1=\E[?3l:k0=\E[10~:\'
> 	':k1=\EOP:k2=\EOQ:k3=\EOR:k4=\EOS:k5=\E[15~:k6=\E[17~:\'
> 	':k7=\E[18~:k8=\E[19~:k9=\E[20~:k;=\E[21~:F1=\E[23~:\'
> 	':F2=\E[24~:F3=\EO2P:F4=\EO2Q:F5=\EO2R:F6=\EO2S:\'
> 	':F7=\E[15;2~:F8=\E[17;2~:F9=\E[18;2~:FA=\E[19;2~:kb=:\'
> 	':K2=\EOE:kB=\E[Z:*4=\E[3;2~:*7=\E[1;2F:#2=\E[1;2H:\'
> 	':#3=\E[2;2~:#4=\E[1;2D:%c=\E[6;2~:%e=\E[5;2~:%i=\E[1;2C:\'
> 	':kh=\E[1~:@1=\E[1~:kH=\E[4~:@7=\E[4~:kN=\E[6~:kP=\E[5~:\'
> 	':kI=\E[2~:kD=\E[3~:ku=\EOA:kd=\EOB:kr=\EOC:kl=\EOD:km: DAGDBUPDATORLOCKFILE=/etc/onasys-dblockfile PWD=/archive/home/waveburst/COHERENT_ONLINE KDEDIR=/usr LIBPATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib:/ldcg/ldg/vdt/globus/lib:/usr/lib:/lib MAIL=/var/spool/mail/waveburst PATH=/ldcg/condor/bin:/ldcg/condor/sbin:/opt/lscsoft/lalapps/bin:/opt/lscsoft/lal/bin:/opt/lscsoft/glue/bin:/opt/lscsoft/libframe/bin:/opt/lscsoft/libmetaio/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/pyglobus-url-copy/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/ftsh/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/bin:/ldcg/condor//sbin:/ldcg/condor//bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt/sbin:/ldcg/stow_pkgs/ldg-4.3/ld
g/!
>  vdt/globus/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/sbin:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/pyglobus-url-copy/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/ftsh/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/bin:/ldcg/condor//sbin:/ldcg/condor//bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt/sbin:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/bin:/ldcg/condor/bin:/ldcg/condor/sbin:/opt/lscsoft/lalapps/bin:/opt/lscsoft/lal/bin:/opt/lscsoft/glue/bin:/opt/lscsoft/libfram
e/!
>  bin:/opt/lscsoft/libmetaio/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/s
> bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/pyglobus-url-copy/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/ftsh/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/bin:/ldcg/condor//sbin:/ldcg/condor//bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt/sbin:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/bin:/usr/kerberos/bin:/usr/bin:/bin:/usr/sbin:/sbin:/ldcg/ldg/vdt/globus/bin:/usr/X11R6/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar/bin:/ligotools/bin:/ldcg/matlab_r2006a/bin:/archive/home/waveburst/bin:/archive/home/igor/SOFT1/root/bin:/archive/home/waveburst/bin:.:/ldcg/matlab_r2006a/bin SHLIB_PATH=/ldcg/stow_pkgs
/l!
>  dg-4.3/ldg/vdt/globus/lib:/ldcg/ldg/vdt/globus/lib STY=14485.pts-0.ldas-grid _=/ldcg/condor/bin/condor_submit JAVA_HOME=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4 CONDOR_LOCATION=/ldcg/condor CONDOR_CONFIG=/ldcg/condor/etc/condor_config VDT_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt ODBCINI=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/etc/odbc.ini LSC_SEGFIND_SERVER=ldas.ligo-la.caltech.edu LOGNAME=waveburst INPUTRC=/etc/inputrc X509_USER_CERT=/archive/home/waveburst/.certificates/ldas-grid.ligo-la.caltech.edu/waveburstcert.pem LANG=C HOME=/archive/home/waveburst LIGOTOOLS=/ligotools GLITE_LOCATION_VAR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/var X509_USER_PROXY=/tmp/x509up_p10457.file9bvIoT.1 BOSSDIR=/etc X509_USER_KEY=/archive/home/waveburst/.certificates/ldas-grid.ligo-la.caltech.edu/waveburstkey.pem VDS_JAVA_HEAPMAX=1024 DYLD_LIBRARY_PATH=/opt/lscsoft/lal/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib:/opt/lscsoft/lal/lib VDT_INSTALL_LOG=vdt-install.log LSC_DATAFIND_SERVER=ldas.li
go!
>  -la.caltech.edu WINDOW=0 CLASSPATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds
> /lib/commons-pool.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cog-jglobus.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/java-getopt-1.0.9.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cryptix-asn1.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cryptix.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cryptix32.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/exist-optional.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/exist.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/gvds.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/jakarta-oro.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/loggerservice-stub.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/jce-jdk13-117.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/jlinker.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/junit.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/log4j-1.2.8.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/resolver.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/puretls.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/mysql-connector-java-3.0.11-stable-bin.jar:/ld
cg!
>  /stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/postgresql-8.1dev-400.jdbc3.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xercesImpl.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/rls.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xmlParserAPIs.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xmldb.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xmlrpc.jar GLOBUS_MYSQL_PATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql LSC_DATAGRID_SERVER_LOCATION=/ldcg/ldg SSH_CONNECTION=130.39.245.165' '37419' '130.39.245.243' '22 PKG_CONFIG_PATH=/opt/lscsoft/lal/lib/pkgconfig:/opt/lscsoft/glue/lib/pkgconfig:/opt/lscsoft/libframe/lib/pkgconfig:/opt/lscsoft/libmetaio/lib/pkgconfig:/opt/lscsoft/lal/lib/pkgconfig:/opt/lscsoft/glue/lib/pkgconfig:/opt/lscsoft/libframe/lib/pkgconfig:/opt/lscsoft/libmetaio/lib/pkgconfig: LDG_DIRECTORY=/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server LESSOPEN=|/usr/bin/lesspipe.sh' '%s VDT_POSTINSTALL_README=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/post-install/README DISPLAY=localhost:10.0 GLITE_LOCATION=/ldcg/stow_
pk!
>  gs/ldg-4.3/ldg/vdt/glite LDG_SOFTWARE_LOCATION=http://www.ligo.mit.edu
> /ldg4.3/software PACMAN_LOCATION=/ldcg/pacman G_BROKEN_FILENAMES=1"' to record '04567848.-1' as it contains a newline, which is not allowed.
> 1/19 18:15:44 (pid:2543) ERROR "write inside a transaction failed, errno = 0" at line 127 in file log_transaction.C
> 1/19 18:17:42 (pid:28232) ******************************************************
> 1/19 18:17:42 (pid:28232) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> 1/19 18:17:42 (pid:28232) ** /ldcg/stow_pkgs/condor-6.8.2/condor/sbin/condor_schedd
> 1/19 18:17:42 (pid:28232) ** $CondorVersion: 6.8.2 Oct 12 2006 $
> 1/19 18:17:42 (pid:28232) ** $CondorPlatform: I386-LINUX_RHEL3 $
> 1/19 18:17:42 (pid:28232) ** PID = 28232
> 1/19 18:17:42 (pid:28232) ** Log last touched 1/19 18:15:44
> 1/19 18:17:42 (pid:28232) ******************************************************
> 1/19 18:17:42 (pid:28232) Using config source: /usr1/condor/condor_config
> 1/19 18:17:42 (pid:28232) Using local config sources: 
> 1/19 18:17:42 (pid:28232)    /usr1/condor/condor_config.local
> 1/19 18:17:42 (pid:28232) DaemonCore: Command Socket at <10.13.0.12:33572>
> 1/19 18:17:42 (pid:28232) History file rotation is enabled.
> 1/19 18:17:42 (pid:28232)   Maximum history file size is: 1000000000 bytes
> 1/19 18:17:42 (pid:28232)   Number of rotated history files is: 100
> 1/19 18:17:45 (pid:28232) 4567427.0: JobLeaseDuration remaining: 1040
> 1/19 18:17:47 (pid:28232) Sent ad to central manager for kleinewelle@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for kleinewelle@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to central manager for inspiralbns@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for inspiralbns@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to central manager for waveburst_test@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for waveburst_test@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to central manager for hoft@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for hoft@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to central manager for pulsar@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for pulsar@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to central manager for lindy@ligo
> 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for lindy@ligo
> 1/19 18:17:47 (pid:28232) Starting add_shadow_birthdate(4567427.0)
> 1/19 18:17:47 (pid:28232) Started shadow for job 4567427.0 on "<10.13.1.163:41265>", (shadow pid = 28238)
> 1/19 18:17:47 (pid:28232) Successfully created sched universe process
> 1/19 18:17:47 (pid:28232) Starting add_shadow_birthdate(4343359.0)
> 
> 
> Igor,
> 	All of these restarts appear to be associated with the waveburst
> account. Until the Condor team can explain/fix this please carefully
> consider what you may have recently changed in the configuration of
> this account. The simplest explanation is that you recently added a TERMCAP
> environment variable setting that includes a newline which Condor apparently
> does not allow.
> 
> Thanks.
> 
> 


===========================================================================
Date mail was appended: Tue Jan 23  8:58:29 2007 (1169564309)

Date: Tue, 23 Jan 2007 10:04:19 -0600
From: Igor Yakushin <igor__AT__ligo-la.caltech.edu>
Subject: Re: [condor-support #1816] LIGO: schedd exit stat 4 due to   
 log_transaction failure
To: condor-support__AT__cs.wisc.edu
CC: anderson__AT__ligo.caltech.edu, espinoza_e__AT__ligo.caltech.edu,
 ldas_admin_llo__AT__ligo.caltech.edu
X-Enigmail-Version: 0.91.0.0

Greg,

>
>I can think of a couple scenarios where a newline in an environment 
>variable may go unnoticed until its too late:
>   1) condor_submit with getenv = true
>  
>


condor_submit with getenv=true from inside 'screen' program.


>   2) Condor-C
>   3) jobs submitted via our SOAP interface
>
>Could you please let us know which (if any?) of these scenarios is 
>ultimately leading to SchedD crashes in your pool?
>
>Thank you,
>
>Greg Quinn
>Condor Team
>  
>
-- 
Igor Yakushin
LIGO at Livingston, LA
http://www.ligo.caltech.edu
(225)686-3170


===========================================================================
Date mail was appended: Tue Jan 23 10:04:40 2007 (1169568281)

Date: Fri, 6 Apr 2007 12:45:41 -0700
From: Stuart Anderson <anderson__AT__ligo.caltech.edu>
To: condor-support response tracking system <condor-support__AT__cs.wisc.edu>
CC: espinoza_e__AT__ligo.caltech.edu, ldas_admin_llo__AT__ligo.caltech.edu
Subject: Re: [condor-support #1816] LIGO: schedd exit stat 4 due to  
 log_transaction failure
X-MIME-Autoconverted: from quoted-printable to 8bit by chopin.cs.wisc.edu 
 id l36Jk6Fd030862

condor-6.8.4 no longer crashes under these circumstances, but rather appears
to just drop the offending environment variable.

Please close this ticket.

Thanks.

On Mon, Jan 22, 2007 at 10:27:18AM -0600, condor-support response tracking system wrote:
> Stuart,
> 
> Indeed, the newlines in a job's environment are causing the SchedD to 
> EXCEPT. I have been able to reproduce this problem locally, and we are 
> working on a fix. Meanwhile, I think the only way to avoid this 
> problem's continuing occurrence is to modify the offending jobs so they 
> don't have newlines in any ClassAd attributes, or to keep them out of 
> the queue.
> 
> Greg Quinn
> Condor Team
> 
> gquinn wrote:
> > ===========================================================================
> > TICKET INFORMATION
> > ===========================================================================
> > Ticket Queue: condor-support
> > Ticket Number: 1816
> > Ticket Creation: Sat Jan 20 17:09:13 2007 (1169334556)
> > Ticket Updated: Mon Jan 22 10:03:27 2007 (1169481807)
> > Ticket Notification: 
> > Ticket Category: user
> > Ticket Subject: LIGO: schedd exit stat 4 due to log_transaction failure
> > Ticket Type: active
> > Ticket User(s): anderson__AT__ligo.caltech.edu
> > Ticket Owner: gquinn
> > Ticket Status: new
> > Ticket Priority: normal
> > Ticket ETA: 
> > ===========================================================================
> > Ticket LOGFILE
> > ===========================================================================
> > espinoza_e__AT__ligo.caltech.edu,ldas_admin_llo__AT__ligo.caltech.edupublicReceived: from shale.cs.wisc.edu (shale.cs.wisc.edu [128.105.6.25]) 	by
> >  chopin.cs.wisc.edu (8.13.6/8.13.6) with ESMTP id l0KN99ax014428 	for
> >  <condor-support__AT__chopin.cs.wisc.edu>; Sat, 20 Jan 2007 17:09:09 -0600
> > Received: from obsidian.cs.wisc.edu (obsidian.cs.wisc.edu [128.105.6.13])
> >  by shale.cs.wisc.edu (8.13.6/8.13.6) with ESMTP id l0KN99Zn016834 	for
> >  <condor-support__AT__cs.wisc.edu>; Sat, 20 Jan 2007 17:09:09 -0600
> > Received: from acrux.ligo.caltech.edu (acrux.ligo.caltech.edu
> >  [131.215.115.14]) 	by obsidian.cs.wisc.edu (8.13.6/8.13.6) with ESMTP id
> >  l0KN8v8O005576 	for <condor-support__AT__cs.wisc.edu>; Sat, 20 Jan 2007
> >  17:09:02 -0600
> > Received: from alphard.ligo.caltech.edu (alphard [131.215.114.160]) 	by
> >  acrux.ligo.caltech.edu (8.12.11/8.12.11) with ESMTP id l0KN8tUg001444
> >  (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT);
> >  Sat, 20 Jan 2007 15:08:55 -0800 (PST)
> > Received: from alphard.ligo.caltech.edu (localhost.localdomain
> >  [127.0.0.1]) 	by alphard.ligo.caltech.edu (8.13.4/8.13.4) with ESMTP id
> >  l0KN8o6W000656; 	Sat, 20 Jan 2007 15:08:50 -0800
> > Received: (from anderson@localhost) 	by alphard.ligo.caltech.edu
> >  (8.13.4/8.13.4/Submit) id l0KN8ogl000655; 	Sat, 20 Jan 2007 15:08:50 -0800
> > Date: Sat, 20 Jan 2007 15:08:50 -0800
> > From: Stuart Anderson <anderson__AT__ligo.caltech.edu>
> > To: condor-support__AT__cs.wisc.edu
> > CC: Erik Espinoza <espinoza_e__AT__ligo.caltech.edu>,
> >  ldas_admin_llo__AT__ligo.caltech.edu
> > Subject: LIGO: schedd exit stat 4 due to log_transaction failure
> > Message-ID: <20070120230850.GA32685__AT__ligo.caltech.edu>
> > MIME-Version: 1.0
> > Content-Type: text/plain; charset=us-ascii
> > Content-Disposition: inline
> > User-Agent: Mutt/1.4.2.1i
> > X-Spam-Score: undef - Domain Whitelisted (ligo.caltech.edu: )
> > X-Canit-Stats-ID: 5852267 - 92b548b85f8e
> > X-Scanned-BY: CanIt (www . roaringpenguin . com) on 131.215.115.14
> > X-CSL-Mailscanner-Information: Please contact lab__AT__cs.wisc.edu for more  
> >  information
> > X-CSL-Mailscanner: Found to be clean
> > Content-Transfer-Encoding: 8bit
> > X-MIME-Autoconverted: from quoted-printable to 8bit by chopin.cs.wisc.edu 
> >  id l0KN99ax014428
> > 
> > The LIGO LLO Condor pool running:
> > 
> > $ condor_version
> > $CondorVersion: 6.8.2 Oct 12 2006 $
> > $CondorPlatform: I386-LINUX_RHEL3 $
> > 
> > has recently had several restarts of condor_schedd due to condor_schedd exiting
> > with status 4 and reporting:
> > 
> > ...
> > /vds/lib/puretls.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/mysql-connector-jav
> > a-3.0.11-stable-bin.jar:/ldcg/stow_pkg
> >  .mit.edu
> > /ldg4.3/software PACMAN_LOCATION=/ldcg/pacman G_BROKEN_FILENAMES=1"' to record
> > '04569965.-1' as it contains a newline, which is not allowed.
> > 1/19 21:11:59 (pid:3944) ERROR "write inside a transaction failed, errno = 0" at
> > line 127 in file log_transaction.C
> > 
> > These appear to happen in bursts, i.e.,
> > 
> > $ grep STARTING SchedLog
> > 1/19 18:17:42 (pid:28232) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> > 1/19 18:56:57 (pid:3944) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> > 1/19 21:13:03 (pid:28001) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> > 1/19 21:32:09 (pid:32438) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> > 1/19 21:33:23 (pid:333) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> > 1/19 21:34:08 (pid:410) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> > 1/19 21:35:12 (pid:477) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> > 1/19 21:37:41 (pid:981) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> > 1/19 21:43:33 (pid:2357) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> > 1/20 12:31:13 (pid:2685) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> > 
> > 
> > Here is a section of the SchedLog file before one of these restarts:
> > 
> > 1/19 18:15:32 (pid:2543) DaemonCore: Command received via UDP from host <10.13.0.12:41912>
> > 1/19 18:15:32 (pid:2543) DaemonCore: received command 421 (RESCHEDULE), calling handler (reschedule_negotiator)
> > 1/19 18:15:32 (pid:2543) Called reschedule_negotiator()
> > 1/19 18:15:39 (pid:2543) Sent ad to central manager for kleinewelle@ligo
> > 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for kleinewelle@ligo
> > 1/19 18:15:39 (pid:2543) Sent ad to central manager for inspiralbns@ligo
> > 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for inspiralbns@ligo
> > 1/19 18:15:39 (pid:2543) Sent ad to central manager for waveburst_test@ligo
> > 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for waveburst_test@ligo
> > 1/19 18:15:39 (pid:2543) Sent ad to central manager for pulsar@ligo
> > 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for pulsar@ligo
> > 1/19 18:15:39 (pid:2543) Sent ad to central manager for hoft@ligo
> > 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for hoft@ligo
> > 1/19 18:15:39 (pid:2543) Sent ad to central manager for lindy@ligo
> > 1/19 18:15:39 (pid:2543) Sent ad to 1 collectors for lindy@ligo
> > 1/19 18:15:39 (pid:2543) DaemonCore: Command received via UDP from host <10.13.0.12:41919>
> > 1/19 18:15:39 (pid:2543) DaemonCore: received command 421 (RESCHEDULE), calling handler (reschedule_negotiator)
> > 1/19 18:15:39 (pid:2543) Called reschedule_negotiator()
> > 1/19 18:15:39 (pid:2543) Shadow pid 27707 for job 4567817.0 exited with status 100
> > 1/19 18:15:41 (pid:2543) Shadow pid 27709 for job 4567818.0 exited with status 100
> > 1/19 18:15:41 (pid:2543) Starting add_shadow_birthdate(4567844.0)
> > 1/19 18:15:41 (pid:2543) Started shadow for job 4567844.0 on "<10.13.1.37:52609>", (shadow pid = 28158)
> > 1/19 18:15:41 (pid:2543) Starting add_shadow_birthdate(4567845.0)
> > 1/19 18:15:41 (pid:2543) Started shadow for job 4567845.0 on "<10.13.1.58:54972>", (shadow pid = 28160)
> > 1/19 18:15:41 (pid:2543) Shadow pid 27713 for job 4567819.0 exited with status 100
> > 1/19 18:15:42 (pid:2543) Starting add_shadow_birthdate(4567846.0)
> > 1/19 18:15:42 (pid:2543) Started shadow for job 4567846.0 on "<10.13.1.121:53621>", (shadow pid = 28162)
> > 1/19 18:15:42 (pid:2543) DaemonCore: Command received via UDP from host <10.13.0.12:41919>
> > 1/19 18:15:42 (pid:2543) DaemonCore: received command 421 (RESCHEDULE), calling handler (reschedule_negotiator)
> > 1/19 18:15:42 (pid:2543) Called reschedule_negotiator()
> > 1/19 18:15:42 (pid:2543) Activity on stashed negotiator socket
> > 1/19 18:15:42 (pid:2543) Negotiating for owner: inspiralbns@ligo
> > 1/19 18:15:42 (pid:2543) Checking consistency running and runnable jobs
> > 1/19 18:15:42 (pid:2543) Tables are consistent
> > 1/19 18:15:42 (pid:2543) Out of servers - 0 jobs matched, 1 jobs idle, 1 jobs rejected
> > 1/19 18:15:42 (pid:2543) Activity on stashed negotiator socket
> > 1/19 18:15:42 (pid:2543) Negotiating for owner: inspiralbns@ligo
> > 1/19 18:15:42 (pid:2543) Checking consistency running and runnable jobs
> > 1/19 18:15:42 (pid:2543) Tables are consistent
> > 1/19 18:15:42 (pid:2543) Out of jobs - 1 jobs matched, 0 jobs idle, flock level = 0
> > 1/19 18:15:42 (pid:2543) Shadow pid 27721 for job 4567820.0 exited with status 100
> > 1/19 18:15:43 (pid:2543) match (<10.13.1.130:59771>#1164737958#9096) out of jobs (cluster id 4567847); relinquishing
> > 1/19 18:15:43 (pid:2543) Sent RELEASE_CLAIM to startd on <10.13.1.130:59771>
> > 1/19 18:15:43 (pid:2543) Match record (<10.13.1.130:59771>, 4567847, 0) deleted
> > 1/19 18:15:43 (pid:2543) Starting add_shadow_birthdate(4567847.0)
> > 1/19 18:15:43 (pid:2543) Started shadow for job 4567847.0 on "<10.13.1.60:54490>", (shadow pid = 28164)
> > 1/19 18:15:43 (pid:2543) DaemonCore: Command received via TCP from host <10.13.1.130:56651>
> > 1/19 18:15:43 (pid:2543) DaemonCore: received command 443 (VACATE_SERVICE), calling handler (vacate_service)
> > 1/19 18:15:43 (pid:2543) Got VACATE_SERVICE from <10.13.1.130:56651>
> > 1/19 18:15:44 (pid:2543) Sent ad to central manager for kleinewelle@ligo
> > 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for kleinewelle@ligo
> > 1/19 18:15:44 (pid:2543) Sent ad to central manager for inspiralbns@ligo
> > 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for inspiralbns@ligo
> > 1/19 18:15:44 (pid:2543) Sent ad to central manager for waveburst_test@ligo
> > 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for waveburst_test@ligo
> > 1/19 18:15:44 (pid:2543) Sent ad to central manager for pulsar@ligo
> > 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for pulsar@ligo
> > 1/19 18:15:44 (pid:2543) Sent ad to central manager for hoft@ligo
> > 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for hoft@ligo
> > 1/19 18:15:44 (pid:2543) Sent ad to central manager for lindy@ligo
> > 1/19 18:15:44 (pid:2543) Sent ad to 1 collectors for lindy@ligo
> > 1/19 18:15:44 (pid:2543) Refusing attempt to add 'Environment' = '"EDITOR=emacs GRID_SECURITY_DIR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/etc MANPATH=/ldcg/condor/man:/opt/lscsoft/lalapps/share/man:/opt/lscsoft/lal/share/man:/opt/lscsoft/glue/man:/opt/lscsoft/libframe/man:/opt/lscsoft/libmetaio/man:/ldcg/condor//man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/man:/ldcg/condor//man:/ldcg/condor/man:/opt/lscsoft/lalapps/share/man:/opt/lscsoft/lal/share/man:/opt/lscsoft/glue/man:/opt/lscsoft/libframe/man:/opt/lscsoft/libmetaio/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/man:/ldcg/condor//man:::/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/
> ma!
> >  n:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/share/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/man:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/share/man TERM=screen LAL_PREFIX=/opt/lscsoft/lal SASL_PATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib/sasl GSTAR_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar MYSQL_UNIX_PORT=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt-app-data/mysql/var/mysql.sock LSCSOFT_PREFIX=/opt/lscsoft HOSTNAME=ldas-grid SHELL=/bin/bash MATLABPATH=/ligotools/matlab WBONLINE=/archive/home/waveburst/S5_online VOMS_USERCONF=/ldcg/stow_pkgs
> /l!
> >  dg-4.3/ldg/vdt/glite/etc EDG_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/
> > edg LDG_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg HISTSIZE=1000 GLOBUS_PATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus SSH_CLIENT=130.39.245.165' '37419' '22 PYLAL_LOCATION=/archive/home/ram/opt/pylal GLOBUS_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus X509_CADIR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/TRUSTED_CA X509_CERT_DIR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/TRUSTED_CA PERL5LIB=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkg
> s/!
> >  ldg-4.3/ldg/vdt/vds/lib/perl:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/5.8.0/i686-linux-thread-multi:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/perl/lib/site_perl/5.8.0/i686-linux-thread-multi::/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar/lib CVSROOT=:pserver:igor__AT__ldas-sw.ligo.caltech.edu:/ldcg_server/common/repository_gds PYTHONPATH=/opt/lscsoft/lalapps/lib/python2.4/site-packages:/opt/lscsoft/glue/lib/python:/opt/lscsoft/libframe/lib/python:/opt/lscsoft/libmetaio/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib/python2.2/site-packages:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib64/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib/python:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib/python2.2/site-packages:/ldcg/sto
> w_!
> >  pkgs/ldg-4.3/ldg/vdt/globus/lib64/python:/ldcg/stow_pkgs/ldg-4.3/ldg/v
> > dt/globus/lib/python:/ldcg/pacman/src:/opt/lscsoft/lalapps/lib/python2.4/site-packages:/opt/lscsoft/glue/lib/python:/opt/lscsoft/libframe/lib/python:/opt/lscsoft/libmetaio/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/lib/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib/python2.2/site-packages:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib64/python:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib/python:/ldcg/pacman/src: QTDIR=/usr/lib/qt-3.3 LAL_LOCATION=/opt/lscsoft/lalapps EXTRAS_LOCATION=/archive/home/ram/opt/extras SHLVL=3 SSH_TTY=/dev/pts/0 TZ=America/Chicago GLITE_LOCATION_LOG=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/log VDS_HOME=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds GLOBUS_TCP_PORT_RANGE=40000,45000 USER=waveburst GLUE_LOCATION=/archive/home/ram/opt/glue GLOBUS_ERROR_VERBOSE=true LALAPPS_LOCATION=/archive/home/ram/opt/lalapps LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.c
> om!
> >  =01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35: LD_LIBRARY_PATH=/opt/lscsoft/lal/lib:/opt/lscsoft/glue/lib:/opt/lscsoft/libframe/lib:/opt/lscsoft/libmetaio/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/myodbc/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/lib/mysql:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/jre/lib/i386:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/jre/lib/i386/server:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/jre/lib/i386/client:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/berkeley-db/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/expat/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib:/ligotools/lib ROOTSYS=/archive/home/igor/SOFT
> 1/!
> >  root GPT_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt LDG_INSTALL_LOG=
> > /ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/etc/ldg-install.log GLITE_LOCATION_TMP=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/tmp TERMCAP=SC|screen|VT' '100/ANSI' 'X3.64' 'virtual' 'terminal:\'
> > 	':DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:UP=\E[%dA:bs:bt=\E[Z:\'
> > 	':cd=\E[J:ce=\E[K:cl=\E[H\E[J:cm=\E[%i%d;%dH:ct=\E[3g:\'
> > 	':do=^J:nd=\E[C:pt:rc=\E8:rs=\Ec:sc=\E7:st=\EH:up=\EM:\'
> > 	':le=^H:bl=^G:cr=^M:it#8:ho=\E[H:nw=\EE:ta=^I:is=\E)0:\'
> > 	':li#55:co#154:am:xn:xv:LP:sr=\EM:al=\E[L:AL=\E[%dL:\'
> > 	':cs=\E[%i%d;%dr:dl=\E[M:DL=\E[%dM:dc=\E[P:DC=\E[%dP:\'
> > 	':im=\E[4h:ei=\E[4l:mi:IC=\E[%d@:ks=\E[?1h\E=:\'
> > 	':ke=\E[?1l\E>:vi=\E[?25l:ve=\E[34h\E[?25h:vs=\E[34l:\'
> > 	':ti=\E[?1049h:te=\E[?1049l:us=\E[4m:ue=\E[24m:so=\E[3m:\'
> > 	':se=\E[23m:mb=\E[5m:md=\E[1m:mr=\E[7m:me=\E[m:ms:\'
> > 	':Co#8:pa#64:AF=\E[3%dm:AB=\E[4%dm:op=\E[39;49m:AX:\'
> > 	':vb=\Eg:G0:as=\E(0:ae=\E(B:\'
> > 	':ac=\140\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:\'
> > 	':po=\E[5i:pf=\E[4i:Z0=\E[?3h:Z1=\E[?3l:k0=\E[10~:\'
> > 	':k1=\EOP:k2=\EOQ:k3=\EOR:k4=\EOS:k5=\E[15~:k6=\E[17~:\'
> > 	':k7=\E[18~:k8=\E[19~:k9=\E[20~:k;=\E[21~:F1=\E[23~:\'
> > 	':F2=\E[24~:F3=\EO2P:F4=\EO2Q:F5=\EO2R:F6=\EO2S:\'
> > 	':F7=\E[15;2~:F8=\E[17;2~:F9=\E[18;2~:FA=\E[19;2~:kb=:\'
> > 	':K2=\EOE:kB=\E[Z:*4=\E[3;2~:*7=\E[1;2F:#2=\E[1;2H:\'
> > 	':#3=\E[2;2~:#4=\E[1;2D:%c=\E[6;2~:%e=\E[5;2~:%i=\E[1;2C:\'
> > 	':kh=\E[1~:@1=\E[1~:kH=\E[4~:@7=\E[4~:kN=\E[6~:kP=\E[5~:\'
> > 	':kI=\E[2~:kD=\E[3~:ku=\EOA:kd=\EOB:kr=\EOC:kl=\EOD:km: DAGDBUPDATORLOCKFILE=/etc/onasys-dblockfile PWD=/archive/home/waveburst/COHERENT_ONLINE KDEDIR=/usr LIBPATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib:/ldcg/ldg/vdt/globus/lib:/usr/lib:/lib MAIL=/var/spool/mail/waveburst PATH=/ldcg/condor/bin:/ldcg/condor/sbin:/opt/lscsoft/lalapps/bin:/opt/lscsoft/lal/bin:/opt/lscsoft/glue/bin:/opt/lscsoft/libframe/bin:/opt/lscsoft/libmetaio/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/pyglobus-url-copy/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/ftsh/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/bin:/ldcg/condor//sbin:/ldcg/condor//bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt/sbin:/ldcg/stow_pkgs/ldg-4.3/ld
> g/!
> >  vdt/globus/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/sbin:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/pyglobus-url-copy/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/ftsh/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/bin:/ldcg/condor//sbin:/ldcg/condor//bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt/sbin:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/bin:/ldcg/condor/bin:/ldcg/condor/sbin:/opt/lscsoft/lalapps/bin:/opt/lscsoft/lal/bin:/opt/lscsoft/glue/bin:/opt/lscsoft/libfram
> e/!
> >  bin:/opt/lscsoft/libmetaio/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/s
> > bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/pyglobus-url-copy/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/netlogger/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/edg/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/ftsh/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4/bin:/ldcg/condor//sbin:/ldcg/condor//bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/logrotate/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/gpt/sbin:/ldcg/pacman/src:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/sbin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vdt/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server/bin:/usr/kerberos/bin:/usr/bin:/bin:/usr/sbin:/sbin:/ldcg/ldg/vdt/globus/bin:/usr/X11R6/bin:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/contrib/gstar/bin:/ligotools/bin:/ldcg/matlab_r2006a/bin:/archive/home/waveburst/bin:/archive/home/igor/SOFT1/root/bin:/archive/home/waveburst/bin:.:/ldcg/matlab_r2006a/bin SHLIB_PATH=/ldcg/stow_pkgs
> /l!
> >  dg-4.3/ldg/vdt/globus/lib:/ldcg/ldg/vdt/globus/lib STY=14485.pts-0.ldas-grid _=/ldcg/condor/bin/condor_submit JAVA_HOME=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/jdk1.4 CONDOR_LOCATION=/ldcg/condor CONDOR_CONFIG=/ldcg/condor/etc/condor_config VDT_LOCATION=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt ODBCINI=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/unixodbc/etc/odbc.ini LSC_SEGFIND_SERVER=ldas.ligo-la.caltech.edu LOGNAME=waveburst INPUTRC=/etc/inputrc X509_USER_CERT=/archive/home/waveburst/.certificates/ldas-grid.ligo-la.caltech.edu/waveburstcert.pem LANG=C HOME=/archive/home/waveburst LIGOTOOLS=/ligotools GLITE_LOCATION_VAR=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/glite/var X509_USER_PROXY=/tmp/x509up_p10457.file9bvIoT.1 BOSSDIR=/etc X509_USER_KEY=/archive/home/waveburst/.certificates/ldas-grid.ligo-la.caltech.edu/waveburstkey.pem VDS_JAVA_HEAPMAX=1024 DYLD_LIBRARY_PATH=/opt/lscsoft/lal/lib:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/globus/lib:/opt/lscsoft/lal/lib VDT_INSTALL_LOG=vdt-install.log LSC_DATAFIND_SERVER=ldas.li
> go!
> >  -la.caltech.edu WINDOW=0 CLASSPATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds
> > /lib/commons-pool.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cog-jglobus.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/java-getopt-1.0.9.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cryptix-asn1.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cryptix.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/cryptix32.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/exist-optional.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/exist.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/gvds.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/jakarta-oro.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/loggerservice-stub.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/jce-jdk13-117.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/jlinker.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/junit.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/log4j-1.2.8.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/resolver.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/puretls.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/mysql-connector-java-3.0.11-stable-bin.jar:/ld
> cg!
> >  /stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/postgresql-8.1dev-400.jdbc3.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xercesImpl.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/rls.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xmlParserAPIs.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xmldb.jar:/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/vds/lib/xmlrpc.jar GLOBUS_MYSQL_PATH=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/mysql LSC_DATAGRID_SERVER_LOCATION=/ldcg/ldg SSH_CONNECTION=130.39.245.165' '37419' '130.39.245.243' '22 PKG_CONFIG_PATH=/opt/lscsoft/lal/lib/pkgconfig:/opt/lscsoft/glue/lib/pkgconfig:/opt/lscsoft/libframe/lib/pkgconfig:/opt/lscsoft/libmetaio/lib/pkgconfig:/opt/lscsoft/lal/lib/pkgconfig:/opt/lscsoft/glue/lib/pkgconfig:/opt/lscsoft/libframe/lib/pkgconfig:/opt/lscsoft/libmetaio/lib/pkgconfig: LDG_DIRECTORY=/ldcg/stow_pkgs/ldg-4.3/ldg/ldg-server LESSOPEN=|/usr/bin/lesspipe.sh' '%s VDT_POSTINSTALL_README=/ldcg/stow_pkgs/ldg-4.3/ldg/vdt/post-install/README DISPLAY=localhost:10.0 GLITE_LOCATION=/ldcg/stow_
> pk!
> >  gs/ldg-4.3/ldg/vdt/glite LDG_SOFTWARE_LOCATION=http://www.ligo.mit.edu
> > /ldg4.3/software PACMAN_LOCATION=/ldcg/pacman G_BROKEN_FILENAMES=1"' to record '04567848.-1' as it contains a newline, which is not allowed.
> > 1/19 18:15:44 (pid:2543) ERROR "write inside a transaction failed, errno = 0" at line 127 in file log_transaction.C
> > 1/19 18:17:42 (pid:28232) ******************************************************
> > 1/19 18:17:42 (pid:28232) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
> > 1/19 18:17:42 (pid:28232) ** /ldcg/stow_pkgs/condor-6.8.2/condor/sbin/condor_schedd
> > 1/19 18:17:42 (pid:28232) ** $CondorVersion: 6.8.2 Oct 12 2006 $
> > 1/19 18:17:42 (pid:28232) ** $CondorPlatform: I386-LINUX_RHEL3 $
> > 1/19 18:17:42 (pid:28232) ** PID = 28232
> > 1/19 18:17:42 (pid:28232) ** Log last touched 1/19 18:15:44
> > 1/19 18:17:42 (pid:28232) ******************************************************
> > 1/19 18:17:42 (pid:28232) Using config source: /usr1/condor/condor_config
> > 1/19 18:17:42 (pid:28232) Using local config sources: 
> > 1/19 18:17:42 (pid:28232)    /usr1/condor/condor_config.local
> > 1/19 18:17:42 (pid:28232) DaemonCore: Command Socket at <10.13.0.12:33572>
> > 1/19 18:17:42 (pid:28232) History file rotation is enabled.
> > 1/19 18:17:42 (pid:28232)   Maximum history file size is: 1000000000 bytes
> > 1/19 18:17:42 (pid:28232)   Number of rotated history files is: 100
> > 1/19 18:17:45 (pid:28232) 4567427.0: JobLeaseDuration remaining: 1040
> > 1/19 18:17:47 (pid:28232) Sent ad to central manager for kleinewelle@ligo
> > 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for kleinewelle@ligo
> > 1/19 18:17:47 (pid:28232) Sent ad to central manager for inspiralbns@ligo
> > 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for inspiralbns@ligo
> > 1/19 18:17:47 (pid:28232) Sent ad to central manager for waveburst_test@ligo
> > 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for waveburst_test@ligo
> > 1/19 18:17:47 (pid:28232) Sent ad to central manager for hoft@ligo
> > 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for hoft@ligo
> > 1/19 18:17:47 (pid:28232) Sent ad to central manager for pulsar@ligo
> > 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for pulsar@ligo
> > 1/19 18:17:47 (pid:28232) Sent ad to central manager for lindy@ligo
> > 1/19 18:17:47 (pid:28232) Sent ad to 1 collectors for lindy@ligo
> > 1/19 18:17:47 (pid:28232) Starting add_shadow_birthdate(4567427.0)
> > 1/19 18:17:47 (pid:28232) Started shadow for job 4567427.0 on "<10.13.1.163:41265>", (shadow pid = 28238)
> > 1/19 18:17:47 (pid:28232) Successfully created sched universe process
> > 1/19 18:17:47 (pid:28232) Starting add_shadow_birthdate(4343359.0)
> > 
> > 
> > Igor,
> > 	All of these restarts appear to be associated with the waveburst
> > account. Until the Condor team can explain/fix this please carefully
> > consider what you may have recently changed in the configuration of
> > this account. The simplest explanation is that you recently added a TERMCAP
> > environment variable setting that includes a newline which Condor apparently
> > does not allow.
> > 
> > Thanks.
> > 
> > 
> 
> 
> 
> ========================================
> MESSAGE INFORMATION
> ========================================
> * From: Greg Quinn <gquinn__AT__cs.wisc.edu>
> * Ticket Email List: anderson__AT__ligo.caltech.edu, espinoza_e__AT__ligo.caltech.edu,ldas_admin_llo__AT__ligo.caltech.edu
> 
> -- 
> ======================================================================
> This mail was sent from the RUST Mail System
> Please direct all replies to condor-support__AT__cs.wisc.edu
> Please include the current subject line in your reply.
> ======================================================================
> 

-- 
Stuart Anderson  anderson__AT__ligo.caltech.edu
http://www.ligo.caltech.edu/~anderson


===========================================================================
Date mail was appended: Fri Apr  6 14:46:09 2007 (1175888770)

Subject: Actions

Ticket resolved by gquinn
===========================================================================
Date of actions: Mon Apr  9 14:55:23 2007 (1176148523)

Subject: Actions

Assigned to tannenba by gquinn
===========================================================================
Date of actions: Thu Mar  5  9:25:02 2009 (1236266702)