LIGO Support Ticket 19343

Ticket Information
  Number:      admin 19343
  User:        ram@gravity.phys.uwm.edu
  Email:       anderson__AT__ligo.caltech.edu
  Status:      open
  Assigned To: psilord

Domainkey-Signature: a=rsa-sha1; c=nofws;        d=gmail.com; s=gamma;    
    h=mime-version:sender:from:date:x-google-sender-auth:message-id        
 :subject:to:cc:content-type:content-transfer-encoding;       
 b=QE8B+yRXMgfz8yH3PtmZCeYtpPgmV9+kVozeT594zPFa8bVZJnWTAu2ksqX0/UVzVL      
   HVubRuOJyJUl1kTytPVs7UBoGTVUcrvPv3CcWHxaR2YCavtgTqU3XM+E/ylo6j/2WXwC    
     p6soUMy5mpJ1Lmuoti1/7++W512uuWiCAFMeA=
From: Adam Mercer <ram__AT__gravity.phys.uwm.edu>
Date: Fri, 29 May 2009 16:31:00 -0500
Subject: LIGO condor_compile and libtool
To: =?UTF-8?B?Q29uZG9y77u/IEFkbWlu77u/IChCdWdzKe+7v++7vw==?=
 <condor-admin__AT__cs.wisc.edu>
CC: =?UTF-8?B?U3R1YXJ077u/77u/IEFuZGVyc29u77u/77u/?=
 <anderson__AT__ligo.caltech.edu>

Hi

One of our projects, that is often condor compiled, is currently
linked using libtool-1.4.2 and for the past several weeks I have been
trying, unsuccessfully, to update to the latest libtool release
(2.2.6a). Is there some restriction on libtool version when condor
compiling?

The problem I am running into seems to be related to the linker, we
have an --enable-condor configure flag which prepends "condor_compile"
to the compiler and when configuring with this option the following is
displayed:

checking for /opt/condor/lib/ld
ERROR: Internal ld was not invoked!
ERROR: Executable may not be linked properly for Condor!
ERROR: For users of Condor on Linux, we highly recommend
ERROR: using the default compiler that came with the
ERROR: distribution. Usually placing /usr/bin first in
ERROR: your path will remedy this error.
ERROR: To learn more about this error, visit this web page:
ERROR: http://www.cs.wisc.edu/condor/manual/faq.html
ERROR: and read the FAQ entry about this problem for your
ERROR: revision of Condor. option to reload object files... -r

AFAICT the FAQ this error is referring to is:

<http://www.cs.wisc.edu/condor/manual/v7.3/7_3Running_Condor.html#SECTION00837000000000000000>

and reading though this it appears that the above error is displayed
when not using the system compiler, however this is using the system
compiler so why is this error message being displayed? We are running
CentOS 5 with the following condor version

$ condor_version
$CondorVersion: 7.2.2 Apr  9 2009 BuildID: 145189 $
$CondorPlatform: X86_64-LINUX_RHEL5 $

libtool seems to pickup the correct linker, but also includes the
above error message, which therefore leads to problems. One of which
is in the generation of Makefiles which uses sed to replace the linker
in various places and as the above contains multiple lines leads to
failures in Makefile generation.

Has anyone any experience with condor_compile are recent libtool
versions that could shed light on this matter?

Cheers

Adam

===========================================================================
Date of creation: Fri May 29 16:31:35 2009 (1243632698)

Subject: Actions

Assigned to psilord by roy
===========================================================================
Date of actions: Sun May 31 19:35:13 2009 (1243816513)

Date: Fri, 28 Aug 2009 11:11:56 -0500
From: Peter Keller <psilord__AT__cs.wisc.edu>
To: roy <condor-admin__AT__cs.wisc.edu>
Subject: Re: [condor-admin #19343] LIGO condor_compile and libtool

Hello,

> From: Adam Mercer <ram__AT__gravity.phys.uwm.edu>
> One of our projects, that is often condor compiled, is currently
> linked using libtool-1.4.2 and for the past several weeks I have been
> trying, unsuccessfully, to update to the latest libtool release
> (2.2.6a). Is there some restriction on libtool version when condor
> compiling?
> 
> The problem I am running into seems to be related to the linker, we
> have an --enable-condor configure flag which prepends "condor_compile"
> to the compiler and when configuring with this option the following is
> displayed:
> 
> checking for /opt/condor/lib/ld
> ERROR: Internal ld was not invoked!
> ERROR: Executable may not be linked properly for Condor!
> ERROR: For users of Condor on Linux, we highly recommend
> ERROR: using the default compiler that came with the
> ERROR: distribution. Usually placing /usr/bin first in
> ERROR: your path will remedy this error.
> ERROR: To learn more about this error, visit this web page:
> ERROR: http://www.cs.wisc.edu/condor/manual/faq.html
> ERROR: and read the FAQ entry about this problem for your
> ERROR: revision of Condor. option to reload object files... -r

So, if I understand correctly, the message gets emitted when you
run ./configure? 

Could I see the portion of the configure.ac which emits the checking for
/opt/condor/lib/ld?

Could it be that you've 'condor_compiled gcc foo.c -c' or otherwise
produced a test which called a condor_compiled gcc that wasn't exepected
to call ld (like making an object file or getting version info)? Under
those conditions you'll get the above error because it really was that
ld was not invoked and condor_compile expects it to be so.

> and reading though this it appears that the above error is displayed
> when not using the system compiler, however this is using the system
> compiler so why is this error message being displayed? We are running
> CentOS 5 with the following condor version

It also could be that you've hard specified something in the calling
of the system compiler which forces the compiler to use the ld it was
compiled against and not the one in the path.

Could I get a sh -x ./configure > configure.out 2>&1 and have the
resultant file made available to me?

Thank you.

-pete

===========================================================================
Date mail was appended: Fri Aug 28 11:12:01 2009 (1251475921)

Subject: Actions

Status changed from open to pending by psilord
===========================================================================
Date of actions: Fri Aug 28 11:32:39 2009 (1251477159)

Domainkey-Signature: a=rsa-sha1; c=nofws;        d=gmail.com; s=gamma;    
    h=mime-version:sender:in-reply-to:references:from:date        
 :x-google-sender-auth:message-id:subject:to:cc:content-type;       
 b=nV6KAY7dCo8czIDJwEY76ibGArQ1A8DvmuVr3s2XlPPl9U2KgH35wgl89rSxWU82nD      
   UOGxmA4PKMywU3B5yak1o0YXLcRVoKJu7hKpiTCKwXFHhOqCyMdNpORoWG16agNmicLl    
     nmdX14VOmLhTbTPeM8WmrKaJAZQnXA/JDA/2c=
From: Adam Mercer <ram__AT__gravity.phys.uwm.edu>
Date: Thu, 22 Oct 2009 15:25:14 -0500
Subject: Re: [condor-admin #19343] LIGO condor_compile and libtool
To: condor-admin__AT__cs.wisc.edu
CC: anderson__AT__ligo.caltech.edu

Hi

I originally responded to this back in August, but Stuart has alerted
me that it didn't seem to get through. I've resent it, sorry for the
delay.

Cheers

Adam

---------- Forwarded message ----------
From: Adam Mercer <ram__AT__gravity.phys.uwm.edu>
Date: Fri, Aug 28, 2009 at 16:45
Subject: Re: [condor-admin #19343] LIGO condor_compile and libtool
To: condor-admin__AT__cs.wisc.edu
Cc: anderson__AT__ligo.caltech.edu


On Fri, Aug 28, 2009 at 11:12, condor-admin response tracking
system<condor-admin__AT__cs.wisc.edu> wrote:

Pete

>> From: Adam Mercer <ram__AT__gravity.phys.uwm.edu>
>> One of our projects, that is often condor compiled, is currently
>> linked using libtool-1.4.2 and for the past several weeks I have been
>> trying, unsuccessfully, to update to the latest libtool release
>> (2.2.6a). Is there some restriction on libtool version when condor
>> compiling?
>>
>> The problem I am running into seems to be related to the linker, we
>> have an --enable-condor configure flag which prepends "condor_compile"
>> to the compiler and when configuring with this option the following is
>> displayed:
>>
>> checking for /opt/condor/lib/ld
>> ERROR: Internal ld was not invoked!
>> ERROR: Executable may not be linked properly for Condor!
>> ERROR: For users of Condor on Linux, we highly recommend
>> ERROR: using the default compiler that came with the
>> ERROR: distribution. Usually placing /usr/bin first in
>> ERROR: your path will remedy this error.
>> ERROR: To learn more about this error, visit this web page:
>> ERROR: http://www.cs.wisc.edu/condor/manual/faq.html
>> ERROR: and read the FAQ entry about this problem for your
>> ERROR: revision of Condor. option to reload object files... -r
>
> So, if I understand correctly, the message gets emitted when you
> run ./configure?

Yes.

> Could I see the portion of the configure.ac which emits the checking for
> /opt/condor/lib/ld?

Essentially this is part of the check for the compiler, which uses the
autoconf macro AC_PROC_CC_C99, if it is available otherwise a local
copy of this macro is used (which is taken from autoconf-2.63)

I've included the entire configure.ac and collection of macros lalapps.m4 at

<http://www.lsc-group.phys.uwm.edu/~ram/files/condor-19343/>

> Could it be that you've 'condor_compiled gcc foo.c -c' or otherwise
> produced a test which called a condor_compiled gcc that wasn't exepected
> to call ld (like making an object file or getting version info)? Under
> those conditions you'll get the above error because it really was that
> ld was not invoked and condor_compile expects it to be so.

I don't think so

>> and reading though this it appears that the above error is displayed
>> when not using the system compiler, however this is using the system
>> compiler so why is this error message being displayed? We are running
>> CentOS 5 with the following condor version
>
> It also could be that you've hard specified something in the calling
> of the system compiler which forces the compiler to use the ld it was
> compiled against and not the one in the path.

We shouldn't, the compiler being used should always be the one found
by AC_PROG_CC_C99, if I'm understanding your point properly

> Could I get a sh -x ./configure > configure.out 2>&1 and have the
> resultant file made available to me?

Also uploaded to:

<http://www.lsc-group.phys.uwm.edu/~ram/files/condor-19343/>

Let me know if you anything further.

Cheers

Adam

===========================================================================
Date mail was appended: Thu Oct 22 15:25:47 2009 (1256243148)

Date: Fri, 23 Oct 2009 11:54:16 -0500
From: Peter Keller <psilord__AT__cs.wisc.edu>
To: condor-admin response tracking system <condor-admin__AT__cs.wisc.edu>
Subject: Re: [condor-admin #19343] LIGO condor_compile and libtool

Hello,

On Thu, Oct 22, 2009 at 03:25:47PM -0500, condor-admin response tracking system wrote:
> I originally responded to this back in August, but Stuart has alerted
> me that it didn't seem to get through. I've resent it, sorry for the
> delay.

No problem.

> > Could it be that you've 'condor_compiled gcc foo.c -c' or otherwise
> > produced a test which called a condor_compiled gcc that wasn't exepected
> > to call ld (like making an object file or getting version info)? Under
> > those conditions you'll get the above error because it really was that
> > ld was not invoked and condor_compile expects it to be so.
> 
> I don't think so

It seems this might be true in my initial glance of the output you provided:

Here is the relavent code from your configure.ac:

##########################################################
if test "x$condor" = "xtrue"; then
  AC_CHECK_PROGS(CONDOR_COMPILE, condor_compile)
  CC="$CONDOR_COMPILE $CC"
  case "$host" in
    *-*-linux* | *-*-osf* | *-*-hpux* ) CFLAGS="$CFLAGS -static" AC_DISABLE_SHARED;;
  esac
  AC_DEFINE(LALAPPS_CONDOR, 1, [LALApps is condor compiled])
fi
##########################################################

You change the variable $CC to "$CONDOR_COMPILE $CC" and that will affect
every single configure test or use of CC later in the configure.ac.

I think this is ultimately the cause, and will dig deeper into the configure
output you sent to see which rule you wrote after the change of $CC called
$CC without expecting to produce an executable.

If you moved the above block of code to after this line:

AC_CHECK_PROGS(BIBTEX, bibtex, echo)

does the error go away?

Thank you.

-pete

===========================================================================
Date mail was appended: Fri Oct 23 11:54:22 2009 (1256316862)

Domainkey-Signature: a=rsa-sha1; c=nofws;        d=gmail.com; s=gamma;    
    h=mime-version:sender:in-reply-to:references:from:date        
 :x-google-sender-auth:message-id:subject:to:cc:content-type        
 :content-transfer-encoding;        b=nQS6y7CL5TA5XBRCv6LniBqOV9/uxx+jCQbtMQsIqhtz2lU4iSd1qlSwfVKtbPOO1/
         j/qGPEveWSNADAu8N5KsQ3X2FWlR6cKlmoVdtHLjLpKbYhryoY0aUXc8Ax1hKenrRlAh
         155QvCeLg4zYjiC3VNdH5Hh49mXtCq5wT/S74=
From: Adam Mercer <ram__AT__gravity.phys.uwm.edu>
Date: Fri, 23 Oct 2009 14:12:05 -0500
Subject: Re: [condor-admin #19343] LIGO condor_compile and libtool
To: condor-admin__AT__cs.wisc.edu
CC: anderson__AT__ligo.caltech.edu

On Fri, Oct 23, 2009 at 11:54, condor-admin response tracking system
<condor-admin__AT__cs.wisc.edu> wrote:

>> > Could it be that you've 'condor_compiled gcc foo.c -c' or otherwise
>> > produced a test which called a condor_compiled gcc that wasn't exepect=
ed
>> > to call ld (like making an object file or getting version info)? Under
>> > those conditions you'll get the above error because it really was that
>> > ld was not invoked and condor_compile expects it to be so.
>>
>> I don't think so
>
> It seems this might be true in my initial glance of the output you provid=
ed:
>
> Here is the relavent code from your configure.ac:
>
> ##########################################################
> if test "x$condor" =3D "xtrue"; then
> =A0AC_CHECK_PROGS(CONDOR_COMPILE, condor_compile)
> =A0CC=3D"$CONDOR_COMPILE $CC"
> =A0case "$host" in
> =A0 =A0*-*-linux* | *-*-osf* | *-*-hpux* ) CFLAGS=3D"$CFLAGS -static" AC_=
DISABLE_SHARED;;
> =A0esac
> =A0AC_DEFINE(LALAPPS_CONDOR, 1, [LALApps is condor compiled])
> fi
> ##########################################################
>
> You change the variable $CC to "$CONDOR_COMPILE $CC" and that will affect
> every single configure test or use of CC later in the configure.ac.
>
> I think this is ultimately the cause, and will dig deeper into the config=
ure
> output you sent to see which rule you wrote after the change of $CC called
> $CC without expecting to produce an executable.
>
> If you moved the above block of code to after this line:
>
> AC_CHECK_PROGS(BIBTEX, bibtex, echo)
>
> does the error go away?

Yes, that allows me to configure without error. Thanks! However I am
getting link errors, essentially its trying to link in the dynamic
libraries instead of the static libraries:

/usr/bin/ld: attempted static link of dynamic object
`/home/ram/opt/lal/lib/liblalframe.so'
collect2: ld returned 1 exit status

But I also get this when trying to building statically when not condor
compiling so I don't think this is related to condor.

Cheers

Adam

===========================================================================
Date mail was appended: Fri Oct 23 14:12:37 2009 (1256325158)

Domainkey-Signature: a=rsa-sha1; c=nofws;        d=gmail.com; s=gamma;    
    h=mime-version:sender:in-reply-to:references:from:date        
 :x-google-sender-auth:message-id:subject:to:cc:content-type;       
 b=A07hPt17W0GuYEjNQppHzyfBn6jgy72TXwzlLK7XTvk4atgj7u4rX1Gacb2O1NfZ1k      
   wqsNvUYW7JZu4rr/vFdoJbdSSAZ0aDBlvZJwQG/uVCmOAQfOMWsEDNbiLm4yHeGkVyIe    
     L6PrfswOfWs3JmR+DG3nFjsQLHlTiYXweiLxI=
From: Adam Mercer <ram__AT__gravity.phys.uwm.edu>
Date: Fri, 23 Oct 2009 15:23:05 -0500
Subject: Re: [condor-admin #19343] LIGO condor_compile and libtool
To: condor-admin__AT__cs.wisc.edu
CC: anderson__AT__ligo.caltech.edu

On Fri, Oct 23, 2009 at 14:12, Adam Mercer <ram__AT__gravity.phys.uwm.edu> wrote:

> But I also get this when trying to building statically when not condor
> compiling so I don't think this is related to condor.

Actually I spoke to soon, this does seem to still be related to
condor_compile. The reason I was getting the link failures is that
libtool was set to link the shared libraries as AC_PROG_LIBTOOL (or
LT_INIT in more recent versions of libtool) was called before
AC_DISABLE_SHARED when condor compiling. The correct way to tell
libtool to link statically is

AC_DISABLE_SHARED
AC_PROG_LIBTOOL

moving the AC_PROG_LIBTOOL to be after the block:

if test "x$condor" = "xtrue"; then
  AC_CHECK_PROGS([CONDOR_COMPILE],[condor_compile])
  CC="$CONDOR_COMPILE $CC"
  case "$host" in
    *-*-linux* | *-*-osf* | *-*-hpux* )
			CFLAGS="$CFLAGS -static"
			AC_DISABLE_SHARED
			;;
  esac
  AC_DEFINE([LALAPPS_CONDOR],[1],[LALApps is condor compiled])
fi

causes standard static linking to work, but configuring to build using
condor_compile results in the same failures I obtained initially, i.e.
no targets in each Makefile

Cheers

Adam

===========================================================================
Date mail was appended: Fri Oct 23 15:25:16 2009 (1256329516)

Date: Fri, 23 Oct 2009 15:28:42 -0500
From: Peter Keller <psilord__AT__cs.wisc.edu>
To: condor-admin response tracking system <condor-admin__AT__cs.wisc.edu>
Subject: Re: [condor-admin #19343] LIGO condor_compile and libtool

Hello,

On Fri, Oct 23, 2009 at 02:12:37PM -0500, condor-admin response tracking system wrote:
> > If you moved the above block of code to after this line:
> >
> > AC_CHECK_PROGS(BIBTEX, bibtex, echo)
> >
> > does the error go away?
> 
> Yes, that allows me to configure without error. Thanks!

Good! I thought it might have been some kind of problem like that....

> However I am
> getting link errors, essentially its trying to link in the dynamic
> libraries instead of the static libraries:
> 
> /usr/bin/ld: attempted static link of dynamic object
> `/home/ram/opt/lal/lib/liblalframe.so'
> collect2: ld returned 1 exit status
> 
> But I also get this when trying to building statically when not condor
> compiling so I don't think this is related to condor.

Ah, it is related a little bit to Condor because condor_compile *enforces*
a static linkage for your program. Any libraries with which you would like
to link should have an archive format available. Also, just in case you
might be using it, -lpthread is forbidden so if you have a threaded program,
you are distinctly out of luck with condor_compile.

Thank you.

-pete

===========================================================================
Date mail was appended: Fri Oct 23 15:28:48 2009 (1256329728)

Date: Fri, 23 Oct 2009 15:32:54 -0500
From: Peter Keller <psilord__AT__cs.wisc.edu>
To: condor-admin response tracking system <condor-admin__AT__cs.wisc.edu>
Subject: Re: [condor-admin #19343] LIGO condor_compile and libtool

On Fri, Oct 23, 2009 at 03:25:16PM -0500, condor-admin response tracking system wrote:
> causes standard static linking to work, but configuring to build using
> condor_compile results in the same failures I obtained initially, i.e.
> no targets in each Makefile

Could I see the new version of those files you link to before, plus config.log, your Makefil.in and the generated Makefile?

Thank you.

-pete

===========================================================================
Date mail was appended: Fri Oct 23 15:33:00 2009 (1256329980)

Dkim-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;        d=gmail.com;
 s=gamma;        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:from:date:x-google-sender-auth:message-id:subject:to:cc
         :content-type;        bh=sW0tlVJRlsd9qErQHwbUyWJIvH4wOXjeWnG2xz5jyjc=;
        b=lx9GFMM8eF4YBg+cB1hVBlEmkVPSd2eVVmhUZRk4/fuNDeNdmQkenbN8lcraQuXs2t
         /klh+YSA5JjVbXSsAxRklL1I12lQ+/9g83ZfGTrPCM/WOV8td+fEMKWoKhVa7/qz/GrE
         5CjN5Q99vZ0CTyTMwFBpw9Gw5YndrqhFmxTDE=
Domainkey-Signature: a=rsa-sha1; c=nofws;        d=gmail.com; s=gamma;    
    h=mime-version:sender:in-reply-to:references:from:date        
 :x-google-sender-auth:message-id:subject:to:cc:content-type;       
 b=gbtw6kp/30eiJEqi7px6M4dB8X0s1IaN2sS/Kv5JPT1wOr+w5maFCBO0oOZPKQ5Qfm      
   YJu0B73YynREG8yWHFATffDSZt7ohrgNAW3Ta5VUx9Hk5yYFdCvXG3/KmKss39KUJ5B7    
     LWbMXDpD4sRNAdjOSJ6x8XSBUkvfn2gjHKeus=
From: Adam Mercer <ram__AT__gravity.phys.uwm.edu>
Date: Fri, 23 Oct 2009 16:21:39 -0500
X-Google-Sender-Auth: bc48a010cd6f65c5
Subject: Re: [condor-admin #19343] LIGO condor_compile and libtool
To: condor-admin__AT__cs.wisc.edu
CC: anderson__AT__ligo.caltech.edu
X-Seen-BY: mailfromd 4.1 obsidian.cs.wisc.edu

On Fri, Oct 23, 2009 at 15:33, condor-admin response tracking system
<condor-admin__AT__cs.wisc.edu> wrote:

> Could I see the new version of those files you link to before, plus config.log, your Makefil.in and the generated Makefile?

Uploaded to:

<http://www.lsc-group.phys.uwm.edu/~ram/files/condor-19343/20091023/>'

I've added the top-level Makefile and Makefile.in, if you need others
let me know.

Cheers

Adam

===========================================================================
Date mail was appended: Fri Oct 23 16:22:08 2009 (1256332928)