Subject: uclibc mips ld.so and undefined symbols with nonzero symbol table entry st_value

From: "Bradley D. LaRonde" <[email-masked]>
Date: Mon, 3 May 2004 17:25:11 -0400

I would like ld.so in uClibc CVS HEAD (last week or so) to pass the
test/dlopen/ tests on mips(el).  I'm currently focusing on dltest.  As-is
dltest seg faults soon after ld.so hands control over to the application.

Some background:  I'm using the uClibc "buildroot" environment, that first
builds a mipsel cross-toolchain and then cross-builds a mipsel uClibc root
file system image with Busybox.  I configure busybox to build statically.
I'm using buildroot's stock gcc 3.3.3 configuration and I have upgraded my
local buildroot to binutils 2.15.90.0.1.1.

When dltest runs, it ends up segfaulting inside ld.so.  I tracked down the
crash to what appears to me to be a bogus pointer to the malloc function.
This only happens if I specify -lpthread in the build of dltest.  uClibc
libc.so contains weak symbols shadowing libpthread symbols so dltest will
build without -lpthread.  dltest doesn't actually use any libpthread
functionality AFAICT.

The apparently bogus pointer-to-malloc get's assigned to pointer-to-function
global variable _dl_malloc_function in libdl's dlopen.c, line 164.  The
pointer that gets assigned falls in libpthread's address space.
Interestingly in points to what appears to be a procedure stub to malloc.
readelf on libpthread.so shows:

       312: 00003680     0 FUNC    GLOBAL DEFAULT  UND malloc

and that's right where the apparently bogus pointer points... libpthread's
base virtual address + 0x3680.  ld.so crashes shortly afterward when it
calls _dl_malloc (which calls _dl_malloc_function) from
uClibc/ldso/ldso/dl-hash.c::_dl_add_elf_hash_table.  At least I think that's
the function; I tracked it exactly but didn't make a note.  :-(

I suppose that binutils is pointing out a stub based on this MIPS PS ABI
paragraph:

    If an executable or shared object contains a reference to a
    function defined in one of its associated shared objects, the
    symbol table section for that file will contain an entry for
    that symbol. The st_shndx member of that symbol table entry
    contains SHN_UNDEF. This signals to the dynamic linker that
    the symbol definition for that function is not contained in
    the executable file. If there is a stub for that symbol in
    the executable file and the st_value member for the symbol
    table entry is nonzero, the value will contain the virtual
    address of the first instruction of that procedure's stub.
    Otherwise, the st_value member contains zero. This stub calls
    the dynamic linker at runtime for lazy text evaluation. See
    "Function Addresses" in Chapter 5 for details.

I guess that the point of these procedure stubs is to keep
pointer-to-function values consistent between executables and share
libraries.  Is that what binutils is trying to accomplish here?

But should stubs really be getting involved here?  As Thiemo Seufer pointed
out to me, readelf shows me that every undefined symbol in every shared
library in /lib on my x86 debian box has the st_value member for the symbol
table entry set to zero.  I think Thiemo verified the same on some mips
linux box.  Does that mean no stubs in shared libraries?

This stuff is way off the map for me.  I would appreciate any help you have
to offer.  Thanks.

Regards,
Brad

---

From: Richard Sandiford <[email-masked]>
Date: Sun, 09 May 2004 14:14:33 +0100

> libraries.  Is that what binutils is trying to accomplish here?

No, it's to enable lazy binding.  The idea is that when the dynamic
loader loads libpthread.so, it doesn't need to resolve malloc()
immediately, it can just leave the GOT entry pointing to the stub.
Then, when the stub is called, it will ask the dynamic linker to
find the true address of malloc() and update the GOT accordingly.
This is supposed to reduce start-up time.

An object should never use stubs if takes the address of the function.
It should only use a stub for some symbol foo if every use of foo is
for a direct call.

If the dynamic loader is choosing libpthread's stub over the real
definition in libc.so, that sounds on the face of it like a dynamic
loader bug.

> table entry set to zero.

The x86 and MIPS ABIs are very different though.

Richard

---

From: "Bradley D. LaRonde" <[email-masked]>
Date: Sun, 9 May 2004 16:52:17 -0400

----- Original Message ----- 
From: "Richard Sandiford" <[email-masked]>
To: "Bradley D. LaRonde" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Sunday, May 09, 2004 9:14 AM
Subject: Re: uclibc mips ld.so and undefined symbols with nonzero symbol
table entry st_value

> It should only use a stub for some symbol foo if every use of foo is

OK.  So in a case where an object does take a pointer, does that mean that
ld.so must fix the GOT entry for that symbol before handing control to the
app (i.e. no lazy binding for that symbol)?

> definition in libc.so, that sounds on the face of it like a dynamic

OK.

> The x86 and MIPS ABIs are very different though.

I notice that the debian mipsel libpthread.so.0 in
http://ftp.debian.org/pool/main/g/glibc/libc6_2.2.5-11.5_mipsel.deb has
st_value == 0 for every UND FUNC, just like my x86 debian libraries.  This
is very different than the uClibc libpthread.so where every UND FUNC has
st_value != 0.  Interestingly if I link glibc's libpthread with uClibc's
libc.so I see that most UND FUNCs then have st_value != 0.

I would like to see how uClibc ld.so behaves I could somehow get ld to not
generate any stubs in  libpthread.  Any idea why libpthread gets full stubs
when linked with uClibc libc.so but no stubs when linked with glibc libc.so?

Regards,
Brad

---

From: Richard Sandiford <[email-masked]>
Date: Mon, 10 May 2004 08:40:37 +0100

> app (i.e. no lazy binding for that symbol)?

Right.  Such objects won't use a stub, they'll just have a normal
reference to an undefined symbol.  The dynamic loader must resolve
it in the usual way.

> libc.so I see that most UND FUNCs then have st_value != 0.

You said in your original message that you'd recently upgraded to binutils
2.15-based tools.  Was your libpthread.so built with them?  If so, that would
explain it.  I think earlier versions of ld were overly pessimistic about when
stubs could be used.

FWIW, I have a glibc-based sysroot built with gcc 3.4 and binutils 2.15.
Its libpthread uses plenty of stubs.

Richard

---

From: "Joakim Tjernlund" <[email-masked]>
Date: Mon, 10 May 2004 14:21:13 +0200

> when linked with uClibc libc.so but no stubs when linked with glibc libc.so?

This looks like the problem we had with dlopen() and friends when introducing
the new WEAK symbol handling.

in libc/misc/pthread/weaks.c you have stuff like:
weak_alias(__phtread_return_0, __phtread_once);

where __phtread_return_0 is non NULL:
int __phtread_return_0(void)
{
 return 0;
}

---

From: "Joakim Tjernlund" <[email-masked]>
Date: Mon, 10 May 2004 14:36:20 +0200

> int __phtread_return_0(void)

Changing the alias for __phtread_once to
extern int __phtread_once(void) __attribute__ ((weak));
makes st_value=0 in libc, making the entry
look the same as in glibc.

 Jocke

---

From: "Joakim Tjernlund" <[email-masked]>
Date: Mon, 10 May 2004 16:23:18 +0200

> look the same as in glibc.

hmm, this is not quite true. The binding in glibc is WEAK, but
the above trick makes the binding GLOBAL in uClibc.

  Jocke
PS

 I am on PPC, so I may be way off here w.r.t MIPS:(

---

From: "Bradley D. LaRonde" <[email-masked]>
Date: Mon, 10 May 2004 14:05:27 -0400

----- Original Message ----- 
From: "Richard Sandiford" <[email-masked]>
To: "Bradley D. LaRonde" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Monday, May 10, 2004 3:40 AM
Subject: Re: uclibc mips ld.so and undefined symbols with nonzero symbol
table entry st_value

> explain it.  I think earlier versions of ld were overly pessimistic about
when

Actually I did this round of testing with 2.14.90.0.7 to try and match
debian.

> Its libpthread uses plenty of stubs.

That's encouraging.  I'll switch back to 2.15 and see if I can find why
uClibc ld.so chooses the libpthread stub for libdl's malloc pointer.

Even though it is pointing libdl to the libpthread stub for malloc, should
it crash?  The first call of the stub should cause the libdl GOT entry to be
adjusted to point to the libc malloc?  Should a subsequent call of the stub
cause a crash?

Regards,
Brad

---

From: Richard Sandiford <[email-masked]>
Date: Mon, 10 May 2004 19:21:04 +0100

"Bradley D. LaRonde" <[email-masked]> writes:
> it crash?

Yeah.  When you call a stub, $gp must already be set to the owning
object's _gp.  That's how the dynamic loader knows which GOT to change.

In your case, libdl will be calling libpthread's stub with $gp set to
libdl's _gp.  The dynamic loader will therefore end up trying to change
libdl's GOT, not libpthread's.

Richard

---

From: "Bradley D. LaRonde" <[email-masked]>
Date: Mon, 10 May 2004 16:36:14 -0400

----- Original Message ----- 
From: "Richard Sandiford" <[email-masked]>
To: "Bradley D. LaRonde" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Monday, May 10, 2004 2:21 PM
Subject: Re: uclibc mips ld.so and undefined symbols with nonzero symbol
table entry st_value

> libdl's GOT, not libpthread's.

I read this in the spec:

    All externally visible symbols, both defined and undefined,
    must be hashed into the hash table.

Should libpthread's malloc stub be added to the hash table?  I guess not,
but I think that might be happening (haven't verified), and libdl finding it
in there and thinking it is the real deal, not realizing it is just a stub.

Regards,
Brad

---

From: Richard Sandiford <[email-masked]>
Date: Mon, 10 May 2004 21:41:53 +0100

"Bradley D. LaRonde" <[email-masked]> writes:
> Should libpthread's malloc stub be added to the hash table?

Yes.

> I guess not, but I think that might be happening (haven't verified),
> and libdl finding it in there and thinking it is the real deal, not
> realizing it is just a stub.

If you have an undefined function symbol with st_value != 0, then
that st_value must be for a stub.  That's how the loader can (and is
supposed to) tell the difference.

It's probably a good idea to look at how glibc handles this.

Richard

---

From: Richard Sandiford <[email-masked]>
Date: Mon, 10 May 2004 21:44:08 +0100

>> Should libpthread's malloc stub be added to the hash table?

Or, to be more precise, the symbol should be added to the hash table.
The fact that the symbol is lazily bound doesn't matter.

Richard

---

From: "Bradley D. LaRonde" <[email-masked]>
Date: Mon, 10 May 2004 23:39:41 -0400

----- Original Message ----- 
From: "Richard Sandiford" <[email-masked]>
To: "Bradley D. LaRonde" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>
Sent: Monday, May 10, 2004 4:41 PM
Subject: Re: uclibc mips ld.so and undefined symbols with nonzero symbol
table entry st_value

> It's probably a good idea to look at how glibc handles this.

uClibc/ldso/ldso/mips/elfinterp.c around line 288 looks like this:

    /* Relocate the global GOT entries for the object */
    while(i--) {
      if (sym->st_shndx == SHN_UNDEF) {
        if (ELF32_ST_TYPE(sym->st_info) == STT_FUNC && sym->st_value)
          *got_entry = sym->st_value + (unsigned long) tpnt->loadaddr;
        else {
          *got_entry = (unsigned long) _dl_find_hash(strtab +
             sym->st_name, tpnt->symbol_scope, ELF_RTYPE_CLASS_COPY);
        }
     }

If I change that ELF_RTYPE_CLASS_COPY to ELF_RTYPE_CLASS_PLT to tell
_dl_find_hash to ignore stubs when resolving undefined functions without
stubs, the dlopen tests all pass.  dlopen gets a pointer to the libc.so
malloc instead of a pointer to the libpthread malloc stub.  Yay!  :-)

Does that look like the correct fix?

Regards,
Brad

---

From: "Joakim Tjernlund" <[email-masked]>
Date: Tue, 11 May 2004 07:48:27 +0200

> Does that look like the correct fix?

Hopefully, when I added ELF_RTYPE_CLASS_COPY/ELF_RTYPE_CLASS_PLT stuff I had to
guess what to do with MIPS. After that several minor error has been fixed by
changing these bits.

  Jocke

---

From: "Joakim Tjernlund" <[email-masked]>
Date: Tue, 11 May 2004 09:27:42 +0200

> the above trick makes the binding GLOBAL in uClibc.

__phtread_once becoms WEAK if add "weak_function" to the
declaration in libc/inet/rpc/rpc_thread.c:
      extern weak_function int __pthread_once (pthread_once_t *__once_control,
			   void (*__init_routine) (void));
and remove the it compleatly from libc/misc/pthread/weaks.c
 
Now __phtread_once matches glibc, execpt for the reloc type. In glibc
you get a R_PPC_GLOB_DAT and in uClibc a R_PPC_ADDR32. Don't think that matters.

Is it desirable to match the pthread_ functions with glibc?
I havn't tried to run any code with these changes, so I have no
idea if it actually works.

 Jocke

---

From: Daniel Jacobowitz <[email-masked]>
Date: Tue, 11 May 2004 10:03:51 -0400

> Does that look like the correct fix?

Probably, since MIPS doesn't have a copy reloc.

-- 
Daniel Jacobowitz

---

From: "Bradley D. LaRonde" <[email-masked]>
Date: Tue, 11 May 2004 10:21:33 -0400

----- Original Message ----- 
From: "Daniel Jacobowitz" <[email-masked]>
To: "Bradley D. LaRonde" <[email-masked]>
Cc: "Richard Sandiford" <[email-masked]>; <[email-masked]>;
<[email-masked]>
Sent: Tuesday, May 11, 2004 10:03 AM
Subject: Re: uclibc mips ld.so and undefined symbols with nonzero symbol
table entry st_value

> Probably, since MIPS doesn't have a copy reloc.

How about the other copy reloc right below there:

    else if (sym->st_shndx == SHN_COMMON) {
      *got_entry = (unsigned long) _dl_find_hash(strtab +
        sym->st_name, tpnt->symbol_scope, ELF_RTYPE_CLASS_COPY);
    }

?

Regards,
Brad

---

From: Daniel Jacobowitz <[email-masked]>
Date: Tue, 11 May 2004 10:23:00 -0400

>         sym->st_name, tpnt->symbol_scope, ELF_RTYPE_CLASS_COPY);

I don't know anything about the uclibc linker code, so I'm not sure.
That's probably OK as future proofing against a copy reloc someday.

-- 
Daniel Jacobowitz

---

From: "Joakim Tjernlund" <[email-masked]>
Date: Tue, 11 May 2004 18:33:42 +0200

>         sym->st_name, tpnt->symbol_scope, ELF_RTYPE_CLASS_COPY);

Perhaps DL_NO_COPY_RELOCS should be defined for MIPS then?
see ldso/include/dl-elf.h.

 Jocke

---

From: "Bradley D. LaRonde" <[email-masked]>
Date: Tue, 11 May 2004 12:55:27 -0400

----- Original Message ----- 
From: "Joakim Tjernlund" <[email-masked]>
To: "Bradley D. LaRonde" <[email-masked]>; "Daniel Jacobowitz"
<[email-masked]>
Cc: <[email-masked]>; "Richard Sandiford" <[email-masked]>;
<[email-masked]>
Sent: Tuesday, May 11, 2004 12:33 PM
Subject: RE: [uClibc] Re: uclibc mips ld.so and undefined symbols with
nonzerosymbol table entry st_value

> see ldso/include/dl-elf.h.

For mips glibc 2.3.3 does:

#define elf_machine_type_class(type)            ELF_RTYPE_CLASS_PLT

and never uses ELF_RTYPE_CLASS_COPY directly (though it does use
ELF_RTYPE_CLASS_PLT directly).

Regards,
Brad

---

From: "Joakim Tjernlund" <[email-masked]>
Date: Tue, 11 May 2004 23:50:44 +0200

> ELF_RTYPE_CLASS_PLT directly).

Good, try replacing all ELF_RTYPE_CLASS_COPY with ELF_RTYPE_CLASS_PLT in 
mips/elfinterp.c and define DL_NO_COPY_RELOCS in mips/dl-sysdep.h

  Jocke