Subject: Re: Huge dynamically linked program does not run on mips-linux

From: Atsushi Nemoto <[email-masked]>
Date: Tue, 04 Nov 2003 14:21:11 +0900 (JST)

>>>>> On Wed, 29 Oct 2003 18:25:04 -0800, Eric Christopher <[email-masked]> said:
>> When building python-qt3 on debian unstable I get an oversize GOT
>> and:
>> 
>> /usr/bin/ld: BFD 2.14.90.0.6 20030820 Debian GNU/Linux assertion
>> fail ../../bfd/elfxx-mips.c:2287
>> 
>> Seems like multi-GOT is broken for this case.

eric> Judging from the line offset from today's sources to that error
eric> message it might have been fixed with rsandifo's last patch that
eric> fixed an uninitialized variable problem. (global_gotno)

I tried binutils-2.14.90.0.7 (based on binutils 2003 1029 in CVS) but
my problem did no solved.  It seems something is still wrong.

---
Atsushi Nemoto

---

From: Eric Christopher <[email-masked]>
Date: Tue, 04 Nov 2003 00:05:56 -0800

> my problem did no solved.  It seems something is still wrong.

This was plain mips-linux? Not mips64-linux?

And where would I find the sources?

-eric

-- 
Eric Christopher <[email-masked]>

---

From: Atsushi Nemoto <[email-masked]>
Date: Tue, 04 Nov 2003 20:02:22 +0900 (JST)

>>>>> On Tue, 04 Nov 2003 00:05:56 -0800, Eric Christopher <[email-masked]> said:
>> I tried binutils-2.14.90.0.7 (based on binutils 2003 1029 in CVS)
>> but my problem did no solved.  It seems something is still wrong.

eric> This was plain mips-linux? Not mips64-linux?

Yes.  mips-linux and mipsel-linux target (host is i386).  Both target
generate broken binary for my test program.

eric> And where would I find the sources?

I'm using plain binutils 2.14 and gcc 3.3.2 from gnu.org FTP site,
binutils 2.14.90.0.7 from
http://www.kernel.org/pub/linux/devel/binutils/.

---
Atsushi Nemoto

---

From: Daniel Jacobowitz <[email-masked]>
Date: Tue, 4 Nov 2003 11:17:41 -0500

On Thu, Oct 30, 2003 at 09:54:53PM +0900, Atsushi Nemoto wrote:
> >>>>> On Wed, 29 Oct 2003 13:15:17 -0500, Daniel Jacobowitz <[email-masked]> said:
> dan> Atsushi-san's program would not even link with a binutils that
> dan> didn't support multiple GOTs; I guess that something is going
> dan> wrong with that support.
> 
> Yes, I guess too.
> 
> dan> I don't suppose you could provide a testcase?
> 
> I wrote a short script to generate a testcase.  Running attached awk
> script create src0.c, src1.c, ..., src3.c.
> 
> $ mips-linux-gcc -c src[0-4].c
> $ mips-linux-gcc -o bigapp src[0-4].o
> 
> This bigapp program compiled with binutils-2.14, gcc-3.3.2, uClibc
> 0.9.21 is corrupted.
> 
> Here is some outputs from readelf:
> 
> $ mips-linux-readelf -S bigapp
> There are 31 section headers, starting at offset 0x36ad78:
> 
> Section Headers:
>   [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
>   [ 0]                   NULL            00000000 000000 000000 00      0   0  0
>   [ 1] .interp           PROGBITS        004000f4 0000f4 00000d 00   A  0   0  1
>   [ 2] .reginfo          MIPS_REGINFO    00400104 000104 000018 18   A  0   0  4
>   [ 3] .dynamic          DYNAMIC         0040011c 00011c 0000d8 08   A  6   0  4
>   [ 4] .hash             HASH            004001f4 0001f4 028784 04   A  5   0  4
>   [ 5] .dynsym           DYNSYM          00428978 028978 061c40 10   A  6   1  4
>   [ 6] .dynstr           STRTAB          0048a5b8 08a5b8 041da5 00   A  0   0  1
>   [ 7] .init             PROGBITS        004cc370 0cc370 0000ac 00  AX  0   0  4
>   [ 8] .text             PROGBITS        004cc420 0cc420 1b7d60 00  AX  0   0 16
>   [ 9] .fini             PROGBITS        00684180 284180 00005c 00  AX  0   0  4
>   [10] .rodata           PROGBITS        006841e0 2841e0 000010 00   A  0   0 16
>   [11] .data             PROGBITS        10000000 285000 000010 00  WA  0   0 16
>   [12] .rld_map          PROGBITS        10000010 285010 000004 00  WA  0   0  4
>   [13] .eh_frame         PROGBITS        10000014 285014 000004 00   A  0   0  4
>   [14] .ctors            PROGBITS        10000018 285018 000008 00  WA  0   0  4
>   [15] .dtors            PROGBITS        10000020 285020 000008 00  WA  0   0  4
>   [16] .jcr              PROGBITS        10000028 285028 000004 00  WA  0   0  4
>   [17] .got              PROGBITS        10000030 285030 02245c 04 WAp  0   0 16
>   [18] .sbss             NOBITS          1002248c 2a7490 000000 00 WAp  0   0  1
>   [19] .bss              NOBITS          10022490 2a7490 000020 00  WA  0   0 16
>   [20] .comment          PROGBITS        00000000 2a7490 0000a2 00      0   0  1
>   [21] .debug_aranges    MIPS_DWARF      00000000 2a7538 000020 00      0   0  8
>   [22] .debug_info       MIPS_DWARF      00000000 2a7558 000069 00      0   0  1
>   [23] .debug_abbrev     MIPS_DWARF      00000000 2a75c1 000014 00      0   0  1
>   [24] .debug_line       MIPS_DWARF      00000000 2a75d5 000044 00      0   0  1
>   [25] .pdr              PROGBITS        00000000 2a761c 0c3660 00      0   0  4
>   [26] .rel.dyn          REL             004cc360 0cc360 000010 08   A  5   0 16
>   [27] .mdebug.abi32     PROGBITS        00000000 36ac7c 000000 00      0   0  1
>   [28] .shstrtab         STRTAB          00000000 36ac7c 0000fb 00      0   0  1
>   [29] .symtab           SYMTAB          00000000 36b250 0620a0 10     30  45  4
>   [30] .strtab           STRTAB          00000000 3cd2f0 041ee7 00      0   0  1
> Key to Flags:
>   W (write), A (alloc), X (execute), M (merge), S (strings)
>   I (info), L (link order), G (group), x (unknown)
>   O (extra OS processing required) o (OS specific), p (processor specific)
> $ mips-linux-readelf -r bigapp
> 
> Relocation section '.rel.dyn' at offset 0xcc360 contains 2 entries:
>  Offset     Info    Type            Sym.Value  Sym. Name
> 00000000  00000000 R_MIPS_NONE      
> 1001f4f0  00314703 R_MIPS_REL32      004cc490   getpid
> $ mips-linux-readelf -x 17 bigapp | grep 1001f4f0
>   0x1001f4f0 004cc490 005ea570 00655ba0 0062c9e0 .L...^.p.e[..b..
> $ mips-linux-readelf -s bigapp | grep getpid
>  12615: 004cc490     0 FUNC    GLOBAL DEFAULT  UND getpid
>   4109: 004cc490     0 FUNC    GLOBAL DEFAULT  UND getpid

Now this is very strange.  Please bear with me; I have the bad habit of
starting my email before I've fully figured out the problem...

MIPS, at least MIPS/Linux, has a very odd ELF ABI.  The following isn't
based on documentation, just on my reading of the glibc loader.  The
.got section is relocated implicitly; dynamic relocations aren't even
_used_ normally, in the single-GOT case.

First we have reserved entres; two of them.  Then local got entries;
DT_MIPS_LOCAL_GOTNO of them.  They get the library base added.  Then
global entries.  We start in the dynsym table at DT_MIPS_GOTSYM, and
count up to just under DT_MIPS_SYMTABNO; GOT entries are set to the
address of the referenced symbol.

The loader does not require or expect any dynamic relocations to refer
to the .got section.  It also doesn't seem to care what's there, at
least not for external functions - defined functions and sections are
another matter.  So it doesn't expect the relocation to be there. 

Now, here's the peculiar thing:
drow@nevyn:~/mips-got% readelf -Ds app | sort -n | grep UND
   55 3777: 00000000     0  NOTYPE   WEAK DEFAULT UND __gmon_start__
  768 1005: 00000000     0  NOTYPE   WEAK DEFAULT UND _Jv_RegisterClasses
  5231 8511: 00000000     0    FUNC GLOBAL DEFAULT UND __libc_start_main
  13350 4887: 00000000     0    FUNC GLOBAL DEFAULT UND getpid
  22116 8185: 00000000     0    FUNC GLOBAL DEFAULT UND printf
drow@nevyn:~/mips-got% readelf -Dr app                     

'REL' relocation section at offset 0x4d8700 contains 24 bytes:
 Offset     Info    Type            Sym.Value  Sym. Name
00000000  00000000 R_MIPS_NONE      
1001c53c  00342603 R_MIPS_REL32      00000000   getpid
100196c4  00566403 R_MIPS_REL32      00000000   printf

i.e. there are five undefined symbols, and I'm guessing that the ones
in the first created GOT don't get REL32 relocations emitted.

A little poking in binutils reveals that this is deliberate.  This
comment, from the multigot patch in January:
  /* A pointer to the primary got, i.e., the one that's going to get
     the implicit relocations from DT_MIPS_LOCAL_GOTNO and
     DT_MIPS_GOTSYM.  */
  struct mips_got_info *primary;

There's only one problem - it doesn't work the way that comment
suggests.  got[2] through got[DT_MIPS_LOCAL_GOTNO - 1] are relocated
by l_addr, as local GOT entries.  Then got[DT_MIPS_LOCAL_GOTNO] through
got[DT_MIPS_SYMTABNO - 1] are relocated as global GOT entries.  So
DT_MIPS_SYMTABNO is probably intended to be the number of symbols _used
by the primary GOT_.  Not the total size of .dynsym.

Reading the comments in elfxx-mips.c it looks like this shouldn't
even be happening; the global symbols should end up in the "master
GOT".  And there should be stubs.  But in my tests the stubs don't get
built, and the value in the GOT for getpid points to func0_4663.  And I
can't find code that ought to create them.  So I'm going to have to
punt to Alexandre for details.  A quick potential fix might be changing
DT_MIPS_SYMTABNO.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

---

From: Eric Christopher <[email-masked]>
Date: Tue, 04 Nov 2003 09:53:06 -0800

> http://www.kernel.org/pub/linux/devel/binutils/.

I'm using mainline gcc, but I meant the python-qt sources you were
compiling.

-eric

---

From: Atsushi Nemoto <[email-masked]>
Date: Fri, 07 Nov 2003 01:54:21 +0900 (JST)

>>>>> On Tue, 04 Nov 2003 09:53:06 -0800, Eric Christopher <[email-masked]> said:

echristo> I'm using mainline gcc, but I meant the python-qt sources
echristo> you were compiling.

The link error of phyton-qt was reported by Thiemo Seufer.  I have not
tried it.

My problem is runtime failure, not link error.  So it may be a
different problem.

---
Atsushi Nemoto

---

From: Thiemo Seufer <[email-masked]>
Date: Fri, 7 Nov 2003 17:40:43 +0100

Eric Christopher wrote:
[snip]
> I'm using mainline gcc, but I meant the python-qt sources you were

It was python-qt-3.8 from debian unstable, compiled with
"gcc (GCC) 3.3.2 (Debian)" and binutils
"2.14.90.0.7 20031029 Debian GNU/Linux"

An attempt to link with CVS ld shows the same BFD assertion.

Thiemo

---

From: Eric Christopher <[email-masked]>
Date: Fri, 07 Nov 2003 12:51:49 -0800

> An attempt to link with CVS ld shows the same BFD assertion.

OK. Well, the one machine I have has current sources for gcc and
binutils so I'll be trying that :)

Pointer to a tarball with the sources?

-eric