Subject: [patch] 2.4, head: PAGE_SHIFT changes break glibc

From: "Maciej W. Rozycki" <[email-masked]>
Date: Fri, 21 Nov 2003 18:33:37 +0100 (CET)

Ralf,

 Recent changes made to <asm/page.h> break a build of glibc 2.2.5 for me.  
Compilation bails out due to PAGE_SHIFT being undeclared -- glibc pulls it
as it uses PAGE_SIZE in linuxthreads/internals.h.  The PAGE_SHIFT macro
depends on configuration now (I use an empty cofinguration for glibc
headers, hence the error) and thus it'd better be simply private to the
kernel.  Glibc will then use sysconf(_SC_PAGE_SIZE) which now better
reflects actual configuration of the system it's run on.

 Here's a patch that limits PAGE_SIZE to the kernel scope.  If there's any
other program that needs PAGE_SIZE, it should be converted to
sysconf(_SC_PAGE_SIZE) as well.

 OK to apply?

 Additionally, I think we should also implement the getpagesize syscall to
benefit statically linked programs (and make glibc use it like for other
platforms that use variable page sizes).  Finally, I'm not sure such a
noticeable change was a good move in these late days of 2.4...

  Maciej

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email-masked], PGP key available        +

linux-include-mips-2.4.22-20031115-mips-page_size.patch
diff -up --recursive --new-file linux-include-mips-2.4.22-20031115.macro/include/asm-mips/page.h linux-include-mips-2.4.22-20031115/include/asm-mips/page.h
--- linux-include-mips-2.4.22-20031115.macro/include/asm-mips/page.h	2003-10-17 02:57:33.000000000 +0000
+++ linux-include-mips-2.4.22-20031115/include/asm-mips/page.h	2003-11-21 09:29:52.000000000 +0000
@@ -12,6 +12,8 @@
 
 #include <linux/config.h>
 
+#ifdef __KERNEL__
+
 /*
  * PAGE_SHIFT determines the page size
  */
@@ -27,8 +29,6 @@
 #define PAGE_SIZE	(1L << PAGE_SHIFT)
 #define PAGE_MASK	(~(PAGE_SIZE-1))
 
-#ifdef __KERNEL__
-
 #ifndef __ASSEMBLY__
 
 #include <asm/cacheflush.h>
diff -up --recursive --new-file linux-include-mips-2.4.22-20031115.macro/include/asm-mips64/page.h linux-include-mips-2.4.22-20031115/include/asm-mips64/page.h
--- linux-include-mips-2.4.22-20031115.macro/include/asm-mips64/page.h	2003-10-17 02:57:34.000000000 +0000
+++ linux-include-mips-2.4.22-20031115/include/asm-mips64/page.h	2003-11-21 09:30:10.000000000 +0000
@@ -11,6 +11,8 @@
 
 #include <linux/config.h>
 
+#ifdef __KERNEL__
+
 /*
  * PAGE_SHIFT determines the page size
  */
@@ -26,8 +28,6 @@
 #define PAGE_SIZE	(1UL << PAGE_SHIFT)
 #define PAGE_MASK	(~(PAGE_SIZE-1))
 
-#ifdef __KERNEL__
-
 #ifndef __ASSEMBLY__
 
 #include <asm/cacheflush.h>

---

From: Ralf Baechle <[email-masked]>
Date: Fri, 21 Nov 2003 19:50:35 +0100

> reflects actual configuration of the system it's run on.

Interesting.  IA-64 does the same thing, for example.  Wonder why they
seem to be able to get away with that.  At the very least including the
kernel header file may pick up a wrong value for PAGE_SHIFT.

> sysconf(_SC_PAGE_SIZE) as well.

Yes, please go ahead.

> platforms that use variable page sizes).

The kernel is already passing AT_PAGESZ to ELF binaries.  Wouldn't that
be sufficient?  Currently it's passing the largest supported page size,
that is 64k.  However this constant is always passed even when a smaller
page size is configured.

>  Finally, I'm not sure such a

TLB reloads have been shown to be a major performance problem; this is an
not yet completed attempt to improve the situation so people don't need
to go for crude hacks such as wired TLB entires and similar.

Other parts of improvments such as hugetlbfs are available in 2.6 only
anyway.  I'm also thinking of changing the pagetable structure back to
the aggressivly optmized thing we were using before Linux 2.0.14 - but
certainly not for 2.4 - too intrusive, as you say.  With most MIPS users
being embedded users I still expect 2.4 to live for quite a while -
certainly longer than I'd like to see ...

  Ralf

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Fri, 21 Nov 2003 21:07:58 +0100 (CET)

> kernel header file may pick up a wrong value for PAGE_SHIFT.

 I think people (this may include distributors as well) build glibc with
Linux headers fully configured or, worse yet, with whatever happens to be
grabbed by gcc as <linux/...> and <asm/...> (i.e. usually headers from the
previous installation).  So in this case PAGE_SHIFT gets defined somehow,
although not necessarily reflecting what will be true for the host system.

 The question is which one is the *right* full configuration.  I think all
are valid, so none is the single right, so I chose not to define kernel
configuration macros for the userland -- the userland ABI shouldn't depend
on a particular kernel configuration.  I hope this issue will disappear
automatically once the kernel header space gets divided into an internal
part and one forming the ABI.

 If you look at my spec file for glibc, you'll see it only does:

echo "#define AUTOCONF_INCLUDED" > linux/include/linux/autoconf.h

to get kernel headers working.

> be sufficient?  Currently it's passing the largest supported page size,

 Well, AFAICS in glibc it's ld.so that is responsible for interpreting the
auxiliary vector -- see _dl_aux_init() in elf/dl-support.c.  If the
dynamic linker isn't run (which is the usual case for static binaries,
although calling dlopen() from them might complicate things), the
dl_pagesize variable remains set to zero.  Please prove me wrong if I am
missing anything.

> that is 64k.  However this constant is always passed even when a smaller

 Are you sure?  I can see create_elf_tables() in fs/binfmt_elf.c sets 
AT_PAGESZ to ELF_EXEC_PAGESIZE, which, in turn, is set in 
include/asm-mips/elf.h to PAGE_SIZE.  Which is the currently used page 
size and probably the optimal solution.

> to go for crude hacks such as wired TLB entires and similar.

 OK, but the safe (and practiced in the mainline) procedure is to test
with the development version and do a backport if necessary.  Well, it
might not work that well this time, as apparently I'm the only one to come
upon such oddities ;-) and, as you probably know, I haven't switched to
2.6, yet.

> certainly longer than I'd like to see ...

 As I wrote, such changes may get backported if there's a need and a
volunteer to do the task.  I won't support 2.4 for a long time, yet -- I
just want to upgrade the toolchain before I upgrade the kernel.  After
then I may only backport some DECstation driver fixes.

 BTW, can you imagine how huge gcc 3.4 is? -- for the i386 it needs 1GB of 
disk space and plenty of hours to be built on a reasonable system.  For 
MIPS I suppose the disk space needed will be yet larger and building may 
require a few days...

  Maciej

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Tue, 25 Nov 2003 16:27:49 +0100 (CET)

> size and probably the optimal solution.

 After rebuilding glibc (2.2.5) with the patch applied, the following
program:

#include <stdio.h>
#include <unistd.h>

int main(void)
{
	printf("%u\n", getpagesize());
	return 0;
}

prints "4096" if dynamically linked and "65536" if statically linked on my
system, as expected.

---

From: Ralf Baechle <[email-masked]>
Date: Wed, 26 Nov 2003 00:24:39 +0100

> prints "4096" if dynamically linked and "65536" if statically linked on my

Guess we'll need a solution along the lines of
unix/sysv/linux/sparc/sparc32/getpagesize.c then ...

  Ralf

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Wed, 26 Nov 2003 01:09:16 +0100 (CET)

> unix/sysv/linux/sparc/sparc32/getpagesize.c then ...

 I suppose so, although being not that fond of insane numbers of syscalls
I wonder how sysdeps/unix/sysv/linux/ia64/getpagesize.c gets away with
static binaries...  Perhaps we should ask glibc hackers?

---

From: Daniel Jacobowitz <[email-masked]>
Date: Wed, 26 Nov 2003 12:02:28 -0500

> static binaries...  Perhaps we should ask glibc hackers?

Look at elf/dl-support.c:_dl_aux_init? dl_pagesize should end up
initialized for static binaries too.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Thu, 27 Nov 2003 13:45:00 +0100 (CET)

> initialized for static binaries too.

 Thanks for the hint.  I can see _dl_aux_init() gets indeed called from
__libc_start_main() in my i386/Linux libc-start.o binary (in libc.a), but
it does not in my MIPSel/Linux one.  So there must be a bug somewhere in
2.2.5, perhaps fixed later.  I'll have a look at it.  Once fixed, I
guess we should choose the IA64 way.

 After a brief look at the sources I suspect sysdeps/mips/elf/ldsodefs.h
overrides sysdeps/unix/sysv/linux/ldsodefs.h -- if that's the case, the
bug still exists in the trunk.  I'll work on a fix later, probably
tonight.

---

From: Daniel Jacobowitz <[email-masked]>
Date: Thu, 27 Nov 2003 12:37:14 -0500

> bug still exists in the trunk.  I'll work on a fix later, probably

_dl_aux_init ought to be called from __libc_start_main in
sysdeps/generic/libc-start.c.  In current sources I can't see any way
for that to go wrong on MIPS (but my MIPS board is off right now so I
haven't tried it)...

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Thu, 27 Nov 2003 20:31:18 +0100 (CET)

> for that to go wrong on MIPS (but my MIPS board is off right now so I

 I did a build of 2.2.5 and my suspicion got confirmed -- the exact 
command line to build libc-start.o is as follows:

mipsel-linux-gcc ../sysdeps/generic/libc-start.c -c -O2 -Wall -Winline
-Wstrict-prototypes -Wwrite-strings -pipe -fomit-frame-pointer -g0 -O99
-fomit-frame-pointer -D__USE_STRING_INLINES -I../include -I. -I..
-I../libio -I../sysdeps/mips/elf -I../linuxthreads/sysdeps/unix/sysv/linux
-I../linuxthreads/sysdeps/pthread -I../sysdeps/pthread
-I../linuxthreads/sysdeps/unix/sysv -I../linuxthreads/sysdeps/unix
-I../linuxthreads/sysdeps/mips -I../sysdeps/unix/sysv/linux/mips
-I../sysdeps/unix/sysv/linux -I../sysdeps/gnu -I../sysdeps/unix/common
-I../sysdeps/unix/mman -I../sysdeps/unix/inet -I../sysdeps/unix/sysv
-I../sysdeps/unix/mips -I../sysdeps/unix -I../sysdeps/posix
-I../sysdeps/mips/mipsel -I../sysdeps/mips/fpu -I../sysdeps/mips
-I../sysdeps/wordsize-32 -I../sysdeps/ieee754/flt-32
-I../sysdeps/ieee754/dbl-64 -I../sysdeps/ieee754 -I../sysdeps/generic/elf
-I../sysdeps/generic -nostdinc -isystem
/usr/lib/gcc-lib/mipsel-linux/2.95.4/include -isystem
/home/macro/src/redhat/BUILD/glibc-2.2.5/linux/include -D_LIBC_REENTRANT
-include ../include/libc-symbols.h -DPIC -DHAVE_INITFINI -o libc-start.o

i.e. it's sysdeps/mips/elf/ldsodefs.h that gets included as a result of
"#include <ldsodefs.h>", not sysdeps/unix/sysv/linux/ldsodefs.h.  Then
sysdeps/mips/elf/ldsodefs.h uses "#include <sysdeps/generic/ldsodefs.h>"  
in 2.2.5, and "#include_next <ldsodefs.h>" in the trunk, so I'll just 
rebuild 2.2.5 with this change and consider the trunk fixed.

  Maciej