Subject: anybody tried NPTL?

From: Jun Sun <[email-masked]>
Date: Wed, 4 Aug 2004 15:29:36 -0700

I am looking into porting NPTL to MIPS.  Just curious if
anybody has tried this before.

I notice there was a discussion about the ABI extension
for TLS (thread local storage) support.  Before that support
becomes a reality it seems one can still use NPTL with 
the help of additional system calls.

A rough search of latest glibc source shows there is
zero MIPS code for nptl.  A couple of other arches
are missing as well (such as ARM)

Jun

---

From: Kumba <[email-masked]>
Date: Wed, 04 Aug 2004 21:08:56 -0400

> are missing as well (such as ARM)

All I've heard about this is that some kernel changes are (still?) 
needed, then just the glibc support along w/ TLS (Maybe compiler support?).

I believe I heard reports that the glibc people were looking to 
deprecate linuxthreads within a another release or two (but don't know 
specifics or anything), so it sounds like NPTL should be something to 
get working.

--Kumba

-- 
"Such is oft the course of deeds that move the wheels of the world: 
small hands do them because they must, while the eyes of the great are 
elsewhere."  --Elrond

---

From: Jun Sun <[email-masked]>
Date: Thu, 5 Aug 2004 10:14:51 -0700

> needed, then just the glibc support along w/ TLS (Maybe compiler support?).

TLS support requires ABI change, which involves work in gcc and binutils.
At current stage I think only a few arches have added TLS support.
MIPS is definitely not one of them.  Does anybody know about the current
status, for MIPS and other arches?

I think the ABI change and TLS support might take a long time to 
be ready.  It appears meanwhile NPTL can run without TLS, but would
need a couple of additional system calls that get and set thread
local area.

> specifics or anything), so it sounds like NPTL should be something to 

That surely puts some urgency on this matter. :)

Jun

---

From: Ralf Baechle <[email-masked]>
Date: Fri, 6 Aug 2004 04:03:04 +0200

> specifics or anything), so it sounds like NPTL should be something to 

Threading support means breaking the ABI compatibility.  There are other
issues that can best be solved by breakin the ABI which is why this is
stretching out.

  Ralf

---

From: Dominic Sweetman <[email-masked]>
Date: Thu, 19 Aug 2004 15:17:10 +0100

Jun Sun,

> the help of additional system calls.

Well, this is an area of substantial interest to MIPS Technologies.
We are working on our multi-threading extension to the MIPS
architecture, and one of our longer-term aims is to achieve really
good NPTL performance.

As you said, this motivates a revision of the ABI.  Any incompatible
change to the ABI is painful, since everything has to be recompiled;
so our reasoning at the moment is that we might as well be radical...

MIPS Technologies would write up the definition; make and circulate
changes to the compiler, binutils and debugger; and make and circulate
supporting changes to the kernel and glibc.  We're aware this is only
a part of the problem, so we do need to figure out what will attract
community approval.

A few years ago I was volubly arguing in favour of keeping o32 until
we could move to n32/n64.  But times change: SGI compatibility
no longer seems important, and Linux on 32-bit MIPS CPUs needs
long-term support.

In many ways it's easier to get away from the ingenious but
troublesome stack-structure-based calling convention.  The "stack
structure" trick was invented so that o32 could work without function
prototypes - but function prototypes are now required, and something
with fixed-size arguments is simpler.  Something like the old
Cygnus/WRS EABI proposal, in fact...

So we're proposing:

o The register name<->number mapping is that of n64.

o Calling convention: register-, not slot-based. Each argument is
  represented by a register value. Arguments 0-7 travel in registers
  a0-7 (or fa0-7 as required for floating point types). If there are
  more than eight arguments, further ones are formed as if put in a
  register and then saved on the stack into a 64-bit slot (more than 8
  arguments is rare enough that we can afford to standardise on the
  big slots).
  
o Use floating point registers for double and float arguments, and
  integer registers for all integer/pointer values which will
  fit. Larger or structured data items are implicitly passed by
  reference: to maintain pass-by-value semantics, the compiler uses a
  copy-on-write trick if software writes a by-reference argument (or
  takes its address).  I'm told gcc is happy enough to do that.

o The return value comes back in two registers, with the second
  return-register used only when the return value consists of two
  scalars (ie a complex or double-precision number). [Folklore insists
  this is essential for Fortran support of complex numbers, and I
  don't want to fight folklore].

  All other non-scalar return values are returned via a pointer
  specified by the caller as an implicit first argument.

o Reserved registers: all the traditional ones. But now:

  - gp will be the GOT pointer in Linux, and should be defined as
    saved (ie a function must preserve values in this registers, which
    means it will need to save-and-restore the register if it is
    written locally).
    
  - we'll define some other register as a per-thread data pointer.

Some details are still to be worked out.  But do you think this is on
the right lines?  And who would like to take an active part in
specifying or reviewing?

--
Dominic Sweetman
MIPS Technologies

---

From: "Alec Voropay" <[email-masked]>
Date: Thu, 19 Aug 2004 18:31:45 +0400

> architecture, and one of our longer-term aims is to achieve really

 Sorry for a bit offtopic..., as far as I remember, the Windows NT
MIPS edition has a working multithread implementation. Is this
implementation very copyrighted and is it possible to use something
ftom there for the NPTL implementation ?

--
-=AV=-

---

From: David Daney <[email-masked]>
Date: Thu, 19 Aug 2004 09:01:25 -0700

> the right lines?  And who would like to take an active part in
All of this sounds good to me.  However my current concerns are how to
make my code run on a mips32[r2] core with no floating point.  We are
using several different systems with variations of this cpu type.

So for me, making sure that a soft-float variant of the ABI is well
specified is also important.  I suppose it would be to treat
float/double values as appropriate encoding of 32/64 bit integer values.

David Daney.

---

From: Jun Sun <[email-masked]>
Date: Thu, 19 Aug 2004 15:16:46 -0700

> the right lines?  And who would like to take an active part in

I am not against any of those.  However, from NPTL implementation
point of view I really just care about the per-thread register.
What is the motivation of other changes?  Taking this chance to make
things all right in one shot?

Jun

---

From: Dominic Sweetman <[email-masked]>
Date: Thu, 19 Aug 2004 23:07:03 -0700

Alec,

> ftom there for the NPTL implementation ?

It's Microsoft, so yes it will be copyrighted and it would be better
if our staff didn't even look at it.

--
Dominic Sweetman
MIPS Technologies

---

From: Dominic Sweetman <[email-masked]>
Date: Thu, 19 Aug 2004 23:19:27 -0700

> float/double values as appropriate encoding of 32/64 bit integer values.

Exactly - that's certainly what we do with o32 -msoft-float.  I don't
think there are even any corner cases to worry about.

We've recently made some major improvements to our soft maths library
to make it many times faster, but still IEEE-compliant.  It was
originally written by Algorithmics - a GPL'd version is used in the
Linux/MIPS kernel to handle FPU exceptions.  I should probably check
before saying for certain, but I think we will distribute this under
straight GPL conditions.

If we had the time to do it, I guess we should put it on one of the OS
project sites... but I'll cc: Nigel Stephens (who wrote it).  If it
might be of use to you (which seems quite likely) perhaps we could do
a deal to get it on a site somewhere?  Let me know if you'd like sight
of it.

--
Dominic

---

From: Thiemo Seufer <[email-masked]>
Date: Fri, 20 Aug 2004 15:12:10 +0200

Dominic Sweetman wrote:
[snip]
>   a0-7 (or fa0-7 as required for floating point types).

This suggests to have no fp temporaries left: fv0-1, fa0-7, fs0-5.
Is this intentional?

Thiemo

---

From: Dominic Sweetman <[email-masked]>
Date: Fri, 20 Aug 2004 14:46:11 +0100

Jun Sun ([email-masked]) writes:

> > [large expanse of new ABI stuff omitted]
>

Yes, and I did kind of cover this in the large expanse of stuff:

> > As you said, this motivates a revision of the ABI.  Any
> > incompatible change to the ABI is painful, since everything has to
> > be recompiled; so our reasoning at the moment is that we might as
> > well be radical...

o32 puts only four arguments in registers, which is less than optimal,
and continually reloads the GOT pointer (which is not defined as a
saved register in o32).

n32 (despite its name) runs exclusively on 64-bit CPUs.

So yes, we have strong reasons for making a larger change to the ABI,
and knowing how difficult it is...

> However, from NPTL implementation point of view I really just care
> about the per-thread register.

I guess our main message was that we felt it would be a mistake just
to add a thread register to o32 (which produces a substantially
incompatible new ABI anyway).

Until that all works, what we had in mind is that we'd do NPTL over
o32 by defining a system call to return a per-thread ID which is or
can be converted into a per-thread data pointer.  We suspected that
NPTL's per-thread-data model allows the use of cunning macros or
library functions to make that look OK.

Ought we to go further and see exactly how that can be done?

--
Dominic

---

From: Dominic Sweetman <[email-masked]>
Date: Fri, 20 Aug 2004 17:52:44 +0100

> This suggests to have no fp temporaries left: fv0-1, fa0-7, fs0-5.

No, there are lots really...

Long, long ago early MIPS CPUs had to use FP registers in pairs to get
FP doubles, so had only 16 effective registers.

But there are 32 real double-precision FP registers in all known
modern MIPS CPUs (well, all known pretty old ones, really) once you
turn off the backward-compatibility bit in the status register.  I
forgot to mention that... (or I could claim, weasel words, that this
is implied by the "name<->number mapping of n64").

--
Dominic

---

From: Ralf Baechle <[email-masked]>
Date: Mon, 23 Aug 2004 14:28:43 +0200

> ftom there for the NPTL implementation ?

In addition to what Dom has already answered - there are very significant
differences between the multithreading as implemented in the Windows OS
family and the varioius threading implementations for Linux like classic
libpthreads, Linuxthreads, NPTL, Mozilla and more.  If we legally could
look at MS's code I'd not expect to find much useful for us there ...

  Ralf

---

From: Daniel Jacobowitz <[email-masked]>
Date: Mon, 23 Aug 2004 09:28:53 -0400

> incompatible new ABI anyway).

Completely agree...

> Ought we to go further and see exactly how that can be done?

It shouldn't be at all hard.  The way NPTL's __thread support works,
the only things that should have to know where the TLS base is are
(A) GCC, so it can load it and (B) GDB, via some new ptrace op.  I
don't know if you'd want to open-code the syscall or take the overhead
of a function call.  Ralf had some ideas?

-- 
Daniel Jacobowitz

---

From: "Alec Voropay" <[email-masked]>
Date: Mon, 23 Aug 2004 19:09:31 +0400

Ralf Baechle [mailto:[email-masked]] wrote:

> In addition to what Dom has already answered - there are very 
> significant differences between the multithreading as
> implemented in the  Windows OS family and the varioius
> threading implementations for Linux  like classic
> libpthreads, Linuxthreads, NPTL, Mozilla and more.
> If we legally could look at MS's code I'd not expect to find
> much useful for us there ...

 OK, OK. You are right. However, as it is known, there is at
least one project "to bridge" Win32/multithread and *NIX :
 WINE.
http://winehq.com/site/docs/wine-devel/x3398

 Yes, the Win32/MIPS API (and ABI) is dead, but MIPS/multithreading
lives in the WindowsCE/MIPS HPCs.

 Unfortunately, I can't find any details about Win32/MIPS
implementation.

--
-=VA=-

---

From: Ralf Baechle <[email-masked]>
Date: Mon, 23 Aug 2004 19:12:57 +0200

> of a function call.  Ralf had some ideas?

Thiemo and have been compiling various pieces of code with different
gcc versions trying to find the best possible register for that purpose.
We used code bloat as (weak ...) indicator for register pressure.  It
turned out that $t9 was the best choice for all tested compiler versions;
thanks to the much improved register allocation of newer gcc the choice
of a particular register made far less difference on recent compilers
than on older compilers.

I've also implemented a fast system call for reading the thread registers.
Benchmarks did show that to have about half the latency of a regular
syscall; the hope was if gcc was doing clever optimization that overhead
would effectivly become zero.

I was favoring this low-overhead syscall approach because it would avoid
the loss of a register thus leaving performance of non-threaded code
unchanged but other developers generally favor the permanent allocation
of $t9 as a thread register.

Other crazy ideas did include a per-thread mapping containing the thread
pointer - and possibly more information in the future.

Finally one of the ideas was using one of $k0 / $k1 as thread pointer.
This would require changes to the exception handlers; any extra
instruction in the TLB refill handler would be particularly painful.

On the positive side if we had multiple register sets on a MIPSxx V2
processor we could exploit that to get rid of this overheade and do
other nice optimizations for TLB reload also.  Unfortunately these
register sets are optional feature of the architecture only.

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Mon, 23 Aug 2004 19:19:21 +0200

>  Unfortunately, I can't find any details about Win32/MIPS

That's because for Windows on x86 there are tons of important binary-only
applications available and Linux would have to be bought if they're
available at all.  On MIPS we're not in this dependency situation, so
we hardly care about Win32.

  Ralf

---

From: Jun Sun <[email-masked]>
Date: Mon, 23 Aug 2004 10:37:31 -0700

On Mon, Aug 23, 2004 at 09:28:53AM -0400, Daniel Jacobowitz wrote:
> (A) GCC, so it can load it and (B) GDB, via some new ptrace op. 

Are you implying one can implement TLS support without changing O32
ABI?  Interesting...

I know Boris Hu has tried to implemented NPTL with another approach which
does not rely on TLS support (use "--without-tls").  According to him
this approach is getting harder these days.

Jun

---

From: Daniel Jacobowitz <[email-masked]>
Date: Mon, 23 Aug 2004 13:44:47 -0400

> of $t9 as a thread register.

Personally, I favor doing the low-overhead syscall for o32 and then
moving to the new ABI that MIPS is talking about with a thread
register.  I'm not sure what to do about n32/n64.

> pointer - and possibly more information in the future.

Does MIPS have an efficient way to do this for SMP?

> register sets are optional feature of the architecture only.

That's more or less what was talked about for ARM v6.

-- 
Daniel Jacobowitz

---

From: Ralf Baechle <[email-masked]>
Date: Mon, 23 Aug 2004 21:13:19 +0200

On Mon, Aug 23, 2004 at 01:44:47PM -0400, Daniel Jacobowitz wrote:

> register.

I was always wondering how far gcc could optimize code to minimize this
special system call.  After all on Alpha something similar PAL-code based
was the method of choice.

> I'm not sure what to do about n32/n64.

I believe N32 / N64 are not widespead enough yet to be a big roadblock
for moving to new ABIs.  Whoever disagress should better speak up soon :-)

If we deciede to move to something entirely different than the existing
ABIs in the future will be able to support compatibility the same way
we're already doing.

> Does MIPS have an efficient way to do this for SMP?

It can be done making the TLB fault for that page about as expensive as
a null syscall.

> That's more or less what was talked about for ARM v6.

I'm unARMed here ;-)

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Mon, 23 Aug 2004 21:25:20 +0200

> this approach is getting harder these days.

The whole TLS pointer thing is about making TLS more efficient.  If
you wanted to use TLS without any kernel changes you could do that
based on the THREAD result value of pthread_create or something like
that.  It'd work but it'd also not be terribly efficient ...

  Ralf