Subject: Re: first packages for mipsel

From: Florian Lohoff <[email-masked]>
Date: Fri, 6 Apr 2001 13:22:14 +0200

On Fri, Apr 06, 2001 at 01:09:58PM +0200, Florian Lohoff wrote:
> Hi,
> i just uploaded glibc 2.2.2 and db2 for debian-mipsel to incoming ...

BTW: This glibc uses "sysmips" for locking purposes - This means
that this lib would only work on "fixed" 2.4 kernels on ll/sc capable
machines. Its the same as for mips.

There are multiple solutions to this problem:

- ll/sc emulation in the kernel 
  Implementing an ll/sc emulation into the fast path of the illegal instruction
  handler and compile the glibc with ll/sc support.

- repair sysmips
  Sysmips is essentially broken - My fix i have sent to the linux-mips list
  is only a workaround but not a fix. Also there is currentl NO ISA I support
  in the sysmips implementation. I hope to get around to build a proper
  fix this weekend in assembly.

Probably we will do both - An even better solution would be to let the glibc
detect the ISA level or the existance of ll/sc in userspace and replace _test_and_set
with one or the other primitiv - Using sysmips on R3000 or ll/sc less systems.

Flo
-- 
Florian Lohoff                  [email-masked]             +49-5201-669912
     Why is it called "common sense" when nobody seems to have any?

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Fri, 6 Apr 2001 13:43:24 +0200

> with one or the other primitiv - Using sysmips on R3000 or ll/sc less
systems.

What advantage would there be to using sysmips() as opposed
to doing the ll/sc emulation?  It seems to me that the decode path
in the kernel would be just as fast, and there would be a single
"ABI" for all programs - the ll/sc instructions themselves.

            Kevin K.

---

From: Florian Lohoff <[email-masked]>
Date: Fri, 6 Apr 2001 13:58:49 +0200

> "ABI" for all programs - the ll/sc instructions themselves.

I dont actually care that much - I want to have a portable way
through all ISAs as debian should be userspace compatible all the way.

If there is a working ll/sc emulation fine - Currently there is none
so the only way to go TODAY is sysmips.

Flo

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Fri, 6 Apr 2001 14:23:53 +0200

> so the only way to go TODAY is sysmips.

I'll work on one in the background.  It should be pretty straightforward
for the uniprocessor case, but an SMP version will be messier, and
possibly require a platform-dependent hook.  Of course, the same
is true of sysmips()...

            Kevin K.

---

From: Florian Lohoff <[email-masked]>
Date: Fri, 6 Apr 2001 15:28:36 +0200

> possibly require a platform-dependent hook.  Of course, the same

For SMP versions we could currently just put an "#error" in it. There has
to be machine dependent support as the older SGI Challenge have. So dont
worry on that case.

Flo

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Fri, 6 Apr 2001 17:33:23 +0200

> to be machine dependent support as the older SGI Challenge have. So dont

I've just glanced at the unfinished ll emulation code in the 2.4.3 baseline.
It looks like someone was trying to do a full and precise emulation of
the instruction as documented, which is admirable but horribly difficult.
The trick is that one can be aggressive in clearing the lock "flip flop".
In real CPU's for example, the flop is cleared on every exception taken.
Emulating that alone would make for a semi-functional operation,
at least in terms of protecting kernel threads.  One needs to do a bit
more to cover user-level multithreading.  Stricktly speaking, one should
search the entire address space of the executing task and tag all
virtual pages that map to the targeted physical page as being
non-writable, and on a write protect error clear both the ll bit and
the protections.  On a sc, one can either do nothing or proactively
make the pages writable.  I find the notion of the *same* task having
a lock cell virtually aliased to be wildly improbable, and would be
tempted to consider elimitating the reverse translation and only
flagging the page where the ll/sc is taking place.  That would not
be 100% correct emulation, but I would be very surprised to find
any code that would notice.

            Regards,

            Kevin K.

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Fri, 6 Apr 2001 18:40:10 +0200 (MET DST)

> "ABI" for all programs - the ll/sc instructions themselves.

 It was discussed a few times already.  It's ugly and is an overkill for
UP machines -- you take at least two faults for ll/sc emulation and only a
single syscall for TAS. 

 Sysmips() is ugly as well but it's a legacy call -- I proposed
implementing _test_and_set() call which would be the underlying
implementation of the ABI _test_and_set() library call for MIPS I systems
(which should probably be the only atomic operation available to the
userland).  Unfortunately the lack of time prevents me from doing it. 

 At least _test_and_set() has well-defined semantics.  It looks
straightforward as well. 

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email-masked], PGP key available        +

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Fri, 6 Apr 2001 18:48:42 +0200 (MET DST)

> to be machine dependent support as the older SGI Challenge have. So dont

 Yep, DS5800 provides hw for atomic ops as well -- a write to a specific
location makes the next memory read and write cycle atomic.

 The kernel may choose whatever way is needed for particular hw (if we
ever support it).

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Sat, 7 Apr 2001 11:05:56 +0200

> UP machines -- you take at least two faults for ll/sc emulation and only a

Depends on your point of view.  Syscalls will be faster than
emulation on processors without LL/SC support, certainly,
but much slower than just executing the instructions on processors
that do support LL/SC.  Intuitively, emulation would be roughly
2x worse for an R3K, but a syscall will be 10-100 times worse
for an R4K.  If we gave an equal weight to both families, that
would argue in favor of LL/SC emulation - and working for
MIPS Technologies (where all our designs for the past
10 years have supported LL/SC) I would consider equal
weighting to be very generous!  ;-)

I've seen the hybrid proposal of having libc determine the LL/SC
capability of the processor and either executing the instructions
or doing the syscall as appropriate. While that would allow
near-optimal performance on all systems, I find it troublesome,
both on the principle that the OS should conceal hardware
implementation details from the user, and on the practical basis
that glibc is the last place I would want to put more CPU-specific
cruft.  But reasonable people can disagree.

            Kevin K.

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Sat, 7 Apr 2001 12:55:24 +0200 (MET DST)

> weighting to be very generous!  ;-)

 You are right, of course.  That's why glibc contains two versions of
_test_and_set() code.  If compiled for MIPS I, glibc uses a syscall
(currently sysmips()), while for MIPS II and higher it uses inline
assembly code which makes use of LL/SC.

 That's exactly the way glibc does CPU-model-specific code for other
archs.

> cruft.  But reasonable people can disagree.

 I don't like run-time detection either.  The compile-time choice is
sufficient enough.  The _test_and_set() library function already hides
implementation details from the user.

---

From: Florian Lohoff <[email-masked]>
Date: Sat, 7 Apr 2001 17:29:00 +0200

>  That's exactly the way glibc does CPU-model-specific code for other

The problem is that it is compile time - I would like to have
a runtime version - Just export the existance of ll/sc to
userspace somewhow.

> implementation details from the user.

It isnt sufficent as we need a glibc beeing able to run on old
decstations AND on newer machines like the Lasat machine
which has an R5000.

Flo

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Mon, 9 Apr 2001 11:48:40 +0200 (MET DST)

> a runtime version - Just export the existance of ll/sc to

 I can't see any problem.  If you want to go for speed -- compile glibc
for MIPS II (assuming you have a MIPS II or better CPU).  If you want to
go for compatibility -- compile glibc for MIPS I (it will still run on
MIPS II or better CPUs, although suboptimally; actually, it won't be the
worst suboptimality then).

 Similarly -- an Alpha EV56-compiled glibc will not run on an EV4 CPU,
while an EV4-compiled glibc will run on an EV56 CPU, although
suboptimally. 

> decstations AND on newer machines like the Lasat machine

 A MIPS-I-compiled glibc will run on any MIPS CPU, using a kernel
_test_and_set() emulation (how kernel emulates _test_and_set() is not the
point here -- it has to be done in an MP-safe way, if necessary; LL/SC is
the most likely candidate for MIPS II or better kernels).  A
MIPS-II-compiled glibc will run on any MIPS II or better CPU using LL/SC
directly.  Similar for MIPS-III-compiled glibc, etc.

 A MIPS-II-compiled kernel will still provide a _test_and_set() emulation
to keep the ABI consistent (e.g. for statically linked MIPS I binaries).

 Still any problems?  I will happily clarify any remaining issues.