Subject: Build failure for R3000 DECstation

From: Jan-Benedict Glaw <[email-masked]>
Date: Sun, 12 Nov 2000 21:00:49 +0100

Hi!

I see this build failure:

mipsel-linux-gcc -D__KERNEL__ -I/usr/src/mipsel/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -G 0 -mno-abicalls -fno-pic -mcpu=r3000 -mips1 -pipe     -c -o sysmips.o sysmips.c
sysmips.c: In function `sys_sysmips':
sysmips.c:109: warning: implicit declaration of function `syscall_trace'
{standard input}: Assembler messages:
{standard input}:337: Error: opcode requires -mips2 or greater `ll'
{standard input}:339: Error: opcode requires -mips2 or greater `sc'
{standard input}:340: Error: opcode requires -mips2 or greater `beqzl'
{standard input}:341: Error: opcode requires -mips2 or greater `ll'
make[1]: *** [sysmips.o] Error 1

jbglaw@bootserver:/usr/src/mipsel/linux/arch/mips/kernel$ mipsel-linux-gcc -v
Reading specs from /usr/lib/gcc-lib/mipsel-linux/egcs-2.90.29/specs
gcc version egcs-2.90.29 980515 (egcs-1.0.3 release)

MfG, JBG

-- 
Fehler eingestehen, Größe zeigen: Nehmt die Rechtschreibreform zurück!!!
/* Jan-Benedict Glaw <[email-masked]> -- +49-177-5601720 */
keyID=0x8399E1BB fingerprint=250D 3BCF 7127 0D8C A444 A961 1DBD 5E75 8399 E1BB
     "insmod vi.o and there we go..." (Alexander Viro on linux-kernel)

---

From: Harald Koerfgen <[email-masked]>
Date: Mon, 13 Nov 2000 22:03:14 +0100 (CET)

> {standard input}:341: Error: opcode requires -mips2 or greater `ll'

Something like 

--- snip here ---
diff -ruN /nfs/cvs/linux-2.3/linux/arch/mips/kernel/sysmips.c
linux/arch/mips/kernel/sysmips.c
--- /nfs/cvs/linux-2.3/linux/arch/mips/kernel/sysmips.c Sun Nov 12 16:05:54 2000
+++ linux/arch/mips/kernel/sysmips.c    Mon Nov 13 19:35:46 2000
@@ -73,13 +73,16 @@
 
        case MIPS_ATOMIC_SET: {
                unsigned int tmp;
-
+#ifndef CONFIG_CPU_HAS_LLSC
+               unsigned long flags;
+#endif
                p = (int *) arg1;
                errno = verify_area(VERIFY_WRITE, p, sizeof(*p));
                if (errno)
                        return errno;
                errno = 0;
 
+#ifdef CONFIG_CPU_HAS_LLSC             
                __asm__(".set\tpush\t\t\t# sysmips(MIPS_ATOMIC, ...)\n\t"
                        ".set\tnoreorder\n\t"
                        ".set\tnoat\n\t"
@@ -100,6 +103,12 @@
                        : "=&r" (tmp), "=o" (* (u32 *) p), "=r" (errno)
                        : "r" (arg2), "o" (* (u32 *) p), "2" (errno)
                        : "$1");
+#else
+               save_and_cli(flags);
+               errno |= __get_user(tmp, p);
+               errno |= __put_user(arg2, p);
+               restore_flags(flags);
+#endif
 
                if (errno)
                        return -EFAULT;
--- snip here ---

should fix this. However, this patch is untested and should be considered as an
RFD. For unknown reasons my R3000 DECstation doesn't grok elf2ecoffed kernels
any more. 

-- 
Regards,
Harald

---

From: Ralf Baechle <[email-masked]>
Date: Mon, 13 Nov 2000 10:47:35 +0100

> make[1]: *** [sysmips.o] Error 1

The sysmips(MIPS_ATOMIC_SET, ...) implementation used to be completly broken.
I fixed it for CPUs with ll/sc and left the part with ll/sc to others.

Obviously none of them seemed to care so now I'm doing the quick fix.
Frankly, a syscall which shouldn't be used doesn't deserve more attention ...

  Ralf

---

From: Harald Koerfgen <[email-masked]>
Date: Tue, 14 Nov 2000 22:30:17 +0100 (CET)

> Frankly, a syscall which shouldn't be used doesn't deserve more attention ...

Well, it seems as if there are people with a different opinion. Fresh from the
glibc CVS (libc/sysdeps/unix/sysv/linux/mips/sys/tas.h):

# if (_MIPS_ISA >= _MIPS_ISA_MIPS2)

_EXTERN_INLINE int
_test_and_set (int *p, int v) __THROW
{
[ll/sc implementation snipped]
}

# else /* !(_MIPS_ISA >= _MIPS_ISA_MIPS2) */

_EXTERN_INLINE int
_test_and_set (int *p, int v) __THROW
{
  return sysmips (MIPS_ATOMIC_SET, (int) p, v, 0);
}

# endif /* !(_MIPS_ISA >= _MIPS_ISA_MIPS2) */

Personally I like this more than a kernel ll/sc emulation. A syscall is likely
to be faster than at least two illegal instruction exceptions. If you're
concerned about binary compatibilty, the syscall should work on ISA>=2 CPUs as
well.

-- 
Regards,
Harald

---

From: Ralf Baechle <[email-masked]>
Date: Wed, 15 Nov 2000 00:41:22 +0100

> glibc CVS (libc/sysdeps/unix/sysv/linux/mips/sys/tas.h):

I know.  Otherwise I'd have plain killed this syscall rsn.

> concerned about binary compatibilty, the syscall should work on ISA>=2 CPUs as

The idea is to punish the least widespread architecture and this are the
non-ll/sc CPUs.  Just like Linux is no longer performing optimally on x86.

In any case, for uniprocessor non-ll/sc machines there is also a better
solution availble with no syscalls at all.  It's easy to implement, just
use the fact that any exception will change the values of k0/k1.  That of
course breaks silently on SMP.

glibc 2.2 calls this sysmips(). very often so optimizing them is fairly
important ...

  Ralf

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Wed, 15 Nov 2000 01:58:21 +0100 (MET)

> course breaks silently on SMP.

 Can you guarantee it???  Well I can guarantee k0 and k1 won't change when
least expected. ;-)  AFAIK, the only fact guaranteed is that exception
handlers do not preserve the values of the scratch registers, but it does
not mean the last value written there is always different from what was
there upon a handler's entry... 

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email-masked], PGP key available        +

---

From: Ralf Baechle <[email-masked]>
Date: Wed, 15 Nov 2000 02:43:58 +0100

> there upon a handler's entry... 

Make that change k0 to a non-zero value.  So a R3000 UP spinlock can look
like:

	move	k0, zero
	li	t0, 1
0:	sw	t0, spin
	bnez	k0, 0b

	[critical section]

	sw	zero, spin

  Ralf

(Who should write thousant times ``I shall not post with a phone in my hand'')

---

From: Jun Sun <[email-masked]>
Date: Tue, 14 Nov 2000 19:13:08 -0800

> (Who should write thousant times ``I shall not post with a phone in my hand'')

You should not post when you are sleepy. :-)

The above code does not work - the loop is infinite as nobody will set
k0 to 0 again if it is not zero for the first time.

In fact, I don't think you can perform automic operation ONLY based on
the knowledge whether a context switch has happened during a specified
period.  (It should be interesting to see if we can actually "prove"
it.)

I also doubt if k0 is absolutely non-zero after a context ...

Jun

---

From: Ralf Baechle <[email-masked]>
Date: Wed, 15 Nov 2000 04:16:57 +0100

> I also doubt if k0 is absolutely non-zero after a context ...

If not the vendor of some R3k based product is in pretty deep sh*t :-)

  Ralf

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Wed, 15 Nov 2000 12:18:57 +0100 (MET)

> Make that change k0 to a non-zero value.  So a R3000 UP spinlock can look

 Great! -- I haven't thought of such a solution.  I'll prepare some code
and see whether there are no races.  It should work fine, indeed.

> (Who should write thousant times ``I shall not post with a phone in my hand'')

 ;-)

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Wed, 15 Nov 2000 12:21:22 +0100 (MET)

> I also doubt if k0 is absolutely non-zero after a context ...

 We might force it if we need to...

---

From: Harald Koerfgen <[email-masked]>
Date: Wed, 15 Nov 2000 20:46:15 +0100 (CET)

> course breaks silently on SMP.

Please, we do not even want to think about SMP boxen without ll/sc, do we?

;-)

-- 
Regards,
Harald

---

From: Harald Koerfgen <[email-masked]>
Date: Wed, 15 Nov 2000 20:46:13 +0100 (CET)

On 15-Nov-00 Jun Sun wrote:
[R3000 UP userland spinlocks]
> period.  (It should be interesting to see if we can actually "prove"

I doubt this as well, although I'd love to be proven wrong.
 
> I also doubt if k0 is absolutely non-zero after a context ...

That's not the problem here, yes, it is. At least for the the CONFIG_CPU_R3000
case. Have a look at include/asm-mips/stackframe.h, especially the R3000
version of the RESTORE_SP_AND_RET macro.

-- 
Regards,
Harald

---

From: Jun Sun <[email-masked]>
Date: Wed, 15 Nov 2000 12:17:54 -0800

> version of the RESTORE_SP_AND_RET macro.

I did not doubt the non-zero value of k0.  I really doubted the
approach: a userland primitive is based on non-documented,
non-guarranteed kernel stack restoring code.  Once something changes in
kernel, you will get really obscure bugs.

Jun

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Wed, 15 Nov 2000 21:17:35 +0100 (MET)

> I doubt this as well, although I'd love to be proven wrong.

 Well, on UP the only events that can break atomicity are exceptions (here
I treat interrupts as exceptions as well) and DMA accesses.  I don't think
we do DMA to user space, so this should not be a problem.  So if we can
detect an exception occured we may assume an operation failed and retry. 
It's not a problem for an exception handler to clobber k0 or k1 upon exit. 

 Unfortunately we cannot use this implementation in the userland or we
risk problems when running on SMP systems -- an ISA-I user binary might
very well be run on an ISA-II (or higher) SMP system.  But we can use it
in the kernel, for sysmips() and everything else.  All we have to be
careful about is not to allow DMA accesses to spinlocks.  I don't think
this is a problem in reality. 

  Maciej

---

From: Jun Sun <[email-masked]>
Date: Wed, 15 Nov 2000 12:21:54 -0800

> It's not a problem for an exception handler to clobber k0 or k1 upon exit.

I gave more thoughts on this.  While your argument sounds plausible, the
devil is in "retry" - without a lower-level atomic operation, you cannot
"restore" the initial condition and conduct a re-try.

Come up with a pseudo code to show I am wrong.

> this is a problem in reality.

Let us solve UP first. :-)

Jun

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Wed, 15 Nov 2000 21:31:41 +0100 (MET)

> kernel, you will get really obscure bugs.

 We need not rely on a non-documented behaviour.  We may clobber k0
explicitly, e.g.:

#define RESTORE_SP_AND_RET			\
		.set	push;			\
		.set	noreorder;		\
		lw	k1, PT_EPC(sp);		\
		lw	sp, PT_R29(sp);		\
		nor	k0, zero, zero;		\
		jr	k1;			\
		 rfe;				\
		.set	pop

  Maciej

---

From: Ralf Baechle <[email-masked]>
Date: Thu, 16 Nov 2000 02:27:10 +0100

> and see whether there are no races.  It should work fine, indeed.

I'm still not completly happy - it's a somewhat hackish solution.  I'm
thinking about a special file which can be mmaped into the process address
space and contains processor specific optimized code.  This also has other
uses.  One that comes to my mind are trampolines.  Right now we have to
make a syscall to flush the cache.  But on the RM7000 some cacheflush
operations are available in userspace.  I'm sure we can come up with more
uses.

  Ralf

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Thu, 16 Nov 2000 14:23:41 +0100 (MET)

> "restore" the initial condition and conduct a re-try.

 You are right -- it's not that trivial, if at all possible (assuming O(1)
memory usage).

> Come up with a pseudo code to show I am wrong.

 I'm thinking -- I'll provide results if I invent something useful.

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Thu, 16 Nov 2000 14:41:13 +0100 (MET)

> operations are available in userspace.  I'm sure we can come up with more

 I don't actually think this is needed.  All I worry about are inline
functions.  They must be fast and work on anything equal or better then
the host they were built for.  Normal functions are not a problem -- they
may be converted to wrappers that call various backends indirectly.