Subject: Can't build a CONFIG_CPU_NEVADA kernel

From: Daniel Jacobowitz <[email-masked]>
Date: Wed, 14 Mar 2001 08:46:33 -0500

I've been trying for a couple of days now to build a MIPS kernel with
CONFIG_CPU_NEVADA, and I can't get it to work.  r4k_switch.S produces a
pile of "opcode not supported by processor" errors.

First, I figured out where the problem is coming from:

r4k_switch.S is included for all processors but the r3000 and r3912. 
Is that really correct?  Then, it references FPU_SAVE_DOUBLE, which
includes:

        cfc1    tmp,  fcr31;                    \
        sdc1    $f0,  (THREAD_FPU + 0x000)(thread); \
        sdc1    $f2,  (THREAD_FPU + 0x010)(thread); \

The sdc1 instruction in binutils is flagged like this:
      if (mips_cpu == CPU_R4650)
        {
          as_bad (_("opcode not supported on this processor"));
          return;
        }

And the IVR sets CONFIG_CPU_NEVADA, which produces
ifdef CONFIG_CPU_NEVADA
GCCFLAGS        += -mcpu=r8000 -mips2 -Wa,--trap -mmad
endif

and -mmad becomes -m4650 to the assembler.

Something is fishy here.  Anyone know what?  I have a suspicion that we
need to change the way we invoke binutils.  Making -mmad imply -m4650
just seems lame, since -m4650 also implies -msingle-float, and I don't
think that's right for the r8000.

I worked back in time in gcc, binutils, and kernel sources and I
couldn't figure out what's changed - I'm sure this worked at some
point.

Any ideas?

-- 
Daniel Jacobowitz                           Debian GNU/Linux Developer
Monta Vista Software                              Debian Security Team
                         "I am croutons!"

---

From: Ralf Baechle <[email-masked]>
Date: Wed, 14 Mar 2001 19:59:19 +0100

> pile of "opcode not supported by processor" errors.

Known and unsolved problem.

> Is that really correct?  Then, it references FPU_SAVE_DOUBLE, which

Yes.  Originally written for the R4000 the Dr. Franksteins at MIPS in the
meantime also have taken r4k-style fpus and put them into 32-bit processors,

> and -mmad becomes -m4650 to the assembler.

Which is pretty much bs because mmad may have been introduced with the
R4640 / R4650 but isn't only available on this processor.  From an ISA
view it's a processor specific extension and as such it should be
controlled by a separate option.  Gcc has -mmad which is fine but passing
it on to as as -m4650 is borken.

> think that's right for the r8000.

R8000 is serious fp, indeed.

> couldn't figure out what's changed - I'm sure this worked at some

You'll have to go back far in time.  I introduced the use of -mmad for
Nevada-class CPUs in late summor '97.

As a second bug which makes this one even more annoying something like

	.set	mips3
	sdc1    $f2, (a0)
	.set	mips0

also doesn't work - the assembler will still throw an "opcode not supported
on this processor" message.  After all MIPS III means double precission fp.
And passing additional assembler options with -Wa,foo doesn't help either
in this case so without the necessary gcc / assembler changes this
optimization is lost for now.

  Ralf

---

From: Daniel Jacobowitz <[email-masked]>
Date: Wed, 14 Mar 2001 14:05:29 -0500

> it on to as as -m4650 is borken.

OK, so that needs to change.  That's pretty easy to do, at least in our
local toolchains.

> optimization is lost for now.

Does -mmad make a sufficient difference on these processors to bother
fixing it?

If it does, I can probably whip up a -mmad patch to binutils to allow
those opcodes - or I could introduce -mnevada, or whatever the
appropriate term would be, to mean "r8000 with the mad* extensions". 
In fact, that would probably be easiest, and sounds like the most
correct.

---

From: Ralf Baechle <[email-masked]>
Date: Wed, 14 Mar 2001 20:20:58 +0100

> OK, so that needs to change.  That's pretty easy to do, at least in our

If it's only in your local toolchains, it's lost.  Send your changes to
the FSF!

> Does -mmad make a sufficient difference on these processors to bother

Not for the kernel but it's a sufficiently important optimization for some
specialised applications (signal processing type etc.) that it should be
fixed.

> In fact, that would probably be easiest, and sounds like the most

Don't think of the r8000; the kernel only uses the -mcpu=r8000 option
because the Nevada CPUs have _somewhat_ similar scheduling properties
to the R8000.  This of it as an independant ISA expension which can
be used with an arbitrary MIPS processor - even a R3000 processor.

  Ralf

---

From: Jun Sun <[email-masked]>
Date: Wed, 14 Mar 2001 11:48:52 -0800

> be used with an arbitrary MIPS processor - even a R3000 processor.

Although -mmad is generic, why do we need it for kernel compiling?  If no good
reason, I propose to remove -mmad from the Makefile for Nevada chip.

Of course, we still need to fix the -mmad implying -m4650 bug ...

Jun

---

From: Ralf Baechle <[email-masked]>
Date: Wed, 14 Mar 2001 21:02:07 +0100

> reason, I propose to remove -mmad from the Makefile for Nevada chip.

The compiler actually emits a few mmad instructions, so this instruction
actually make a small difference.

> Of course, we still need to fix the -mmad implying -m4650 bug ...

I guess I leave the -mmad flag in the kernel source as reminder for somebody
to fix this ...

  Ralf

---

From: Daniel Jacobowitz <[email-masked]>
Date: Wed, 14 Mar 2001 15:56:17 -0500

> If it's only in your local toolchains, it's lost.  Send your changes to

Of course.  Let me clarify that statement - it's easy to do in a way
that would be acceptable in our local toolchains, and somewhat harder
to do in a way acceptable to the FSF.

In this case, though, not much harder.  I'm going to try to have a
-mmad patch later today for binutils, and a trivial patch for GCC to
use it instead of -m4650.

> be used with an arbitrary MIPS processor - even a R3000 processor.

Oh, I see.  Thanks for the clarification.

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Wed, 14 Mar 2001 23:11:47 +0100

> be used with an arbitrary MIPS processor - even a R3000 processor.

In the interests of historical accuracy and general pedantry,
let me point out that -mcpu=r8000 is in effect a rather klugy
way of saying "-mips4" to compilers that predate official
MIPS IV ISA support.  The R8000 was the first MIPS IV
CPU, followed by the R10000 and the R5000.  The "Nevada"
processors added three implementation specific instructions
to the MIPS IV ISA: MAD, MADU, and MUL (targeted multiply).
"Correct" usage would be to enable those three instructions
with a "-mcpu=nevada", or better still, "-mcpu=r5200" (for 
consistency), and to enable the rest of the MIPS IV ISA 
with "-mips4" instead of the archaic r8000 hack.

Note, furthermore, that -mmad needs to be handled with care: 
Prior to MIPS standardizing the instruction as part of 
MIPS32, there were four or five subtly (or not so subltly) 
different definitions of integer multiply-accumulate for MIPS.  
Most use the same opcode, but even those can differ in side 
effects (is the rd field interpreted, etc.). A R4650 madd operation
will probably behave equivalently on a Nevada CPU, 
but certainly not on a Vr41xx part, for example.

            Regards,

            Kevin K.

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Wed, 14 Mar 2001 23:47:31 +0100

> with "-mips4" instead of the archaic r8000 hack.

I should add "Correct", but not useful for 32-bit
kernels.  "-mips32", once that percolates through
the gcc food chain, would be *almost* perfect
for 32-bit Nevada kernels.  I say almost, because
while MIPS32 is 32-bit MIPS IV plus MADD, it also
has a handfull of instructions that are not supported 
by the R52xx, of which it is wildly improbable but 
theoretically possible that the multiply-subtracts
might somehow get emitted in compiled application
code. It should work just fine for kernel purposes, though.

Meanwhile, try piping objdump of a "-mmad" kernel
through "grep -i mad".  I'd be stunned if a single MADD
turned up.  If it gains nothing, but breaks builds, then
for heaven's sake banish -mmad from the kernel
makefiles!

            Regards,

            Kevin K.

---

From: Pete Popov <[email-masked]>
Date: Wed, 14 Mar 2001 17:50:06 -0800

> for heaven's sake banish -mmad from the kernel

I managed to compile a 2.4.2 kernel with our bleeding edge toolchain. I
had to get rid of the -mmad from the Makefile though. The kernel boots
and runs and I ran Bonnie over NFS as well as on an IDE hard disk. I
tried some other tests as well, and they all passed.  

Without the -mmad flag, I see the following mad instructions below. No
"madd" though, just madd.s, madd16, mad, etc. So, can we get rid of this
flag in the Makefile?

Pete

0000000080132690 <sys_madvise>:
    801326dc:   1080fffc        beqz    $a0,801326d0 <sys_madvise+40>
    801326e4:   04810004        bgez    $a0,801326f8 <sys_madvise+68>
    801326f8:   5440002e        bnezl   $v0,801327b4 <sys_madvise+124>
    80132714:   54400027        bnezl   $v0,801327b4 <sys_madvise+124>
    8013271c:   12510024        beq     $s2,$s1,801327b0
<sys_madvise+120>
    80132734:   1200001e        beqz    $s0,801327b0 <sys_madvise+120>
    80132744:   10400003        beqz    $v0,80132754 <sys_madvise+c4>
    80132758:   1440000c        bnez    $v0,8013278c <sys_madvise+fc>
    80132764:   10400007        beqz    $v0,80132784 <sys_madvise+f4>
    80132770:   0c04c988        jal     80132620 <madvise_vma>
    8013277c:   5660000d        bnezl   $s3,801327b4 <sys_madvise+124>
    80132784:   0804c9ec        j       801327b0 <sys_madvise+120>
    80132790:   0c04c988        jal     80132620 <madvise_vma>
    8013279c:   56600005        bnezl   $s3,801327b4 <sys_madvise+124>
    801327a8:   0804c9cd        j       80132734 <sys_madvise+a4>
    801327c8:   1080fffc        beqz    $a0,801327bc <sys_madvise+12c>
    801327d0:   1c800004        bgtz    $a0,801327e4 <sys_madvise+154>
    80250590:   4d4d2030        nmadd.s $f0,$f10,$f4,$f13
    80250a84:   4f4e2820        madd.s  $f0,$f26,$f5,$f14
    802517bc:   00000029        dmadd16 $zero,$zero
    80253c94:   00000029        dmadd16 $zero,$zero
    802555a4:   00000029        dmadd16 $zero,$zero
    80256178:   00000029        dmadd16 $zero,$zero
    80259804:   4d445b20        madd.s  $f12,$f10,$f11,$f4
    80259810:   4d445b20        madd.s  $f12,$f10,$f11,$f4
    8025981c:   4d445b20        madd.s  $f12,$f10,$f11,$f4
    8025aa34:   00000029        dmadd16 $zero,$zero
    8025bbf0:   4c257830        nmadd.s $f0,$f1,$f15,$f5
    802618a8:   4f2f4920        madd.s  $f4,$f25,$f9,$f15
    80262018:   4e203a70        nmadd.s $f9,$f17,$f7,$f0
    802625c4:   4c554e20        madd.s  $f24,$f2,$f9,$f21
    80262700:   4c554e20        madd.s  $f24,$f2,$f9,$f21
    80264d5c:   4c545220        madd.s  $f8,$f2,$f10,$f20
    80264d64:   00000029        dmadd16 $zero,$zero
    80265148:   4e203d21        madd.d  $f20,$f17,$f7,$f0
    80265194:   4e203d21        madd.d  $f20,$f17,$f7,$f0
    8026534c:   00000029        dmadd16 $zero,$zero
    80265b50:   4f495020        madd.s  $f0,$f26,$f10,$f9
    80265f70:   4c554e20        madd.s  $f24,$f2,$f9,$f21
    80266294:   4e554c20        madd.s  $f16,$f18,$f9,$f21
    802670bc:   4c4c4120        madd.s  $f4,$f2,$f8,$f12
    80267690:   4d207361        madd.d  $f13,$f9,$f14,$f0
    80267f5c:   00000029        dmadd16 $zero,$zero
    80268314:   00000029        dmadd16 $zero,$zero
    80268abc:   00000029        dmadd16 $zero,$zero
    80269cd4:   4d435020        madd.s  $f0,$f10,$f10,$f3
    8026a504:   00000029        dmadd16 $zero,$zero
    8026df88:   4c4c5020        madd.s  $f0,$f2,$f10,$f12
    8026e01c:   4c502070        nmadd.s $f1,$f2,$f4,$f16
    8026f6bc:   4c622020        madd.s  $f0,$f3,$f4,$f2
    8026f71c:   4e622020        madd.s  $f0,$f19,$f4,$f2
    8026f944:   4c622020        madd.s  $f0,$f3,$f4,$f2
    8026f9f0:   4d772020        madd.s  $f0,$f11,$f4,$f23
    802702f8:   4d203231        nmadd.d $f8,$f9,$f6,$f0
    80273830:   00000029        dmadd16 $zero,$zero
    80276a24:   4f6d7361        madd.d  $f13,$f27,$f14,$f13
    80277498:   4e203d21        madd.d  $f20,$f17,$f7,$f0
    80279c10:   4d202020        madd.s  $f0,$f9,$f4,$f0
    80279ea4:   4d434920        madd.s  $f4,$f10,$f9,$f3
    8027c588:   4f494520        madd.s  $f20,$f26,$f8,$f9
    80280048:   4d642520        madd.s  $f20,$f11,$f4,$f4
    802802c8:   4d4f5220        madd.s  $f8,$f10,$f10,$f15
    802a0e00:   4e414d20        madd.s  $f20,$f18,$f9,$f1
    802a2988:   4c4f5720        madd.s  $f28,$f2,$f10,$f15
    802a2a30:   00000029        dmadd16 $zero,$zero
    802a2c14:   4f2f4920        madd.s  $f4,$f25,$f9,$f15
    802a336c:   4f2f4920        madd.s  $f4,$f25,$f9,$f15
    802a39e4:   00000029        dmadd16 $zero,$zero
    802a3a00:   00000029        dmadd16 $zero,$zero
    802a3a1c:   00000029        dmadd16 $zero,$zero
    802a3a70:   00000029        dmadd16 $zero,$zero
    802a3ac8:   00000029        dmadd16 $zero,$zero
    802a3ad0:   4d353531        nmadd.d $f20,$f9,$f6,$f21
    802a3b3c:   00000029        dmadd16 $zero,$zero
    802a427c:   4f2f4920        madd.s  $f4,$f25,$f9,$f15
    802a50bc:   4d4f4320        madd.s  $f12,$f10,$f8,$f15
    802a5504:   4e414c20        madd.s  $f16,$f18,$f9,$f1
    802a57ec:   4c2e6920        madd.s  $f4,$f1,$f13,$f14
    802a5f18:   4f2f4920        madd.s  $f4,$f25,$f9,$f15
    802a6378:   4d544120        madd.s  $f4,$f10,$f8,$f20
    802a63b4:   4d207161        madd.d  $f5,$f9,$f14,$f0
    802a6d5c:   00000029        dmadd16 $zero,$zero
    802a6ea8:   4f2f4920        madd.s  $f4,$f25,$f9,$f15
    802a6ebc:   4f2f4920        madd.s  $f4,$f25,$f9,$f15
    802a73c0:   4e414c20        madd.s  $f16,$f18,$f9,$f1
    802a74cc:   4e414c20        madd.s  $f16,$f18,$f9,$f1
    802a75a0:   4f505420        madd.s  $f16,$f26,$f10,$f16
    802a7994:   4d502031        nmadd.d $f0,$f10,$f4,$f16
    802a7ac0:   4e414c20        madd.s  $f16,$f18,$f9,$f1
    802a7b54:   4e206870        nmadd.s $f1,$f17,$f13,$f0
    802a7e8c:   4d502820        madd.s  $f0,$f10,$f5,$f16
    802a7f84:   4e5b2061        madd.d  $f1,$f18,$f4,$f27
    802a7f90:   4e5b2061        madd.d  $f1,$f18,$f4,$f27
    802a7f9c:   4e5b2061        madd.d  $f1,$f18,$f4,$f27
    802a7fa8:   4e5b2061        madd.d  $f1,$f18,$f4,$f27
    802a8068:   00000029        dmadd16 $zero,$zero
    802a8290:   4d5b2030        nmadd.s $f0,$f10,$f4,$f27
    802a8c94:   4f525020        madd.s  $f0,$f26,$f10,$f18
    802a9088:   4e207370        nmadd.s $f13,$f17,$f14,$f0
    802a94ec:   4d544120        madd.s  $f4,$f10,$f8,$f20
    802a962c:   4d544120        madd.s  $f4,$f10,$f8,$f20
    802a9c10:   00000029        dmadd16 $zero,$zero
    802a9d64:   00000029        dmadd16 $zero,$zero
    802a9d80:   00000029        dmadd16 $zero,$zero
    802a9e2c:   00000029        dmadd16 $zero,$zero
    802ab81c:   00000029        dmadd16 $zero,$zero
    802ac17c:   00000029        dmadd16 $zero,$zero
    802ac65c:   4d207861        madd.d  $f1,$f9,$f15,$f0
    802ac740:   4d207861        madd.d  $f1,$f9,$f15,$f0
    802ac79c:   4c525320        madd.s  $f12,$f2,$f10,$f18
    802ad3e4:   4e206c61        madd.d  $f17,$f17,$f13,$f0
    802ad954:   4d202d20        madd.s  $f20,$f9,$f5,$f0
    802ae994:   4f2f4920        madd.s  $f4,$f25,$f9,$f15
    802af200:   4e415720        madd.s  $f28,$f18,$f10,$f1
    802b10cc:   4e414320        madd.s  $f12,$f18,$f8,$f1
    802b167c:   4e454420        madd.s  $f16,$f18,$f8,$f5
    802b1690:   4c554d20        madd.s  $f20,$f2,$f9,$f21
    802b1d88:   4d4f4320        madd.s  $f12,$f10,$f8,$f15
    802b1df0:   00000029        dmadd16 $zero,$zero
    802b2428:   00000029        dmadd16 $zero,$zero
    802b2458:   4d206e61        madd.d  $f25,$f9,$f13,$f0
    802b30e0:   00000029        dmadd16 $zero,$zero
    802b3140:   00000029        dmadd16 $zero,$zero
    802b3c68:   00000029        dmadd16 $zero,$zero
    802b3ff8:   4f525020        madd.s  $f0,$f26,$f10,$f18
    802b4010:   4f525020        madd.s  $f0,$f26,$f10,$f18
    802b4120:   4d472030        nmadd.s $f0,$f10,$f4,$f7
    802b453c:   4f2f4920        madd.s  $f4,$f25,$f9,$f15
    802b5d80:   00000029        dmadd16 $zero,$zero
    802b6e38:   00000028        madd16  $zero,$zero
    802b6e40:   00000029        dmadd16 $zero,$zero
    802b7350:   00000028        madd16  $zero,$zero
    802b80a8:   00000028        madd16  $zero,$zero
    802b80b0:   00000029        dmadd16 $zero,$zero
    802b81d8:   00000028        madd16  $zero,$zero
    802b81e0:   00000029        dmadd16 $zero,$zero
    802bdda4:   00000028        madd16  $zero,$zero
    802bddd8:   00000028        madd16  $zero,$zero
    802bde7c:   00000029        dmadd16 $zero,$zero
    802bdeac:   00000028        madd16  $zero,$zero
    802bdeb8:   00000028        madd16  $zero,$zero
    802bdf14:   00000028        madd16  $zero,$zero
    802be6d8:   00000028        madd16  $zero,$zero
    802be764:   00000028        madd16  $zero,$zero
    802c0578:   70000000        mad     $zero,$zero
    802c07f4:   72070000        mad     $s0,$a3
    802c1490:   00000028        madd16  $zero,$zero
    802c957c:   00000029        dmadd16 $zero,$zero
    802c96b0:   00000028        madd16  $zero,$zero
    802ccdcc:   00000028        madd16  $zero,$zero
    802cddb0:   00290028        madd16  $at,$t1
    802cdfb0:   00290028        madd16  $at,$t1
    802ce1b0:   00290028        madd16  $at,$t1
    802ce6b8:   00290028        madd16  $at,$t1
    802d4150:   4e534331        nmadd.d $f12,$f18,$f8,$f19
    802d667c:   00000028        madd16  $zero,$zero
    802d7614:   00290028        madd16  $at,$t1
    802d9770:   00000028        madd16  $zero,$zero
    802d9820:   00000029        dmadd16 $zero,$zero

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Thu, 15 Mar 2001 09:01:49 +0100

> "madd" though, just madd.s, madd16, mad, etc. So, can we get rid of this

"MAD" is a synonym for "MADD", but it looks as if this was
a grep of the output of "objdump --diassemble-all".  There
can't possibly be that many usefull floating point MADDs
in the kernel, and the actual bytes of the instructions look
suspiciously like ASCII strings.  Likewise, "MADD16" and
"DMADD16" are R4100 instructions with a different opcode
and semantics from the Nevada/MIPS32 MADD.  And of
course instructions like "mad $zero,$zero" aren't likely to
be real code.  It's pretty obvious from context, but you
should really confirm that address 0x802c07f4 is data
before we confirm that *no* MAD/MADDS are being
generated in the kernel.

It is certainly conceivable that MADDs in some form
could be used in the kernel.  Specifically, it would not
surprise me in the least to see them used in a software
modem on a VrLinux platform.  But those modules can
and should be handled seperately (and are probably
written in assembler anyway).  There's not need for
-mmad in the general set of flags.

            Kevin K.

---

From: Ralf Baechle <[email-masked]>
Date: Fri, 16 Mar 2001 16:34:40 +0100

I looked over the MIPS backend and was depressed how little it still knows
about various newer MIPS CPUs.  So I tought gcc a bit about the R10000
and R12000.  Less than perfect but this patch against cvs-gcc should
deliver some improvments.

GCC still knows shit about the R7000 and that's definately a CPU worth some
effort ...

If you want the R10k patch, drop me a mail.

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Fri, 16 Mar 2001 15:04:23 +0100

> with "-mips4" instead of the archaic r8000 hack.

Your historic facts may be right but the GCC fact aren't.  -mcpu=xxx tell
GCC to schedule instructions for a certain processor xxx.  This does not
enable the full use of it's instruction set.  Back in time when I choose
these options I choose because GCC didn't know -mcpu=r5000 but the R8000
was supported and it was the closest fit.  Gcc 1.1.2 knows this option
so I just changed all instances of -mcpu=r8000 into -mcpu=r5000.

> but certainly not on a Vr41xx part, for example.

Unfortunately true but there is a reason that QED's manual marks it as an
proprietary extension ...

  Ralf

---

From: Daniel Jacobowitz <[email-masked]>
Date: Fri, 16 Mar 2001 13:02:08 -0500

> so I just changed all instances of -mcpu=r8000 into -mcpu=r5000.

Are you saying that the -mcpu=r8000 options in linux/arch/mips/Makefile
for the nevada should be -mcpu=r5000 instead?

---

From: Ralf Baechle <[email-masked]>
Date: Fri, 16 Mar 2001 19:16:33 +0100

> for the nevada should be -mcpu=r5000 instead?

Correct for egcs >= 1.1.2.

  Ralf

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Fri, 16 Mar 2001 19:46:23 +0100

> so I just changed all instances of -mcpu=r8000 into -mcpu=r5000.

As I understand it, the original intention behind -mcpu was to optimise 
instruction scheduling, but it got perverted over time to enable 
instructions as well.  In any case, the R5000 and R8000 have the 
same ISA, but the pipelines are radically different.  The R8K is 
4-way superscalar, with a 2-cycle branch penalty and branch 
prediction.  The R5K is two way supercalar (Int/FP pairs only) 
with classic MIPS branch behavior, etc.

> proprietary extension ...

Yup.  The quesiton is, does gcc's -mmad option actually
select based on -mcpu or some other variable which
semantics to use, or does it assume R4650 semantics
(I had the impression that it was the R4650 that drove
the implementation of MIPS madd's into gcc - correct
me if I'm wrong).

            Regards,

            Kevin K.

---

From: Ralf Baechle <[email-masked]>
Date: Fri, 16 Mar 2001 20:35:40 +0100

> with classic MIPS branch behavior, etc.

Gcc's knowledge about MIPS architecture is so limited that an R5000 isn't
very much different from an R8000 ...

> semantics to use, or does it assume R4650 semantics

If you have a collection which of the CPUs have implemented mmad in what
way I'd like to use that to put it into gcc.

> the implementation of MIPS madd's into gcc - correct

Guess that's right.

  Ralf