Subject: MIPS32 patches breaking DecStation

From: Florian Lohoff <[email-masked]>
Date: Tue, 9 Jan 2001 00:41:01 +0100

Hi,
i found this snippet from arch/mips/kernel/head.S breaking DecStations:

@@ -68,9 +76,9 @@
        mtc0    k0, CP0_ENTRYLO0                # load it
        srl     k1, k1, 6                       # convert to entrylo1
        mtc0    k1, CP0_ENTRYLO1                # load it
-       b       1f
-        tlbwr                                  # write random tlb entry
-1:     
+       nop                                     # Need 2 cycles between mtc0
+       nop                                     #  and tlbwr (CP0 hazard).
+       tlbwr                                   # write random tlb entry
        nop
        eret                                    # return from trap
        END(except_vec0_r4000)

>From the Documentation and how i understand it this is perfectly
valid as the mtc0 instruction causes a cp0 hazard within the next 2 instruction
thought the delay of 2 cycles would be ok.

This is probably related to the Decstations having REALLY old R4000 silicion
revisions - Probably one should look through the erratas for these

flo@repeat:~$ cat /proc/cpuinfo 
cpu			: MIPS
cpu model		: R4000SC V3.0
system type		: Digital DECstation 5000/1xx

OK - I just had a look at the errata - This IS a workaround to a 
Mips R4000SC 2.0, 3.0 errata - I restored the original code back.

Flo
-- 
Florian Lohoff                  [email-masked]             +49-5201-669912
     Why is it called "common sense" when nobody seems to have any?

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Tue, 9 Jan 2001 01:34:47 +0100

Florian,

Could you do me a huge favor and try a build that
uses 3 or 4 nops instead of the branch to the instruction
after the delay slot?   There was a reason that I eliminated
the branch construct from the MIPS internal Linux source
base - it's a hack that works perfectly on R4000's, but
it's pretty much a coincidence that it does so.  Yes,
the code fragment in question is R4K-specific, but
we really need to migrate towards the use of consistent
mechanisms that work across the full range of MIPS
CPUs.  Ideally, *all* CP0 hazards should some day be 
padded out with "ssnops" (sll $0,$0,1, if I recall), which 
force a 1 cycle delay per instruction even on superscalar 
MIPS CPUs.

            Kevin K.

---

From: Florian Lohoff <[email-masked]>
Date: Tue, 9 Jan 2001 09:54:38 +0100

> after the delay slot?   There was a reason that I eliminated

No problem - Done - doesnt work

> force a 1 cycle delay per instruction even on superscalar 

It immediatly after starting init goes crazy with "Illegal instruction"
and dies like this:

[ ... a couple hundret Illegal instruction ... ]
[init:1] Illegal instruction 8f9983d4 at 0fb68df8 ra=0fb68a20
[init:1] Illegal instruction 8fbc0018 at 0fb68e08 ra=0fb68a20
[init:1] Illegal instruction 02402021 at 0fb68e10 ra=0fb68a20
[init:1] Illegal instruction 00000000 at 0fb68e18 ra=0fb68a20
Kernel panic: Attempted to kill init!
BUG IN DYNAMIC LINKER ld.so: dl-minimal.c: 67: malloc: Assertion `n <= _dl_page!

It seems we need a R4000 2.0/3.0 special except_vec_r4000 like we already
have for the R4600 and some other kinds of mips CPUs.

Flo

---

From: Ralf Baechle <[email-masked]>
Date: Tue, 9 Jan 2001 13:09:58 -0200

> force a 1 cycle delay per instruction even on superscalar 

While we could come up with a common TLB fault handler I really don't want
to.  For many applications this TLB fault handler is extremly performance
sensitive; every single cycle directly translates into application
performance.  Seems like we'll need some more TLB fault handler.  And a
complete TLB fault handler rewrite would be a good ide anyway, sigh ...

  Ralf

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Tue, 9 Jan 2001 16:50:34 +0100

> > Yes, the code fragment in question is R4K-specific, but

Sorry if I wasn't clear.  I am not suggesting a "one size
fits all" TLB handler - though as the old SGI hardware
gets retired and the newer, more standardized MIPS32
and MIPS64 parts become the principal targets, we
may see a greater convergence.  I am simply suggesting
that, even if there are differences in policy necessitated
by different CPU implementations, they should use the
same mechanisms.  If all CP0 hazards are avoided by
using ssnops, for example, we could evolve from writing
a new handler for every R4K variant to having a routine
that generates a handler with the right pipeline hazard
management, based on a set of paramters for the CPU.
And, while Ralf and I sometimes disagree on the importance
of this, in my own opinion, being consistent in these small
details helps avoid errors when a systems programmer
new to the architecture and/or the OS needs to work on
the system.

You may say that I'm a dreamer, but I'm not the only one.  ;-)

            Regards,

            Kevin K.

---

From: Harald Koerfgen <[email-masked]>
Date: Tue, 09 Jan 2001 17:43:08 +0100 (CET)

> Mips R4000SC 2.0, 3.0 errata - I restored the original code back.

Where have you found those errata?

-- 
Regards,
Harald

---

From: Ralf Baechle <[email-masked]>
Date: Tue, 9 Jan 2001 15:01:42 -0200

> Where have you found those errata?

Go to www.mips.com, follow the publications link.  Yes, MIPS be praised
for having errata and docs for rotten old silicon online!

  Ralf

---

From: Harald Koerfgen <[email-masked]>
Date: Tue, 09 Jan 2001 18:11:00 +0100 (CET)

> No problem - Done - doesnt work

Same here on my /260 (R4400SC V4.0). Neither inserting four "sll $0,$0,1" nor
four "nop" seem to work. The branch, on the other hand, does.

-- 
Regards,
Harald

---

From: Ralf Baechle <[email-masked]>
Date: Tue, 9 Jan 2001 16:28:35 -0200

> four "nop" seem to work. The branch, on the other hand, does.

Note the ssnops only make sense on superscalar CPUs, so not on the R4000.
Also note that the branch is equivalent to three nops.  One for the
branch instruction itself, the two more for instructions in the pipeline
that get killed.  On the R4600 / R500 where the hazard is only a single
instruction the branch is equivalent to only a single nop.  So while
unobvious the branch is a rather neat idea.

  Ralf

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Tue, 9 Jan 2001 20:30:05 +0100

> > four "nop" seem to work. The branch, on the other hand, does.

Then it's apparently more than just a garden-variety CP0 hazard.

> Note the ssnops only make sense on superscalar CPUs, so not on the R4000.

My point is that an SSNOP should cause a 1 cycle stall on *any* MIPS
implementation to date, superscalar or not.  It's not documented that
way for the R10000, but if I recall correctly, it works there too.  If one
wants to standardize on a single mechanism, that's the one to use.
That's all I'm saying.

> unobvious the branch is a rather neat idea.

Yes, it's cute, but it relies on accidents of implementation to work,
and could easily fail on other CPUs otherwise compatible with
the R4000.  In principle, such a branch might incur no delay at
all on an advanced 64-bit processor.  By all means, use it for
the specific cases of the CPUs on which it is known to work,
but it should not be used in "default" MIPS CP0 handlers.

            Regards,

            Kevin K.

---

From: Ralf Baechle <[email-masked]>
Date: Tue, 9 Jan 2001 17:54:16 -0200

> wants to standardize on a single mechanism, that's the one to use.

I agree on that - except that I haven't seen the various various flavours
of nops documented anywhere except in IRIX's <sys/asm.h> ...

> but it should not be used in "default" MIPS CP0 handlers.

This behaviour of the R4000 branch is documented in the R4000 manual's
description of the pipeline.

  Ralf

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Tue, 9 Jan 2001 21:33:12 +0100

> description of the pipeline.

Yes, yes, like I said, use it whenever you see an R4000 PrID
if you like.  Just don't use it as the handler installed for a PrID
not otherwise known to the kernel.

            Kevin K.