Subject: Re: Indy crashes

From: "Kevin D. Kissell" <[email-masked]>
Date: Wed, 16 Feb 2000 11:42:29 +0100

Ralf Baechle writes:
>On Tue, Feb 15, 2000 at 11:23:49PM +0100, Kevin D. Kissell wrote:
>>                                   The R5000 manual states that there
>> should be "at least two integer instructions between" any
>> instruction modifying the PageMask, EntryHi, or EntryLo[01]
>> registers and the subsequent tlbw[ir] or tlbp.  That's different
>> from the R4000.  In the current Linux arch/mips/head.S file, 
>> this interval does not seem to be respected by any of the TLB 
>> miss handlers.  Indeed, the default except_vec0_r4000 handler,
>> which I believe is what is used if an R5000 is detected, has 
>> the sequence
>> 
>>         mtc0    k1, CP0_ENTRYLO1                # load it
>>         b       1f
>>          tlbwr                                  # write random tlb entry
>> 1:
>>         nop
>>         eret
>> 
>> wherin the purpose of the branch is obscure (I imagine
>> it fixed a bug seen somewhere on some CPU but it
>> makes me rather queasy)  but wherein in any case we 
>> do not seem to be assured 2 issues between the mtc0 
>> and the tlbwr.  It should be OK for an R4000, but not for 
>> an R5000.
>
>No, it's not a bug workaround.  The reason for this branch is that the
>R4000 and R4400 have a penalty of three cycles for a taken branch.  So
>the branch above is equivalent with 
>
> mtc0 k1, CP0_ENTRYLO1
> nop
> tlbwr
> nop
> nop
> nop
> eret
>
>Funky trick, isn't it?  I don't have the the R4600 / R5000 docs at hand
>but as I understood them the above code should also work just perfect
>for them.

No.  Not as I read the specs.  There are three problems here.

First, the question is *not* one of no-ops between the TLBWR
and the ERET, but of no-ops between the MTC0 and the
TLBWR - re-read the quoted text above from my previous
message.  So the code may well be broken as I conjectured
even if your assumption about the branch delay was valid.

Second, the R5000 and R4600 piprlines are not as deep
as those of the R4000/4400.   The R5000 documentation
calls out a branch implementation with a *single* delay cycle.
I quote: "The one cycle branch delay is a result of the branch
comparison logic operating during the 1A pipeline stage of
the branch.  This allows the branch target address calculated
in the previous stage to be used for the instruction access in
the following 1I phase."   So even if the execution of the
branch were inserting delay between the MTC0 and the
TLBWR as you seemed to assume, it might not be inserting
as much delay as you think.

Thirdly, this whole thread underscores why "clever" solutions that 
depend on implementation features of particular CPUs should 
be avoided whenever possible. If you want to be assured of
getting a delay cycle in a MIPS instruction stream, you should
use a "SSNOP", (sll r0,r0,1 as opposed to the "nop" sll r0,r0,0),
which forces delays even in superscalar implementations.

>> So someone with the ability to reproduce the R5000
>> problem should really try sticking a nop between the
>> mtc0 and the branch (sigh) to see if that stabilizes 
>> the system.

I still think this would be a good idea.  Further, from Bill Earl's
comment on this same thread, it sounds like, to be conservative,
 trap_init() in arch/mips/kernel/traps.c needs to detect the R5000
case and patch in except_vec0_r45k_bvahwbug instead
of except_vec0_r4000, and that furthermore a nop (or ssnop) 
be added between the MTC0 and the branch of 
except_vec0_r45k_bvahwbug.

>Talking about CPU bugs - the R5230 and maybe it's relatives needs a nasty
>workaround.  I think I only put the workaround into the Cobalt kernel.
>Of course IDT doesn't admit that this bug even exists ...

Um, why should they, when IDT didn't do the R5230?  ;-)
Seriously, did you mean to refer to the R323xx from IDT,
or to QED as the design house for the R5230?  I have been 
running 2.2.12 on an R5260 for months and it has been very 
stable.   To which bug and which workaround are you referring?

            Regards,

            Kevin K.
__

Kevin D. Kissell
MIPS Technologies European Architecture Lab
[email-masked]
Tel. +33.4.78.38.70.67
FAX. +33.4.78.38.70.68

---

From: "William J. Earl" <[email-masked]>
Date: Wed, 16 Feb 2000 10:50:47 -0800 (PST)

> >On Tue, Feb 15, 2000 at 11:23:49PM +0100, Kevin D. Kissell wrote:
...
 > even if your assumption about the branch delay was valid.

      Empirically, this does not appear to be the case.  Here are 
the handlers for the r4600 and r5000 on IRIX, which have been stable
for years:

eutlbmiss3_250mhz:
[1023] 0x880849b0:  40 1a 20 00       mfc0	k0,context
[1023] 0x880849b4:  00 1a d0 43       sra	k0,k0,1
utlbmiss_r4600:
[1031] 0x880849b8:  8f 5b 00 00       lw	k1,0(k0)
[1031] 0x880849bc:  8f 5a 00 04       lw	k0,4(k0)
[1032] 0x880849c0:  00 1b d9 80       sll	k1,k1,6
[1032] 0x880849c4:  00 1b d9 82       srl	k1,k1,6
[1033] 0x880849c8:  40 9b 10 00       mtc0	k1,tlblo
[1034] 0x880849cc:  00 1a d1 80       sll	k0,k0,6
[1034] 0x880849d0:  00 1a d1 82       srl	k0,k0,6
[1035] 0x880849d4:  40 9a 18 00       mtc0	k0,tlblo1
[1036] 0x880849d8:  00 00 00 00       nop
[1037] 0x880849dc:  42 00 00 06       c0	tlbwr
[1038] 0x880849e0:  00 00 00 00       nop
utlbmiss_eret_3:
[1039] 0x880849e4:  0a 01 c9 59       j		_r4600_2_0_cacheop_eret
[1039] 0x880849e8:  00 00 00 00       nop

_r4600_2_0_cacheop_eret:
[ 211] 0x88072564:  00 00 00 00       nop
[ 211] 0x88072568:  00 00 00 00       nop
[ 211] 0x8807256c:  00 00 00 00       nop
[ 211] 0x88072570:  00 00 00 00       nop
[ 211] 0x88072574:  00 00 00 00       nop
[ 211] 0x88072578:  00 00 00 00       nop
[ 211] 0x8807257c:  00 00 00 00       nop
[ 211] 0x88072580:  00 00 00 00       nop
_r4600_2_0_cacheop_eret_inst:
[ 211] 0x88072584:  42 00 00 18       c0	eret

eutlbmiss3_250mhz:
[1023] 0x880849b0:  40 1a 20 00       mfc0	k0,context
[1023] 0x880849b4:  00 1a d0 43       sra	k0,k0,1
utlbmiss_r5000:
[1061] 0x8007ede8:  8f 5b 00 00       lw	k1,0(k0)
[1061] 0x8007edec:  8f 5a 00 04       lw	k0,4(k0)
[1062] 0x8007edf0:  42 00 00 08       c0	tlbp
[1063] 0x8007edf4:  00 1b d9 00       sll	k1,k1,4
[1063] 0x8007edf8:  00 1b d9 02       srl	k1,k1,4
[1064] 0x8007edfc:  40 9b 10 00       mtc0	k1,tlblo
[1065] 0x8007ee00:  40 1b 00 00       mfc0	k1,index
[1066] 0x8007ee04:  00 1a d1 00       sll	k0,k0,4
[1066] 0x8007ee08:  00 1a d1 02       srl	k0,k0,4
[1067] 0x8007ee0c:  07 61 00 04       bgez	k1,0x8007ee20
[1068] 0x8007ee10:  40 9a 18 00       mtc0	k0,tlblo1
[1069] 0x8007ee14:  42 00 00 06       c0	tlbwr
[1070] 0x8007ee18:  00 00 00 00       nop
[1071] 0x8007ee1c:  42 00 00 18       c0	eret

Note that, in the R4600 case, the "j _r4600_2_0_cacheop_eret" is
replaced with an eret on the R4600 Rev. 1.*.  (The code is patched
at system startup time.)  

     Also notice that IRIX does not index a segment table directly.
Instead, a second level TLB miss handler drops a mapping into a reserved
part of K2SEG for the PTE page required, where the base of that reserved
area is loaded into $context at startup time.

 > as much delay as you think.

      As you can see from the above examples, the IRIX handlers
for these processors are indeed different.  The R4000 and R4400 
require more nops in other places:

utlbmiss_prolog_up:
[ 144] 0x88084600:  40 1a 20 00       mfc0	k0,context
[ 144] 0x88084604:  00 00 00 00       nop
[ 145] 0x88084608:  00 1a d0 43       sra	k0,k0,1
utlbmiss:
[ 219] 0x88084680:  8f 5b 00 00       lw	k1,0(k0)
[ 219] 0x88084684:  8f 5a 00 04       lw	k0,4(k0)
[ 223] 0x88084688:  00 1b d9 80       sll	k1,k1,6
[ 223] 0x8808468c:  00 1b d9 82       srl	k1,k1,6
[ 224] 0x88084690:  40 9b 10 00       mtc0	k1,tlblo
[ 225] 0x88084694:  00 1a d1 80       sll	k0,k0,6
[ 225] 0x88084698:  00 1a d1 82       srl	k0,k0,6
[ 236] 0x8808469c:  40 9a 18 00       mtc0	k0,tlblo1
[ 237] 0x880846a0:  00 00 00 00       nop
[ 238] 0x880846a4:  42 00 00 06       c0	tlbwr
[ 239] 0x880846a8:  00 00 00 00       nop
[ 239] 0x880846ac:  00 00 00 00       nop
[ 239] 0x880846b0:  00 00 00 00       nop
[ 246] 0x880846b4:  42 00 00 18       c0	eret

Notice that the R4000 requires a nop after the mfc0 from $context,
and also requires extra nops after the tlbwr.

     Also, here is the utlbmiss for the 250 MHZ R4400, with another
workaround:

utlbmiss_prolog_up:
[ 144] 0x88084600:  40 1a 20 00       mfc0	k0,context
[ 144] 0x88084604:  00 00 00 00       nop
[ 145] 0x88084608:  00 1a d0 43       sra	k0,k0,1
eutlbmiss3:
[ 805] 0x88084830:  8f 5b 00 00       lw	k1,0(k0)
[ 805] 0x88084834:  8f 5a 00 04       lw	k0,4(k0)
[ 809] 0x88084838:  00 1b d9 80       sll	k1,k1,6
[ 809] 0x8808483c:  00 1b d9 82       srl	k1,k1,6
[ 810] 0x88084840:  40 80 10 00       mtc0	zero,tlblo
[ 811] 0x88084844:  40 9b 10 00       mtc0	k1,tlblo
[ 812] 0x88084848:  00 1a d1 80       sll	k0,k0,6
[ 812] 0x8808484c:  00 1a d1 82       srl	k0,k0,6
[ 824] 0x88084850:  40 80 18 00       mtc0	zero,tlblo1
[ 825] 0x88084854:  40 9a 18 00       mtc0	k0,tlblo1
[ 826] 0x88084858:  00 00 00 00       nop
[ 827] 0x8808485c:  42 00 00 06       c0	tlbwr
[ 828] 0x88084860:  00 00 00 00       nop
[ 828] 0x88084864:  00 00 00 00       nop
[ 828] 0x88084868:  00 00 00 00       nop
[ 829] 0x8808486c:  42 00 00 18       c0	eret

    Emprirically, it appears that the manual is incorrect in regard
to the number of nop instructions.  The above sequences are known
to work (via years of testing, and also via validation in discussions
with people familiar with the hardware pipelines).
        
 > which forces delays even in superscalar implementations.

      This is not realistic, given the number of workarounds required
for various processors, unless you are willing to have most processors
run quite a bit slower.  (Extra cycles in utlbmiss are noticeable.)

---

From: Ralf Baechle <[email-masked]>
Date: Thu, 17 Feb 2000 17:07:44 +0100

> even if your assumption about the branch delay was valid.

Exactly _that_ is the trick.  The pipeline will execute the two extra
branch penalty cycles _after_ the tlbwr instruction.  That is exactly
what according to my reading of the manual is necessary in order to
guarantee correct operation.

The c0 hazard between the mtc0 and tlbwr is also taken care of by the
branch instruction itself, there is only one instruction necessary and
as per r4k manual any instruction does.

Yes, it's obfuscating.  However David Miller put this in the code, I
guess he took this trick from IRIX and I took it from him.  I think
it's highly unlikely that we all made the same mistake - and that nobody
got bit by this for a the past few years.

That said, the whole TLB exception handling is sometimes very performance
sensitive.  We've got good reason to optimize it at almost any price.
So the whole thing should be rewritten anyway.

> as much delay as you think.

On the R4600 and R5000 the branch will be taken with a delay of a single
cycle, that is the instruction in the branch delay slot and no additional
penalty.  That's exactly what we want.

> be avoided whenever possible.

That's why we have a number of implementations of TLB exception handlers.
Let me reiterate that they are extremly performance sensitive.

> If you want to be assured of

Indeed, and I'll do that.  The fact that we do not handle this bug is
simply an omission on my side.  The conditions that need to be true for
this bug to actually hit a system are sufficiently rare that I don't
wonder why nobody ever got hit by this.

> stable.   To which bug and which workaround are you referring?

Sorry, I permanently missattribute the R5230 to IDT even though the manual
says different in big letters on the frontpage.  This is the exception
handler from the Cobalt kernel.

	/*
	 * This version has a bug workaround for the Nevada.  It seems
	 * as if under certain circumstances the move from cp0_context
	 * might produce a bogus result when the mfc0 instruction and
	 * it's consumer are in a different cacheline or a load instruction,
	 * probably any memory reference, is between them.  This is
	 * potencially slower than the R4000 version, so we use this
	 * special version.
	 */
	LEAF(except_vec0_nevada)
	.set	mips3
	mfc0	k0, CP0_BADVADDR		# Get faulting address
	mfc0	k1, CP0_TAGLO
	srl	k0, k0, 22			# get pgd only bits
	sll	k0, k0, 2
	addu	k1, k1, k0			# add in pgd offset
	lw	k1, (k1)
	mfc0	k0, CP0_CONTEXT			# get context reg
	srl	k0, k0, 1			# get pte offset
	and	k0, k0, 0xff8
	addu	k1, k1, k0			# add in offset
	lw	k0, 0(k1)			# get even pte
	lw	k1, 4(k1)			# get odd pte
	srl	k0, k0, 6			# convert to entrylo0
	mtc0	k0, CP0_ENTRYLO0		# load it
	srl	k1, k1, 6			# convert to entrylo1
	mtc0	k1, CP0_ENTRYLO1		# load it
	tlbwr					# write random tlb entry
	nop
	nop
	eret					# return from trap
	END(except_vec0_nevada)

Note that this is a handler for Cobalt's 2.0 so it's not a drop in
replacement for what we have in other kernels.

What we observed was that the first access of the kernel to virtual
memory, that was buffer_init() which was still using vmalloc'ed memory
somehow was not being processed correctly.  We've never observed that the
bug hits at some other place.  The explanation given in above comment is a
possible explanation of what I suspect after playing with exception handler
modifications for a few hours.  Since the CPU is a blackbox for me I cannot
guarantee that this explanation is correct or even close to reality.

Two people at Cobalt did run independently into this problem and found that
c0_context was corrupted.  However when later on some of there other at
Cobalt were talking to QED they denied the possible existence of a bug like
this in their silicon.  Having an empirically correct workaround for this
problem we then stopped further tracking this problem.

  Ralf