Subject: R4600 fun ...

From: [email-masked]
Date: Mon, 24 Nov 1997 14:49:42 +0100

Hi,

I've found a problem with the cacheflush routines in the kernel.  The
following routine from arch/mips/mm/r4xx0.c:

static void r4k_flush_cache_sigtramp(unsigned long addr)
{
	addr &= ~(dc_lsize - 1);
	/* These nops handle a processor errata in the R4600 silicon */
	__asm__ __volatile__("nop;nop;nop;nop");
	protected_writeback_dcache_line(addr);
	protected_writeback_dcache_line(addr + dc_lsize);
	protected_flush_icache_line(addr);
	protected_flush_icache_line(addr + dc_lsize);
}

which is being compiled into the following assembler code:

0000d5a0 <r4k_flush_cache_sigtramp> lui $v1,0x0
0000d5a4 <r4k_flush_cache_sigtramp+4> lw $v1,12($v1)
0000d5a8 <r4k_flush_cache_sigtramp+8> negu $v0,$v1
0000d5ac <r4k_flush_cache_sigtramp+c> and $a0,$a0,$v0
...
0000d5c0 <r4k_flush_cache_sigtramp+20> cache Hit_Writeback_Inv_D,0($a0)
0000d5c4 <r4k_flush_cache_sigtramp+24> addu $v1,$a0,$v1
0000d5c8 <r4k_flush_cache_sigtramp+28> cache Hit_Writeback_Inv_D,0($v1)
0000d5cc <r4k_flush_cache_sigtramp+2c> cache Hit_Invalidate_I,0($a0)
0000d5d0 <r4k_flush_cache_sigtramp+30> cache Hit_Invalidate_I,0($v1)
0000d5d4 <r4k_flush_cache_sigtramp+34> jr $ra
...

... is called by setup_frame() in arch/mips/kernel/signal.c when
setting up a signal frame:

[...]
	/*
	 * Set up the return code ...
	 *
	 *         .set    noreorder
	 *         addiu   sp,0x20
	 *         li      v0,__NR_sigreturn
	 *         syscall
	 *         .set    reorder
	 */
	__put_user(0x27bd0000 + scc_offset, &frame->code[0]);
	__put_user(0x24020000 + __NR_sigreturn, &frame->code[1]);
	__put_user(0x0000000c, &frame->code[2]);

	/*
	 * Flush caches so that the instructions will be correctly executed.
	 * (flush_cache_sigtramp is a function pointer)
	 */
	flush_cache_sigtramp((unsigned long) frame->code);
[...]

Unless I've got tomatoes of the size of a space ship on my eyes these
cache flushes do the right thing in order to guarantee the coherence
of Icache with the Dcache.  Nevertheless when executing the code on a
R4600 v2.0 I get illegal instruction exceptions for the instructions
on the stack.  The other MIPS processors seem not to have any problem
with this code.

Silicon problem?  Anybody seen this before?

  Ralf

---

From: "William J. Earl" <[email-masked]>
Date: Mon, 24 Nov 1997 10:45:56 -0800

[email-masked] writes:
...
...
 > Silicon problem?  Anybody seen this before?

     This is a known R4600 Rev. 2.0 bug.  From the R4600 errata list:

        3.  The CACHE instructions Hit_Writeback_Invalidate_D,
            Hit_Writeback_D, Hit_Invalidate_D, and
            Create_Dirty_Exclusive_D will only operate correctly if
            the internal data cache refill buffer is empty.  These
            cache instructions should separated from any potential
            data cache miss by a load instruction to an uncached
            address to empty the response buffer.

A brute-force workaround for this is to do the load instruction to an
uncached address (such as a memory controller register) before the
cache invalidation code, and to also do such a load instruction just
before every eret.  

     Alternately, one could put enough nop instructions in front of
every eret to allow time for the refill buffer to empty.  The latter
is what I did on Indy, where 12 cycles are needed to allow the refill
buffer to empty in the worst case.  I arranged to replace every eret
with a labelled jump to a patch area (with a nop in the jump delay
slot), where the patch area was 8 nop instructions and an eret.  I had
a table of the jump addresses, and, at startup time on a CPU other
than an R4600 Rev. 2.0, I patched the jump instructions to eret
instructions, so only the R4600 Rev. 2.0 paid the penalty.

     Yet another alternative would be to disable interrupts around the 
cacheops, probe the tlb to be sure a TLB miss could not occur
during the cacheop, pad 12 nop instructions after the probe,
do the cacheops, and enable interrupts.   (If the TLB probe fails,
enable interrupts, do a load from the first address, and retry
from the beginning.)