Subject: R5000 caches

From: Ralf Baechle <[email-masked]>
Date: Thu, 11 Sep 1997 05:51:07 +0200 (MET DST)

Hi,

Can anybody give me a pointer to where the R5000 caches, especially
the cache instruction, are documented?  My two IDT R5000 manuals don't
contain the least bit of information regarding the cache instruction.
I'm primarily interested in how the indexed operations select the
cache set of the primary caches to operate on.  On the R4600 which has
16kb per cache bit 13 selects the set.  So I assume it's bit 14 on the
R5000 with it's 32kb per cache?  The code from David handles the
R5000 like a R4000 CPU but this doesn't look very credible to me as
this is a QED CPU and the other members of the R5k family like the
Nevada (which run Linux now also!) have two way primary caches.

  Ralf

---

From: "David S. Miller" <[email-masked]>
Date: Thu, 11 Sep 1997 00:09:23 -0400

From: Ralf Baechle <[email-masked]>
   Date: Thu, 11 Sep 1997 05:51:07 +0200 (MET DST)

   Can anybody give me a pointer to where the R5000 caches, especially
   the cache instruction, are documented?  My two IDT R5000 manuals
   don't contain the least bit of information regarding the cache
   instruction.  I'm primarily interested in how the indexed
   operations select the cache set of the primary caches to operate
   on.  On the R4600 which has 16kb per cache bit 13 selects the set.
   So I assume it's bit 14 on the R5000 with it's 32kb per cache?  The
   code from David handles the R5000 like a R4000 CPU but this doesn't
   look very credible to me as this is a QED CPU and the other members
   of the R5k family like the Nevada (which run Linux now also!) have
   two way primary caches.

In SGI boxes, if my memory serves, the R5k's were the chips which
needed the special:

	cli();
	jump_into_64_bit_mode();
	{peek,poke}_magic_physaddr();
	leave_64_bit_mode();
	sti();

sequence, both to enable/disable the cache and to perform flush
operations.  Although this could have been for the R4600.  I do
remember that IRIX had special code to work the R5k caches, but this
might have been for the L2 cache operations only, not L1.

All of this was very SGI specific and was mostly for the L2 cache
operations.  I think the R5k had a special "flush command" which would
just pull a pin going to the cache and invalidate all the lines in one
cycle (earle told me this, he may be able to elaborate).

The R5k, by design at least in the SGI boxes, lacks a L2 cache, it was
added externally on the SGI motherboard's, and thus all the funcy
methods to access/enable/disable/flush it...  Again, I could be
confusing r4600 and r5k here, so who knows.

Later,
David "Sparc" Miller
[email-masked]

---

From: Ralf Baechle <[email-masked]>
Date: Thu, 11 Sep 1997 06:28:17 +0200 (MET DST)

> operations.  Although this could have been for the R4600.  I do

Afaik it's for both CPUs.

> cycle (earle told me this, he may be able to elaborate).

It's the L2 enable bit for the "true", means CPU controlled L2 cache
of the R5000.  The fact that we're dealing with two different types
of L2 caches for the R5000 complicates things slightly.  By design
the R5000 supports an external cache.  What SGI did was more or less
just ignoring that feature and using the R5000 as a R4600 replacement.
That's why they're using an external cache which is controlled by
the chipset.

> confusing r4600 and r5k here, so who knows.

Well, your memory serves right.  However my problem are especially
the primary caches (I've got not box with a "true" R5000 L2 cache), so
that doesn't solve my problem.  As far as I can tell your R4600 code seems
to work for the R5000 Indy with the "fake" cache.  I suspect that the
problem handling the L1 caches was causing the disk corruption I observed
on my Indy while on this Nevada box I'm using here SCSI works as realiable
as it is supposed to be.

Anyway, there is still enough fun to have.  I'm planning a little
lmbenchmania followed by a cycle-massacre and I'm just hacking
microsecond timers for that ...

  Ralf

---

From: "William J. Earl" <[email-masked]>
Date: Thu, 11 Sep 1997 18:02:20 -0700

> Afaik it's for both CPUs.

     It applies to the secondary cache on the Indy R4600SC and R5000SC
and the Indigo2 R4600SC, but not to any other systems or processors.

     For primary caches, the set is selected by the high order bit of
the cache index (virtual address bit 13 on the R4600, virtual
address bit 14 on the R5000).  (Note that the R10000 is different: it
uses the low order address bit to select the set.)  

 > That's why they're using an external cache which is controlled by

       Yes, that is correct for hte Indy, but the O2 R5000SC does
use the real R5000 secondary cache.  The L2 enable bit is set on O2
R5000SC and is not set on the Indy.

 > on my Indy while on this Nevada box I'm using here SCSI works as realiable
...

      If you were using the index bits incorrectly, that would cause
disk corruption, and, as noted above, the set selection bit differs,
due to the size of the cache.  Basically, if the size of the cache is
X, then (X >> 1) is the set selection bit.  The R4600SC secondary
cache code should apply equally to the Indy R5000SC, without change.

---

From: Ralf Baechle <[email-masked]>
Date: Fri, 12 Sep 1997 03:22:15 +0200 (MET DST)

Hi,

> uses the low order address bit to select the set.)  

Indeed, but currently our only R10k problem is that we don't support
that thing at all ...

> cache code should apply equally to the Indy R5000SC, without change.

Ok, thanks for the information.  It's what I was expecting.  In that
case the version which I'm just about to check in is still not R5000
safe.  I'm going to fix that rsn.

Quickfix for R5000 hackers: change the #define for waybit in
arch/mips/mm/r4xx0.c from 0x2000 to (dcache_size >> 1).

  Ralf

---

From: "William J. Earl" <[email-masked]>
Date: Thu, 11 Sep 1997 18:34:54 -0700

Ralf Baechle writes:
...
 > arch/mips/mm/r4xx0.c from 0x2000 to (dcache_size >> 1).

      By the way, in case you are not familiar with the problem, watch
out for the R4600 errata in regard to index invalidate.  Basically,
you need to disable interrupts while using the index invalidate
and index writeback invalidate instructions on R4600 Rev. 1.7, so that
invalidating a given line in one set is atomic with invalidating
the corresponding line in the second set.  You can turn on interrupts
periodically, but it is important to do both sets for a given index
at the same time, without the possibility of an interrupt or exception.
The errata can result in the same line winding up in both sets,
leading to stale data.  The bug is not present in the R4600 Rev. 2.0
or the R5000.

---

From: Ralf Baechle <[email-masked]>
Date: Fri, 12 Sep 1997 03:57:20 +0200 (MET DST)

> leading to stale data.  The bug is not present in the R4600 Rev. 2.0

There is a more elegant solution for this bug.  Just leave the interrupts
as they are and invalidate set b before set a.  Anyway, Linux currently
still handles this erratum by disabling interrupts.

  Ralf