Subject: Re: Indy crashes

From: "Kevin D. Kissell" <[email-masked]>
Date: Mon, 14 Feb 2000 22:15:02 +0100

>On Thu, 3 Feb 2000, Ralf Baechle wrote:
>> Today I exchanged the R5000 CPU module in my Indy against a R4600 module
>> and found that on R4600SC the kernel runs reliable while it crashs pretty
>> soon after booting on a R5000SC module.  This is consistent with the
>> reports that I've looked at.
>
>That could explain the crashes I see on the DDB Vrc-5074 as well, which has a
>NEC VR5000.
>
>I'll try to fix the bootmem stuff ASAP and upgrade to 2.3.38.

The problem may be in the assumption made even in the
most recent 2.3.x code that a hit-writeback-invalidate cache
operation on the secondary cache automagically affects the
primary.  That's the way it is on the R4000/4400, but that's
not the way the R5000 is specified.  So rather than set
dma_cache_wback_inv to r4k_dma_cache_wback_inv_sc
or r4k_dma_cache_wback_inv_pc, depending on the
presence or absence of a primary cache,  in the MIPS 
Technologies I bound it to a function:

static void
r4k_dma_cache_wback_inv(unsigned long addr, unsigned long size)
{
        r4k_dma_cache_wback_inv_pc(addr, size);
        if(sc_present) r4k_dma_cache_wback_inv_sc(addr, size);
}

However, I have not had the opportunity to test this code on
an R5000SC platform.   Looking at the R5000 spec, it is a
little ambiguous.  The special case of sc HWIs affecting
the primary isn't there, but then again sc HWIs aren't even
called out in the table of defined cache operations.  Indeed,
one *could* interpret the spec to mean that HWI on the 
*primary* flushes the secondary, the reverse of the R4000,
but it's by no means clear.   Thus I suggest hitting 'em both.

Does anybody on this list have an R527x manual?   How
is HWI of the primary/seconday caches defined there?

            Regards,

            Kevin K.

---

From: "William J. Earl" <[email-masked]>
Date: Mon, 14 Feb 2000 13:57:23 -0800 (PST)

Kevin D. Kissell writes:
...
 > is HWI of the primary/seconday caches defined there?

      I am very familiar with how the R5000 and similar processors
works.  The secondary cache is essentially independent of the primary
caches, and is write-through.  It hangs on the SysAD bus and captures
cache lines being read, and returns a captured cache line on a later
read.

     You must separately invalidate the primary and secondary caches.
If you care about instruction cache coherency (as when reading in executable
pages), you have to invalidate the primary instruction cache, not just
the primary data cache.

     Since the secondary cache is write-through, you need only invalidate
it on reads; you can ignore it on writes.

---

From: Ralf Baechle <[email-masked]>
Date: Tue, 15 Feb 2000 01:13:46 +0100

> primary.  That's the way it is on the R4000/4400, but that's

Yep.

> Technologies I bound it to a function:

I don't even pretend that Linux is running on a R5000 with L2 except on
Indy R5000SC's.  These R5000 modules are different in that they don't use
the L2 support which is part of the processor but rather use the same
external cache implementation as the R4600SC CPU modules do.

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Tue, 15 Feb 2000 01:37:39 +0100

On Mon, Feb 14, 2000 at 11:45:06PM +0000, Dominic Sweetman wrote:

> QED's RM7000 has an internal secondary cache which is more like (but
> not compatible with) an R4000.  As part of our Linux familiarisation
> we'll try to get that working.  It's 4-way set associative, 256Kbytes
> big, unified and write-back; so that should shake out some more bugs.

[...]

> I used to be an advocate of the MIPS position; but what this argument
> ignores is the difficulty of maintaining a portable software base,
> particulary an OS which (like Linux) has grown up with invisible x86
> caches.
> 
> To make sure that Linux/MIPS kernels are and remain robust we have to
> convince lots of people to take the trouble to learn about visible
> caches - even though the systems they're working on don't have them.
> And we have to hope that driver writers will do the right thing, even
> though they've never used a system which needs cache management code
> and have none at hand to test their drivers with...
> 
> This would be unlikely even if we all agreed what the word 'flush'
> meant (does it mean "invalidate"?  or "write-back"?  or perhaps "both,
> if necessary"? and can you build a robust OS without knowing the
> answer?)

Yes.

> It *could* happen.  MIPS are not the only family of CPUs with visible
> caches - though I think they probably have by far the most complicated
> ones, because of their high-end history.

Sparc.

> The meeting of Linux and MIPS is in this respect really good for Linux
> - great at finding portability problems.  But it's not particularly
> good for MIPS.  For now, Linux/MIPS kernels will only be reliable if
> maintained by people who understand this stuff, keep a safe distance
> from the x86 mainstream, and use only sanity-checked drivers.

No longer.  With Linux 2.3.41 Linux received a number of new software
interfaces which also could easily be backported to previous kernels.
As the side effect you even get scatter gather capabilities for every
driver.

Btw, please take a look at the new interfaces, they're documented in
Documentation/DMA-mapping.txt.  They're not yet cast into stone, so if
anybody has something to complain about these interfaces, better do it
soon.

> Kevin, I think you'd better tell your colleagues who design the chips
> that they'd better put some snooping in, and soon...

For I/O isn't not that bad to handle except a few cases like the
NCR 53c8xx driver.  And there is always the silver bullet solution of
using uncached accesses.

Things like mmap() and read/write right in the presence of virtual indexed
caches are the real vomit bag.  Linus & Co. so far refuse to accept any of
the optimal solutions.  And while I'd of course would like to such a
solution to get into the kernel I can well understand Linus' refusal.
>From that point of view the RM7000 is the first MIPS CPU that got caches
right since the R6000 ...

  Ralf

---

From: "William J. Earl" <[email-masked]>
Date: Tue, 15 Feb 2000 09:49:19 -0800 (PST)

...
...

     If someone wants to do the real R5000SC (and RM5271 and RM7000)
cache routines, for some other platform, I can supply the required 
details.

---

From: [email-masked]
Date: Tue, 15 Feb 2000 10:46:00 -0800

> primary.  That's the way it is on the R4000/4400, but that's

In that case it doesn't affect the NEC DDB Vrc-5074, since there's no L2.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven ------------- Sony Software Development Center Europe (SDCE)
[email-masked] ------------------- Sint-Stevens-Woluwestraat 55
Voice +32-2-7248632 Fax +32-2-7262686 ---------------- B-1130 Brussels, Belgium