Subject: Question concerning cache coherency

From: Jeff Harrell <[email-masked]>
Date: Wed, 19 Jan 2000 10:00:11 -0700

I have an interesting issue that I would like to run past the MIPS/Linux
newsgroup.  I am
currently porting the MIPS/Linux code to a development board that has a
IDT64475 MIPS
core (64-bit R4xxx core).  I notice that this part does not have any
method of maintaining
cache coherency (i.e., no hardware support for cache coherency).  It is
highly likely that we
will be plugging in a network card on a PCI bus that would be DMA'ing to
a shared memory
space in SDRAM.  I assume that the problem of cache coherency is fixed
by mapping the shared
memory as uncached.  I have not dug into the network drivers (or the
kernel) enough to know whether
this is how the problem is addressed on typical MIPS architectures.  I
guess I have two questions
related to this issue;  Do devices that DMA, typically access uncached
memory  and if so, is a second buffer
required to copy from kernel to user space?  The second question is
concerning the performance hit in
running out of uncached memory,  Have people seen significant
performance degradation when
using uncached memory.  Any insight that anybody can provide would be
greatly appreciated.

Thanks,
Jeff

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Jeff Harrell                    Work:  (801) 619-6104
Broadband Access group/TI
[email-masked]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

---

From: Ralf Baechle <[email-masked]>
Date: Thu, 20 Jan 2000 00:22:20 +0100

On Wed, Jan 19, 2000 at 10:00:11AM -0700, Jeff Harrell wrote:

> newsgroup.  I am currently porting the MIPS/Linux code to a development
> board that has a IDT64475 MIPS core (64-bit R4xxx core).  I notice that
> this part does not have any method of maintaining cache coherency (i.e.,
> no hardware support for cache coherency).  It is highly likely that we
> will be plugging in a network card on a PCI bus that would be DMA'ing to a
> shared memory space in SDRAM.  I assume that the problem of cache
> coherency is fixed by mapping the shared memory as uncached.  I have not
> dug into the network drivers (or the kernel) enough to know whether this
> is how the problem is addressed on typical MIPS architectures.  I guess I
> have two questions related to this issue; Do devices that DMA, typically
> access uncached memory and if so, is a second buffer required to copy from
> kernel to user space?  The second question is concerning the performance
> hit in running out of uncached memory, Have people seen significant
> performance degradation when using uncached memory.  Any insight that
> anybody can provide would be greatly appreciated.

The performance hit by using uncached memory is tremenduous.  Avoid it, if
you can.  Even if you cannot exploit the locality effects of caches you will
still gain from cached access because of prefetch / burst access and write
gathering.

The is one special case where you can not use caching, that is a cacheline
worth of data might concurrently be manipulated both by both processor and a
DMA device.  The typical example are processors with 32-byte cache lines
like the R4000 and a Ethernet chip like the Sonic which has ring entries of
only 16 byte size.  For such a configuration there is a case where

  1)  processor fetches cacheline
  2)                                   NIC write to that cacheline
  3)  processor writes cacheline back

-> the processor just corrupted the NIC written data.

The only way you can deal with that is by either stopping the NIC which you
don't want to or by using uncached access.

Take a look at the bottom of <asm/io.h> which defines three functions which
do the cache flushing for you.  On machines that are cache coherent by
hardware like SGI's Origins these functions will simply be no-ops.

  Ralf