Subject: RE: problems on D-cache alias in 2.4.22

From: "Bob Breuer" <[email-masked]>
Date: Tue, 18 May 2004 13:17:38 -0500

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Peter Horton
> Sent: Friday, May 14, 2004 2:53 AM
> To: wuming
> Cc: [email-masked]
> Subject: Re: problems on D-cache alias in 2.4.22
> 
> 
> wuming wrote:
> 
...
> > at last, when I replaced flush_page_to_ram( ) with 
> flush_dcache_page( ),
> > the internal compiler error disappeared.
> >
...
> 
> This is probably just hiding your problem. flush_page_to_ram() is not 
> used anymore.
> 
> P.
> 
> 

Changing that same place also fixes my problem.  However, I came across
the mips cobalt patches and after applying a variation of the IDE cache
fix from there, that also fixes the problem.  So it would seem that this
is the same problem as already fixed in the cobalt patch, but showing up
on non-cobalt hardware.

flush_page_to_ram() was made useless around the release of 2.4.21.  I
suspect that this was broken at that time, seeing how it is broken in
2.4.22 and 2.4.26.  From browsing the debian-mips mailing list archives,
it appears that they have not had a stable mips kernel since 2.4.19,
could this bug be the cause?  Are the recent Debian mips kernels still
unstable?

Would anyone with an unstable 2.4.2x kernel be willing to try one of the
attached patches to see if the situation improves?

Bob

---

From: Jun Sun <[email-masked]>
Date: Tue, 18 May 2004 11:45:19 -0700

On Tue, May 18, 2004 at 01:17:38PM -0500, Bob Breuer wrote:
> Changing that same place also fixes my problem.  

<snip>

Like others suggested, this is not the right fix.  flush_page_to_ram()
is correctly nullified.  Its job should be done somewhere else
by other routines.

Here are a couple of random ideas for finding the true root cause:

. If a page is shared by multiple user processes, make sure either the CPU
  does not have d-cache alaising problem (i.e., cache way size is 4KB or less)
  or their virtual addresses lie on the "same color strip" of the d-cache.
  In other words, they would be cached in the same cache way.

. If a page is modified by kernel and accessed by user land, make sure a 
  flush_dcache_page() is called right after the modifying.

. If a page is modified by userland and accessed by kernel, I _think_ currently
  kernel would still do a flush_dcache_page() call.  However, this won't
  work on MIPS because the cache at user virtual addresses are not flushed.
  Either try to flush with user virtual address, or do a flush_cache_all(). *ick*

BTW, I _think_ the last problem stilled exists in 2.6.  We probably need
to use the reverse maping info to fix it.

Jun

---

From: Peter Horton <[email-masked]>
Date: Tue, 18 May 2004 20:10:19 +0100

> Here are a couple of random ideas for finding the true root cause:

We know what the true root cause is :-)

IDE PIO fills the D-cache with the read data (write allocate) as it
copies it to the page cache.

The kernel maps the page cache page into user space ... BANG! possible
D-cache alias.

The kernel doesn't bother flushing the page cache page from the D-cache
as it's never accessed at it's page cache address.

The current fix in the Cobalt patches (2.4 & 2.6) just flushes the read
data out of the D-cache after every IDE insw()/insl(). This is the least
intrusive fix.

Some Sparc machines also see this problem.

P.

---

From: Ralf Baechle <[email-masked]>
Date: Tue, 18 May 2004 21:50:55 +0200

> as it's never accessed at it's page cache address.

It is - after all the driver is copying the data to there.  The same
problem also exists in the ramdisk driver and there it has been fixed
properly, it seems.

> Some Sparc machines also see this problem.

Carelessly written PIO drivers on any architecture would suffer from this
kind of problem.

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Tue, 18 May 2004 22:02:05 +0200

> attached patches to see if the situation improves?

flush_page_to_ram has been deprecated since a long, long time and so it's
use in memory managment was no longer correct for all cases - MIPS was
basically the last major Linux architecture left using it.  Replacing
it with flush_dcache_page fixed those correctness problem and delivered
a major speedup.  So no sense in whining - flush_page_to_ram won't return.

  Ralf

---

From: Thiemo Seufer <[email-masked]>
Date: Tue, 18 May 2004 22:10:10 +0200

Bob Breuer wrote:
[snip]
> From browsing the debian-mips mailing list archives,

The latest 2.4.25/2.4.26 (Debian-)Kernels seem to work well on all
Debian-supported machines. They include some cobalt patch which
isn't in the linux-mips.org CVS (yet).

Thiemo

---

From: Peter Horton <[email-masked]>
Date: Tue, 18 May 2004 22:21:53 +0100

> problem also exists in the ramdisk driver and there it has been fixed

I had a dig around but couldn't decide where the proper fix should go.

> Carelessly written PIO drivers on any architecture would suffer from this

Are you saying it's a driver problem ? :-)

P.

---

From: Alan Cox <[email-masked]>
Date: Tue, 18 May 2004 19:24:34 +0100

> Carelessly written PIO drivers on any architecture would suffer from this

It isnt a driver problem. (Ramdisk is a bit special as you get aliases
in funny ways). The block and page cache is responsible for sorting this
out.

---

From: Jun Sun <[email-masked]>
Date: Tue, 18 May 2004 15:25:39 -0700

> as it's never accessed at it's page cache address.

The kernel (or driver) should flush the page if it is mapped to user space 
and the content is modified.

> data out of the D-cache after every IDE insw()/insl(). This is the least

It should be fixed at a higher layer before we return back to userland.

If you can illustrate the call stack, I can probably take a look and
give my opinion.

Jun

---

From: Peter Horton <[email-masked]>
Date: Wed, 19 May 2004 00:29:21 +0100

> and the content is modified.

We just need a hook so that we can flush a page from the D-cache once
it's read from a block device into the page cache.

> If you can illustrate the call stack, I can probably take a look and

No call stack, sorry. It was months ago that I debugged this.

IIRC I picked up the aliases with memcmp() in do_no_page() in
mm/memory.c.

P.