Subject: Memory corruption

From: Ulf Carlsson <[email-masked]>
Date: Tue, 22 Jun 1999 03:39:31 +0200

Hi,

The compiler may stop working sometimes on certain files, giving bogus error
messages which I don't understand (the compiler is probably not the only
application affected).  Running this program I just wrote forces the corrupted
caches to be flushed or something and ``fixes'' the problems:

int main(void)
{
	unsigned long tot = 0;
	unsigned long i = 1 << 20;
	void *p;
	int failures = 0;

	while (i) {
		p = malloc(i);
		if (!p) {
			if (failures++ < 10)
				continue;
			i = i >> 1;
			failures = 0;
			continue;
		}
		memset(p, 0, i);
		tot += i;
	}
	printf("Total memory set: %u kb\n", tot >> 10);
}

Maybe I should put this in my crontab along with sync :-)

Does anyone else notice these problems?

- Ulf

---

From: "William J. Earl" <[email-masked]>
Date: Tue, 29 Jun 1999 18:01:47 -0700

> caches to be flushed or something and ``fixes'' the problems:
...

      This problem sounds like a cache flushing problem.  Do you also
get SIGILL, SIGBUS, and SIGSEGV failures?  One possibility is that the icache
is not being flushed on a page fault, when a page is read in from disk,
and the icache still has old data in it.  This could lead to a cache line
of bogus instructions being executed.

      What model of CPU do you have in your machine?

---

From: Ulf Carlsson <[email-masked]>
Date: Wed, 30 Jun 1999 04:47:02 +0200

>  > The compiler may stop working sometimes on certain files, giving bogus
>  > error messages which I don't understand (the compiler is probably not the
>  > only application affected).  Running this program I just wrote forces the
>  > corrupted caches to be flushed or something and ``fixes'' the problems:
> ...
> 
>       This problem sounds like a cache flushing problem.  Do you also get
>       SIGILL, SIGBUS, and SIGSEGV failures?  One possibility is that the
>       icache is not being flushed on a page fault, when a page is read in from
>       disk, and the icache still has old data in it.  This could lead to a
>       cache line of bogus instructions being executed.

Sometimes when this happens I think I only get a SIGSEGV or a SIGBUS, otherwise
I get internal compiler errors.  It's hard to say since these problems are very
hard to reproduce, and I forget what happens from time to time.  I have
unfortunately not written down the results.  It sounds like this may be the
cause of the type of file corruption I have when only a little part of the file
is damaged (sounds like the problem covers both icache and dcache).  That type
of file corruption goes away after reboot.  I haven't had a chance to try this
with my discard-disk-cache program since this happens very seldom..

>       What model of CPU do you have in your machine?

I have a 133 MHz R4600 with 512kb board cache, 16kb dcache and 16kb icache.

Regards,
Ulf

---

From: "William J. Earl" <[email-masked]>
Date: Wed, 30 Jun 1999 15:01:27 -0700

Ulf Carlsson writes:
...
 > I have a 133 MHz R4600 with 512kb board cache, 16kb dcache and 16kb icache.

     I have been looking at the fault handling and the cache flushing routines
for the R4600.  In do_no_page() in mm/memory.c, we have:

	flush_page_to_ram(page);

I don't see where any code invalidates the icache, which might have
cached lines from a previous incarnation of the page.
flush_page_to_ram(), for the R4600, essentially does a writeback of
the dcache, if I understand the code correctly.  I believe that an
icache invalidate is also needed, at least for executable pages
(including any page for which mprotect() with PROT_EXEC has been
called, not just for text pages from an executable file).  Also,
unless something has changed, my understanding is that conflicting
virtual aliases (in the dcache) are still possible, which will also
lead to data corruption when it happens.

     In particular, if process A mmaps a file page at virtual index
0 and process B happens to mmap the same file page at virtual index
1, they will in general corrupt each other's view of the data.

     There is a comment in memory.c that a non-present page shouldn't
be cached, but it is not yet clear to me that this is guaranteed for
the icache.  Also, the flush_page_to_ram() slows down processing on
machines which physical cache tags, for cases where the virtual
index used by the kernel and the virtual index used by the application
are the same.  It should have an extra argument of the intended user virtual
address, so that it can decide whether to flush or not on architectures
such as MIPS.

    Handling the virtual index conflicts requires dynamic ownership
switching (including cache flushing), which means that we have to record
those hardware-valid PTEs currently referencing the page, so that we can
invalidate the PTEs and flush the cache when a fault happens for a mapping
of a different color.  We could take a brute-force approach, and record
just one mapping, forcing a fault on each use of a different message,
which would allow us to keep the reverse map in an array parallel to mem_map,
or we could use some more complex structure to record mappings.  Also,
to reduce the frequency of conflicts, address assignment in do_mmap()
should take cache color into account on machines with virtually indexed
caches which lack hardware cache coherency (such as the R4000PC, R4600,
and R5000).

---

From: "William J. Earl" <[email-masked]>
Date: Wed, 30 Jun 1999 17:53:58 -0700

Ralf Baechle writes:
...
 > Oh, the common case is either shared r/o mappings or SysV SHM which per
 > ABI is 64kb aligned, so the hairy case doesn't hit us.  Usually ...
 > 
 > Especially I don't see why anything should corrupt executable pages
 > which are r/o mapped.

     Suppose physical page X has been used as logical page 100 of executable
file ABC, and is then freed, but is still partially in the icache at 
virtual index 0.  Then suppose the page X is reused as logical page 200 of
executable DEF, at virtual index 0.  The writeback of the data cache is
good, but there are still cache lines from file ABC in the icache.  If
nothing flushes the icache (and there is no reason to flush the icache
when reusing a page for data), the icache will have stale data with respect
to the new identity of page X as logical page 200 of executable DEF.

     Also, if there are incompatible aliases for a page, and there are
dirty lines left in the cache when the mapping for, say, virtual index
1 is released, and then the mapping for virtual index 0 is also released,
and the page, which has KSEG0 virtual index 0 is used for I/O, the normal
flushing will flush only virtual index 0.  A later victim writeback of
the dirty lines for virtual index 1 will overwrite the new data with
stale data, even if the new data is instructions.  This case can apply
even if the one alias is a kernel KSEG0 alias and the other is a 
user alias.  For regular file I/O, this is not a problem, but it is a problem
with mmap(), particularly since Linux mmap() makes no attempt to keep multiple
mappings of the same page of a file color-congruent.  (mmap() addresses
are essentially arbitrary.)  

     The icache issue applies to all processors.  The dcache issue applies only
to the R4000PC, R4600, and R5000.

 > > the icache.
 > Flushing the caches for pages which are being unmapped is done by
 > flush_cach_page and takes care of the VM_EXEC flag.
 > 
 > On exec, fork or exit we flush the entire cache so that problems shouldn't
 > hit us either.

      It is not clear this works as expected if the page is stolen by
vmscan.

 > Actually we're pretty generous with our cacheflushed, we flush more than we
 > should.

     Yes, but it is not clear that all paths are covered.

 > > Also, the flush_page_to_ram() slows down processing on
 > For R3000 and R6000 flush_page_to_ram() is a no-op, see arch/mips/mm/r2300.c
 > and arch/mips/mm/r6000.c.

    Yes, since those have write-through caches.  The icache
invalidation is still an issue, if there are any paths, such as
try_to_swap_out(), which break a virtual-to-physical mapping without
flushing the icache.

 > For virtual indexed CPUs something like change_page_colour(oldvaddr, newvaddr)
 > would usually do a more efficient job than always flushing the page to
 > memory especially when combined with an allocator which takes the vaddr where
 > the page will be mapped as a hint.

      Right.  Also, for IRIX and RISCos, I had mmap prefer an mmap
address for which color(address) == color(file_offset), so that
applications not using MAP_FIXED would always map a given file page at
the same virtual color, and I had the kernel use page_mapin() to make
a page addressable, so that I could have page_mapin() create a KSEG2
mapping of the appropriate color if it were different from the KSEG0
color of the page (for cases where the allocator could not allocate a
page with KSEG0 color to match the desired virtual color).
page_mapin() would of course return the KSEG0 address if the KSEG0
color matched the virtual color.  The color changing code is still
neaded to deal with MAP_FIXED and so on, but it is much less
performance-critical.