Subject: Linux Shared Memory Issue

From: "Maurice Turcotte" <[email-masked]>
Date: Wed, 24 Apr 2002 17:27:12 -0400

Greetings:

I am having a problem with Linux Kernel 2.4.5 on a mips.

I have two processes using share memory for IPC. This same
code works fine with Kernel 2.4.7 on a x86. The problem is 
that the second process reads old data out of the shared
memory.

The executive summary->

Process #1 writes "A" to shared memory at 0x2aac7210
Process #2 reads 0 from shared memory at address 0x2aaca210
Process #1 writes "B" to shared memory at 0x2aac7210
Process #2 read "A" from shared memory at address 0x2aaca210
Process #1 writes "C" to shared memory at 0x2aac7210
Process #2 read "B" from shared memory at address 0x2aaca210

It is interesting that the processes get different addresses
associated with the same shmId. I assume this is because of 
some user-space mapping that is going on.

I left out the semaphore diddling, but I believe that part of
the code is correct because it works flawlessly on the 2.4.7 x86.

Any tips on debugging this would be greatly appreciated. If this
is not the proper forum for questions like this, please point me
in the right direction.

Thanks,

mturc

---

From: Jun Sun <[email-masked]>
Date: Wed, 24 Apr 2002 15:03:15 -0700

Looks like the infamous cache aliasing problem.  Steve Longerbeam had a patch 
which may help.  Please try it and let me know the results.

Thanks.

Jun

> is not the proper forum for questions like this, please point me

diff -Nuar -X /home/stevel/dontdiff linux-2.4.17.orig/arch/mips/kernel/syscall.c linux-2.4.17/arch/mips/kernel/syscall.c
--- linux-2.4.17.orig/arch/mips/kernel/syscall.c	Sun Sep 16 16:29:10 2001
+++ linux-2.4.17/arch/mips/kernel/syscall.c	Thu Jan 24 15:02:18 2002
@@ -24,6 +24,7 @@
 #include <linux/slab.h>
 #include <linux/utsname.h>
 #include <linux/unistd.h>
+#include <linux/shm.h>
 #include <asm/branch.h>
 #include <asm/offset.h>
 #include <asm/ptrace.h>
@@ -53,6 +54,50 @@
 out:
 	return res;
 }
+
+/*
+ * To avoid cache aliases, we map the shard page with same color.
+ */
+#define COLOUR_ALIGN(addr)    (((addr)+SHMLBA-1)&~(SHMLBA-1))
+
+unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr,
+				     unsigned long len, unsigned long pgoff,
+				     unsigned long flags)
+{
+	struct vm_area_struct *vma;
+
+	if (flags & MAP_FIXED) {
+		/*
+		 * We do not accept a shared mapping if it would violate
+		 * cache aliasing constraints.
+		 */
+		if ((flags & MAP_SHARED) && (addr & (SHMLBA - 1)))
+			return -EINVAL;
+		return addr;
+	}
+
+	if (len > TASK_SIZE)
+		return -ENOMEM;
+	if (!addr)
+		addr = TASK_UNMAPPED_BASE;
+
+	if (flags & MAP_SHARED)
+		addr = COLOUR_ALIGN(addr);
+	else
+		addr = PAGE_ALIGN(addr);
+
+	for (vma = find_vma(current->mm, addr); ; vma = vma->vm_next) {
+		/* At this point:  (!vma || addr < vma->vm_end). */
+		if (TASK_SIZE - len < addr)
+			return -ENOMEM;
+		if (!vma || addr + len <= vma->vm_start)
+			return addr;
+		addr = vma->vm_end;
+		if (flags & MAP_SHARED)
+			addr = COLOUR_ALIGN(addr);
+	}
+}
+
 
 /* common code for old and new mmaps */
 static inline long
diff -Nuar -X /home/stevel/dontdiff linux-2.4.17.orig/include/asm-mips/pgtable.h linux-2.4.17/include/asm-mips/pgtable.h
--- linux-2.4.17.orig/include/asm-mips/pgtable.h	Thu Jan 24 14:35:06 2002
+++ linux-2.4.17/include/asm-mips/pgtable.h	Thu Jan 24 14:56:52 2002
@@ -64,6 +64,9 @@
 #define flush_icache_all()		do { } while(0)
 #endif
 
+/* We provide our own get_unmapped_area to avoid cache aliasing */
+#define HAVE_ARCH_UNMAPPED_AREA
+
 /*
  * - add_wired_entry() add a fixed TLB entry, and move wired register
  */

---

From: Atsushi Nemoto <[email-masked]>
Date: Thu, 25 Apr 2002 14:25:18 +0900 (JST)

>>>>> On Wed, 24 Apr 2002 15:03:15 -0700, Jun Sun <[email-masked]> said:
jsun> Looks like the infamous cache aliasing problem.  Steve
jsun> Longerbeam had a patch which may help.  Please try it and let me
jsun> know the results.

jsun> +#define COLOUR_ALIGN(addr)    (((addr)+SHMLBA-1)&~(SHMLBA-1))

Recent sparc64's COLOUR_ALIGN macro have pgoff argument like this.
We should do it same way for MIPS?

#define COLOUR_ALIGN(addr,pgoff)		\
	((((addr)+SHMLBA-1)&~(SHMLBA-1)) +	\
	 (((pgoff)<<PAGE_SHIFT) & (SHMLBA-1)))

Thank you.
---
Atsushi Nemoto

---

From: Florian Lohoff <[email-masked]>
Date: Thu, 25 Apr 2002 17:36:22 +0200

> that the second process reads old data out of the shared

This also shows with x-windows on mips with shared pixmaps.

We see this for a long time and did not come to a solution.

Flo
-- 
Florian Lohoff                  [email-masked]             +49-5201-669912
Nine nineth on september the 9th              Welcome to the new billenium

---

From: Jun Sun <[email-masked]>
Date: Thu, 25 Apr 2002 11:46:04 -0700

> 	 (((pgoff)<<PAGE_SHIFT) & (SHMLBA-1)))

What is the purpose of adding the pgoff part?  To avoid mapping all shared 
regions into the beginning of cache?

Jun

---

From: Atsushi Nemoto <[email-masked]>
Date: Fri, 26 Apr 2002 12:11:07 +0900 (JST)

>>>>> On Thu, 25 Apr 2002 11:46:04 -0700, Jun Sun <[email-masked]> said:
>> #define COLOUR_ALIGN(addr,pgoff) \ ((((addr)+SHMLBA-1)&~(SHMLBA-1))
>> + \ (((pgoff)<<PAGE_SHIFT) & (SHMLBA-1)))

jsun> What is the purpose of adding the pgoff part?  To avoid mapping
jsun> all shared regions into the beginning of cache?

I suppose so.  For example, mmapping offset 0x1000 of /dev/kmem will
return 0xXXXX0000 address if we ignore pgoff.  This can cause aliasing
problem with KSEG0 address (0x80001000).  Adding pgoff, mmap will
return 0xXXXX1000.

In case of IPC shm, pgoff is always 0 so this change does not effect.

---
Atsushi Nemoto

---

From: Florian Lohoff <[email-masked]>
Date: Sat, 27 Apr 2002 23:45:05 +0200

Hi Ralf, *,

On Wed, Apr 24, 2002 at 03:03:15PM -0700, Jun Sun wrote:
> Looks like the infamous cache aliasing problem.  Steve Longerbeam had a 
> patch which may help.  Please try it and let me know the results.

> >Process #2 read "B" from shared memory at address 0x2aaca210

I today tried the patch in conjunction with XFree86 and XSHM on
an Indy R5000/150. Usually all gtk shared-pixmaps get destroyed
and written lines in it. With this patch all these garbage disappears.
As long as there a no serious issues against this patch i would
vote for inclused. It applies against todays cvs with only little
offsets and gave me no problems. It definitly solves some serious
issues with shm.

Flo