Subject: Re: More than 512MB of memory

From: Ralf Baechle <[email-masked]>
Date: Thu, 3 Feb 2005 16:35:20 +0100

On Tue, Jan 18, 2005 at 03:44:51PM +0100, Rojhalat Ibrahim wrote:

> is there anything special I have to do
> when I want to use more than 512MB of memory?
> My Yosemite board works fine with 512MB
> but when I try 1GB it crashes in 32bit mode
> with highmem and also in 64bit mode.
> The boot monitor (PMON) maps the 1024MB
> to physical addresses 0x0000.0000 - 0x4000.0000.

Interesting.  It's supposed to work but I don't have enough memory for
testing this.

  Ralf

---

From: Manish Lachwani <[email-masked]>
Date: Thu, 03 Feb 2005 10:20:56 -0800

>Interesting.  It's supposed to work but I don't have enough memory for
IIRC, you had posted the kernel boot log in a message as well. Can you 
post it again? For both the HIGHMEM case and the 64-bit case.

Thanks
Manish Lachwani

---

From: Ralf Baechle <[email-masked]>
Date: Fri, 4 Feb 2005 01:01:22 +0100

> post it again? For both the HIGHMEM case and the 64-bit case.

http://www.linux-mips.org/archives/linux-mips/2005-01/msg00145.html :-)

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Fri, 4 Feb 2005 01:40:28 +0100

> to physical addresses 0x0000.0000 - 0x4000.0000.

Can you try below patch?

  Ralf

--- linux/arch/mips/mm/c-r4k.c	2004-12-07 02:30:50.000000000 +0000
+++ linux/arch/mips/mm/c-r4k.c	2005-02-04 00:31:34.623814760 +0000
@@ -566,7 +566,10 @@
 
 	if (!cpu_has_ic_fills_f_dc) {
 		unsigned long addr = (unsigned long) page_address(page);
-		r4k_blast_dcache_page(addr);
+		if (addr)
+			r4k_blast_dcache_page(addr);
+		else
+			r4k_blast_dcache();
 		if (!cpu_icache_snoops_remote_store)
 			r4k_blast_scache_page(addr);
 		ClearPageDcacheDirty(page);

---

From: Rojhalat Ibrahim <[email-masked]>
Date: Mon, 07 Feb 2005 09:10:12 +0100

> +++ linux/arch/mips/mm/c-r4k.c	2005-02-04 00:31:34.623814760 +0000

With a slightly extended patch it actually works. But afterwards
I get a lot of Illegal instructions and Segmentation faults, where
there shouldn't be any. Below is the patch I used.

Thanks
Rojhalat Ibrahim

--- linux/arch/mips/mm/c-r4k.c  2005-01-03 10:23:27.000000000 +0100
+++ linux-2.6.10/arch/mips/mm/c-r4k.c   2005-02-07 09:04:27.000000000 +0100
@@ -566,9 +566,17 @@

         if (!cpu_has_ic_fills_f_dc) {
                 unsigned long addr = (unsigned long) page_address(page);
-               r4k_blast_dcache_page(addr);
+               if (addr)
+                       r4k_blast_dcache_page(addr);
+               else
+                       r4k_blast_dcache();
                 if (!cpu_icache_snoops_remote_store)
-                       r4k_blast_scache_page(addr);
+               {
+                       if (addr)
+                               r4k_blast_scache_page(addr);
+                       else
+                               r4k_blast_scache();
+               }
                 ClearPageDcacheDirty(page);
         }

---

From: Ralf Baechle <[email-masked]>
Date: Tue, 8 Feb 2005 00:17:42 +0000

> +++ linux-2.6.10/arch/mips/mm/c-r4k.c   2005-02-07 09:04:27.000000000 +0100

Blowing away the entire S-cache is extreme overkill.  We really want to
avoid this if possible as it'll have serious performance impact.  As for
the RM9000 that means we have a struct page pointer, therefore we know
the physical address of the page and can perform a selective flush on the
second level cache.  See below for a patch which tries that.

Obviously this one only tries to optimize performance a bit further but
doesn't really solve the remaining problem.

  Ralf

Index: arch/mips/mm/c-r4k.c
===================================================================
RCS file: /home/cvs/linux/arch/mips/mm/c-r4k.c,v
retrieving revision 1.101
diff -u -r1.101 c-r4k.c
--- arch/mips/mm/c-r4k.c	7 Feb 2005 21:53:39 -0000	1.101
+++ arch/mips/mm/c-r4k.c	8 Feb 2005 00:18:17 -0000
@@ -566,9 +566,21 @@
 
 	if (!cpu_has_ic_fills_f_dc) {
 		unsigned long addr = (unsigned long) page_address(page);
-		r4k_blast_dcache_page(addr);
-		if (!cpu_icache_snoops_remote_store)
-			r4k_blast_scache_page(addr);
+
+		if (addr)
+			r4k_blast_dcache_page(addr);
+		else
+			r4k_blast_dcache();
+
+		if (!cpu_icache_snoops_remote_store) {
+			if (addr)
+				r4k_blast_scache_page(addr);
+			else {
+				addr = page_to_pfn(page) << PAGE_SHIFT;
+				addr = CKSEG + (addr & ~((1UL << 24) - 1));
+				r4k_blast_scache_page_indexed(addr);
+			}
+		}
 		ClearPageDcacheDirty(page);
 	}

---

From: Rojhalat Ibrahim <[email-masked]>
Date: Tue, 08 Feb 2005 10:57:14 +0100

> Index: arch/mips/mm/c-r4k.c

I presume CKSEG is CKSEG0 in the above patch. With that it works
about the same as before. So do you have any clue what the problem
behind all that really is? Furthermore I still have all those
"Illegal instruction" and "Segmentation fault" messages that
shouldn't be there.

Thanks
Rojhalat Ibrahim

---

From: Ralf Baechle <[email-masked]>
Date: Wed, 9 Feb 2005 00:06:40 +0000

> "Illegal instruction" and "Segmentation fault" messages that

Sorry, yes I indeed meant CKSEG0. And this version of the patch really was
only meant to optimize the large performance impact your previous patch
had; it wasn't meant to fix anything beyond that.

As I can't replicate your configuration I'm still starring at the code to
find what's wrong ...

  Ralf

---

From: Rojhalat Ibrahim <[email-masked]>
Date: Wed, 09 Feb 2005 09:06:42 +0100

> As I can't replicate your configuration I'm still starring at the code to

Ok, thanks. If I can try anything that might help track down
the problem, please let me know.

Rojhalat

---

From: Rojhalat Ibrahim <[email-masked]>
Date: Wed, 09 Feb 2005 16:41:33 +0100

Rojhalat Ibrahim wrote:
> Furthermore I still have all those

Those messages actually go away when I do not
define cpu_has_64bit_gp_regs, which is strange
because the RM9000 definitely has 64bit gp regs.
Any ideas?

Thanks
Rojhalat Ibrahim

---

From: Ralf Baechle <[email-masked]>
Date: Wed, 9 Feb 2005 19:14:44 +0100

> because the RM9000 definitely has 64bit gp regs.

cpu_has_64bit_gp_regs is meant to indicate "CPU has 64-bit registers and
it's actually safe to use them".  This is why it's defined as 0 for all
32-bit kernels where an interrupt would corrupt the upper 32-bits which
we don't save during any kind of exception.

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Thu, 10 Feb 2005 14:40:43 +0100

> the problem, please let me know.

Try this patch which is meant to be used in combination with the previous
patch.  It moves a test which determines if we actually need to perform a
cacheflush to the right place.  That's a bug which is harmless on UP but
a severe bug on SMP.

Thanks,

  Ralf

 c-r4k.c |   14 +++++++-------
 1 files changed, 7 insertions(+), 7 deletions(-)

Index: linux-sgi-ip27-ua/arch/mips/mm/c-r4k.c
===================================================================
--- linux-sgi-ip27-ua.orig/arch/mips/mm/c-r4k.c	2005-02-10 14:25:54.935572026 +0100
+++ linux-sgi-ip27-ua/arch/mips/mm/c-r4k.c	2005-02-10 14:28:02.329527242 +0100
@@ -376,6 +376,13 @@
 	pmd_t *pmdp;
 	pte_t *ptep;
 
+	/*
+	 * If ownes no valid ASID yet, cannot possibly have gotten
+	 * this page into the cache.
+	 */
+	if (cpu_context(smp_processor_id(), vma->vm_mm) == 0)
+		return;
+
 	page &= PAGE_MASK;
 	pgdp = pgd_offset(mm, page);
 	pudp = pud_offset(pgdp, page);
@@ -433,13 +440,6 @@
 {
 	struct flush_cache_page_args args;
 
-	/*
-	 * If ownes no valid ASID yet, cannot possibly have gotten
-	 * this page into the cache.
-	 */
-	if (cpu_context(smp_processor_id(), vma->vm_mm) == 0)
-		return;
-
 	args.vma = vma;
 	args.page = page;

---

From: Rojhalat Ibrahim <[email-masked]>
Date: Thu, 10 Feb 2005 15:54:36 +0100

> cacheflush to the right place.  That's a bug which is harmless on UP but

I did a CVS update, which apparently includes this
patch, and I get this:

   LD      .tmp_vmlinux1
arch/mips/mm/built-in.o(.init.text+0x98): In function `fixrange_init':
: undefined reference to `__pud_offset'
make: *** [.tmp_vmlinux1] Error 1

What's __pud_offset?

Thanks
Rojhalat Ibrahim

---

From: Rojhalat Ibrahim <[email-masked]>
Date: Thu, 10 Feb 2005 16:10:31 +0100

> cacheflush to the right place.  That's a bug which is harmless on UP but

Also tried it on this morning's source (without the
most recent patches). Doesn't make a difference.
Still works with 512 MB and crashes with 1 GB.
The problem is not SMP related anyway. It appears
the same with an UP kernel.

Thanks
Rojhalat Ibrahim