Subject: MIPS 64?

From: "Matthew Dharm" <[email-masked]>
Date: Wed, 15 May 2002 14:34:47 -0700

So... I'm looking at porting Linux to a system with 1.5GiB of RAM.
That kinda blows the 32-bit MIPS port option right out of the water...

What does it take to do a 64-bit port?  The first problem I see is the
boot loader -- do I have to be in 64-bit mode when the kernel starts,
or can I start in 32-bit mode and then transfer to 64-bit mode?

I looked in the arch/mips64/ directory, but I don't see much for
specific boards there, but there are references to the Malta
boards....

Matt

--
Matthew D. Dharm                            Senior Software Designer
Momentum Computer Inc.                      1815 Aston Ave.  Suite 107
(760) 431-8663 X-115                        Carlsbad, CA 92008-7310
Momentum Works For You                      www.momenco.com

---

From: Daniel Jacobowitz <[email-masked]>
Date: Wed, 15 May 2002 17:48:18 -0400

> That kinda blows the 32-bit MIPS port option right out of the water...

Not unless you count bits differently than I do... 32-bit is 4 GiB.  Is
there any reason MIPS has special problems in this area?

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

---

From: "Kip Walker" <[email-masked]>
Date: Wed, 15 May 2002 14:55:39 -0700

Since traditionally the mips kernel could only manage RAM reachable by
Kseg you were limited to 512MB, right?  But now the High Memory stuff is
stable enough that you can reach any RAM that's physically addressable
in 32-bits.  And after that, there's the 64bit-physical-address
extension which allows you to reach physical pages that need > 32-bits
to address.

Kip

---

From: Jun Sun <[email-masked]>
Date: Wed, 15 May 2002 14:59:34 -0700

> there any reason MIPS has special problems in this area?

MIPS has lower 2GB fixed for user space.  Then you have kseg0, .5GB for cached 
physical address 0-0.5GB, and kseg1, 0.5GB uncached mapping of the same area. 
  You can map another 1GB of RAM into kseg2/3, but you will then have no space 
left for IO.

So you really can't do 1.5GB on 32 bit kernel.

It is interesting that PPC allows one to adjust user space size and kernel 
space size.  So on PPC you can get up to 2.5GB system RAM with 1GB user space.

Back to 64bit port, it seems to me much of the 32bit work we have done in the 
past a year or so needs to be moved over.  Or better yet, if we can clean up 
integer/long issues, we might be able to use 32bit kernel code straight for 
64bit kernel.

Jun

---

From: "Matthew Dharm" <[email-masked]>
Date: Wed, 15 May 2002 15:08:05 -0700

I don't suppose anyone has a primer or white paper on the High Memory
stuff?  i.e. Applications, requirements, or a quick HOWTO?

Matt

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Wednesday, May 15, 2002 2:56 PM
> To: Matthew Dharm
> Cc: Linux-MIPS
> Subject: Re: MIPS 64?
>
>
>

---

From: "Kip Walker" <[email-masked]>
Date: Wed, 15 May 2002 15:15:07 -0700

> stuff?  i.e. Applications, requirements, or a quick HOWTO?

Well, the CONFIG option is at the bottom of the Machine Selection menu. 
With a fairly recent 2.4 or 2.5 kernel, it should build at work. 
Basically, if your firmware/boot code conveys info about regions above
physical address 0x1fffffff, the kernel will allocate "struct page"
entries for it, and add them to the pool of allocatable memory.  The
kernel gets at them by mapping them into Kseg2/Kseg3 temporarily.

turn it on, see what happens!  I haven't looked for a primer.

Kip

---

From: "Matthew Dharm" <[email-masked]>
Date: Wed, 15 May 2002 15:16:29 -0700

I personally think it's long overdue that we really invest some
time/effort in the 64-bit version.  I mean, anything past 0.5 GiB is
really a hack on 32-bit....

I, for one, am willing to lend a pair of hands to this... but I need
to know what needs doing, first.

Could someone give me an overview of how you're supposed to do a
handoff between a 32-bit loader and a 64-bit app?  I'm guessing there
has to be a way to do it, but what I do know about the 64-bit stuff
doesn't show me how this is accomplished (I have visions of UX bits
floating in my head...)

Matt

> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Jun Sun
> Sent: Wednesday, May 15, 2002 3:00 PM
> To: Daniel Jacobowitz
> Cc: Matthew Dharm; Linux-MIPS
> Subject: Re: MIPS 64?
>
>
> Daniel Jacobowitz wrote:
>

---

From: "Matthew Dharm" <[email-masked]>
Date: Wed, 15 May 2002 15:26:08 -0700

Right....

So, how should my boot code convey that info?  With more
add_memory_region() calls?  Is that really all I need?

Matt

> [mailto:[email-masked]]On Behalf Of Kip Walker
> Sent: Wednesday, May 15, 2002 3:15 PM
> To: Matthew Dharm
> Cc: Linux-MIPS
> Subject: Re: MIPS 64?
>
>
> Matthew Dharm wrote:
> >

---

From: "Kip Walker" <[email-masked]>
Date: Wed, 15 May 2002 15:29:01 -0700

Yes... in my environment, I have firmware calls to find out what
physical memory exists, then I call add_memory_region as necessary.  See
arch/mips/sibyte/swarm/setup.c if you're interested.  Nothing too
interesting about it.

Kip

---

From: Dan Malek <[email-masked]>
Date: Wed, 15 May 2002 23:28:18 -0400

Jun Sun wrote:

> It is interesting that PPC allows one to adjust user space size and 
> kernel space size.  So on PPC you can get up to 2.5GB system RAM with 
> 1GB user space.

Don't be confusing virtual/physical addressing.  The highmem stuff allows
access to larger physical address space by providing "windows" through
the virtual space.  The memory space configuration on PPC was done before
highmem was working properly, but it has remained as an embedded configuration
option.  When using highmem, you obviously can't map all of the physical
memory at once, so your applications and drivers have to coordinate what
is mapped at a single instance.  The PPC configuration option gives us a
little flexibility to allow a little more kernel mapping for drivers when
it is necessary (or to work around weird I/O mapping problems).

Thanks.

	-- Dan

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Thu, 16 May 2002 09:15:40 +0200

> So you really can't do 1.5GB on 32 bit kernel.

Is this to say that Linux cannot function unless all physical memory
on the system is mapped at all times into kernel space?  I would
have thought that (a) all that really needs to be mapped is all
memory used by the kernel itself, plus that of the currently active
process (which is mapped in the 2GB kuseg), and that (b) one 
could anyway manage kseg2 or 3 dynamically to provide a window 
into a larger physical memory, and that this window could be
used for any functions that need to do arbitrary phys-to-phys
copies.

I'm not sure what people's definition of a "64-bit kernel"
is around here, but to me, that's a kernel which supports
64-bit virtual addressing and 64-bit registers.  It strikes
me as being rather foolish to impose the overhead of all
that on people whose only real requirement is 2G of RAM
on a 32-bit CPU.  Particularly when many of the new
MIPS parts that could plausibly be connected to 2GB
RAM arrays, such as the Alchemy/AMD and MIPS 4K
parts, don't support 64-bit operation.  So running away
from the problem isn't really an option.

            Regards,

            Kevin K.

---

From: Jun Sun <[email-masked]>
Date: Thu, 16 May 2002 10:13:30 -0700

> used for any functions that need to do arbitrary phys-to-phys

You are right - my above statement is a grossly simplified way of saying "You 
can't really have 1.5GB system RAM accessible to kernel at the same time on a 
32bit MIPS kernel".

There is nothing preventing you from having more physical RAM and use 
dynamically by using HIGHMEM support.

BTW, I have been under the impression that demand for larger system RAM mainly 
comes from large router/switches to store routing table.  Does anybody know on 
such systems whether the routing code runs in kernel or user space and whether 
  it requires all the memory space accessible at the same time or can live 
with dynamically managed memory space?

Jun

---

From: Ralf Baechle <[email-masked]>
Date: Sun, 19 May 2002 12:23:37 -0700

> That kinda blows the 32-bit MIPS port option right out of the water...

Not really, there is still highmem though that certainly a sucky solution
as compared to a 64-bit kernel.

> or can I start in 32-bit mode and then transfer to 64-bit mode?

Same loader as before - the build procedure will result in a 32-bit kernel
binary which is loaded to the same old KSEG0 addresses.

> specific boards there, but there are references to the Malta

Exactly.  The aim is to support both kernels without replicating the board
support code.

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Sun, 19 May 2002 12:24:19 -0700

> there any reason MIPS has special problems in this area?

The basic assumption is that we can address all memory through KSEG0.

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Sun, 19 May 2002 12:25:01 -0700

> stuff?  i.e. Applications, requirements, or a quick HOWTO?

Same old technology as on Intel, so also the same old limitions ...

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Sun, 19 May 2002 12:29:11 -0700

> add_memory_region() calls?  Is that really all I need?

Yes.  At this time you just have to make sure you don't add_memory_region()
areas which aren't supported by your particular kernel configuration.  There
is also a second limit which is memory above physical address 4GB; if you
need that you have to enable CONFIG_64BIT_PHYS_ADDR as well.

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Sun, 19 May 2002 12:30:59 -0700

> integer/long issues, we might be able to use 32bit kernel code straight for 

Int vs. long was a very small issue as I discovered during porting for the
first 64-bit machines, the IP22 and IP27.

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Sun, 19 May 2002 12:32:19 -0700

> doesn't show me how this is accomplished (I have visions of UX bits

If you're placing the kernel in KSEG0 like all other current targets are
doing then your 32-bit pointers are valid 64-bit pointers, no magic,
no thinking necessary.

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Sun, 19 May 2002 12:41:03 -0700

> used for any functions that need to do arbitrary phys-to-phys

Highmem is your dynamic 4MB window into all memory that's not low memory.
KSEG2/3 are currently mostly reserved for vmalloc / ioremap and these
4MB of highmem mapping space - a usage that's certainly wasteful.

As opposed to that all the permanently mapped kernel memory in Linux is
called lowmem.

> from the problem isn't really an option.

Highmem doesn't come without a price either - in particular processes doing
heavy I/O will possibly as much as 40% performance penalty (numbers from
Intel systems) and for sanity's sake - 64-bit is keeping people from going
nuts :)

A solution that will use most of KSEG2/3 for mapping more lowmemory is in
the works but it turned out to be drastically more complex that originally
thought so will still take a bit.

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Sun, 19 May 2002 12:44:35 -0700

On Thu, May 16, 2002 at 10:13:30AM -0700, Jun Sun wrote:

> BTW, I have been under the impression that demand for larger system RAM
> mainly comes from large router/switches to store routing table.  Does
> anybody know on such systems whether the routing code runs in kernel or
> user space and whether it requires all the memory space accessible at
> the same time or can live with dynamically managed memory space?

Kernel routing is more for the lower end systems; the highend systems use
hardware assisted routing / switching where software only handles the
boundary cases; such software might either live in user or in kernel space.

  Ralf

---

From: Matthew Dharm <[email-masked]>
Date: Sun, 19 May 2002 23:05:52 -0700

> binary which is loaded to the same old KSEG0 addresses.

Call me a bit slow, but...

Are you saying that my 32-bit loader (which is designed to load a 32-bit
ELF file) will do exactly that... but this 32-bit ELF file has the magic in
it to switch to 64-bit internally?

Nice... Very nice.  I'm used to slick Open Source solutions, but I have to
admit that this is a pretty elegant one that solves a great many
problems...

Matt

-- 
Matthew Dharm                              Work: [email-masked]
Senior Software Designer, Momentum Computer

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Mon, 20 May 2002 12:06:45 +0200 (MET DST)

> first 64-bit machines, the IP22 and IP27.

 Well, the surprise is going to happen in drivers, I'm afraid...

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email-masked], PGP key available        +

---

From: Greg Lindahl <[email-masked]>
Date: Mon, 20 May 2002 08:57:44 -0700

>  Well, the surprise is going to happen in drivers, I'm afraid...

Linux drivers as a whole are 64-bit clean; alpha's been around for a
long time. MIPS-only devices might be dirtier.

greg

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Mon, 20 May 2002 18:05:57 +0200 (MET DST)

> long time. MIPS-only devices might be dirtier.

 Well, that is wishful thinking.  Obviously if a driver is actually used
on a 64-bit system, one can be moderately confident the driver is clean,
although there might still be corner cases.  Otherwise expect all kinds of
weirdness.  See e.g. the ISA stuff.

---

From: Ralf Baechle <[email-masked]>
Date: Mon, 20 May 2002 12:05:52 -0700

>  Well, the surprise is going to happen in drivers, I'm afraid...

That depends on how much people were thinking ahead when writing drivers.

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Mon, 20 May 2002 13:30:56 -0700

> admit that this is a pretty elegant one that solves a great many

It's a fairly natural solution to the problem we had.  The scenario we had
to deal with was

 - a machine (SGI Origin 200/2000 aka IP27) where configurations are
   commonly way too large for a 32-bit kernel.
 - 64-bit MIPS/ELF support that was close to entierly unusable
 - fixing binutils fully would have taken a serious amount of time which
   we didn't have.

As it turned out gas is perfectly able to assemble the assembler code that
is generated for the N64 ABI by gcc into 32-bit ELF and linking that.
That code model still uses 64-bit pointers just all intra-kernel refernces
that were generated from la or dla macros get expanded into just the
two instruction sequence for 32-bit code, not the upto 7 instruction used
for 64-bit code.  Result: much less tools work necessary yet better code.

  Ralf

---

From: Florian Lohoff <[email-masked]>
Date: Tue, 21 May 2002 18:47:30 +0200

> long time. MIPS-only devices might be dirtier.

Not really true - I just stumbled over the cyclades multiport driver
which says to work on alpha for a long time - But it doesnt on
Sparc64 due to the porters misunderstanding on typedefs for the 
driver internals. (Alpha and i386 are happy with char irqs e.g.).

Flo
-- 
Florian Lohoff                  [email-masked]             +49-5201-669912
                        Heisenberg may have been here.

---

From: Greg Lindahl <[email-masked]>
Date: Tue, 21 May 2002 10:00:40 -0700

> driver internals. (Alpha and i386 are happy with char irqs e.g.).

I must say I find this entire discussion bewildering: one example does
not mean that Linux drivers are not "as a whole" 64-bit clean.

Another poster thought that things like ISA and Turbochannel devices
were a counter-example. Well, no.

greg