Subject: Re: What about...

From: Alex deVries <[email-masked]>
Date: Fri, 17 Jul 1998 01:24:43 -0400 (EDT)

On Thu, 16 Jul 1998, Igor Loncarevic wrote:
> What about Linux for Origin2000 SGI?
> Is there any possibility to run Linux on Origin?

I would say there's almost no chance of running Linux on an Origin 2000
anytime this century.

- Alex

---

From: "Greg Chesson" <[email-masked]>
Date: Thu, 16 Jul 1998 22:30:48 -0700

Alex is right.
Linux on Origin 2000 would be a huge project - not just a port.

We have much better chances getting Linux up on SGIs yet-to-be-revealed
Intel-based platforms.

g

-- 
Greg Chesson

---

From: "William J. Earl" <[email-masked]>
Date: Fri, 17 Jul 1998 07:11:42 -0700

> Linux on Origin 2000 would be a huge project - not just a port.
...

     Note that the reason it would be a big project is not so much the
hardware itself, but rather the scale of the system.  IRIX needed a lot
of infrastructure work to be useful on very large system with a very large
number of I/O buses and devices.  That is, a toy port would not be all
that giant a project, but it would not be particularly useful.  If there
were a good linux on some other very large ccNUMA machine, then an Origin
port would be much simpler.  By "good", I mean a linux which scale well
for large (greater than 32) processor and I/O count (many I/O buses
and thousands of disk).  It expect such a linux will happen eventually,
but not yet.

---

From: [email-masked] (Alan Cox)
Date: Fri, 17 Jul 1998 16:07:04 +0100 (BST)

> and thousands of disk).  It expect such a linux will happen eventually,

The obvious starting point would probably be the older (386/486) sequent
boxes. I almost got somewhere with this but one bit of sequent wasnt 
willing to be helpful and counted its bus arbitrators as staying NDA.

That was 2 years ago so I ought to chase them again I guess

---

From: [email-masked]
Date: Fri, 17 Jul 1998 19:29:54 +0200

> and thousands of disk).  It expect such a linux will happen eventually,

Actually some people begin to be interested in Linux for huge installations.
For example a bank recently offer us some machine time on a maximum
configuration (64 CPU, 64GB RAM) E10000 for porting Linux.  This didn't
happen (yet?) for other reasons, but it shows that there is interest.

For the time being most large Linux installations are based on the Beowulf
clustering software - and I think place 318 on the list on the top 500 list
of supercomputer installations is quite remarkable for something which's
OS software is being shipped for USD 29.95 by RedHat.

Anyway, as far as our MIPS port goes we first need to implement SMP at
all before we should think about scaling.  More about Linux in general -
most people are currentl using Linux/SMP on dual processors, few on
quad CPU systems and even fewer on larger systems.  This is about where
the market of Linux and it's limits are.  But the tradition of Linux
shows that technology for which a demand exists will be developed sooner
or later.  Where Linux technology exists, usually a market will develop.
So far I see a demand for older SGI SMP systems, Origin 200 systems (quite
some inquiries) but not yet for Origin 2000 systems.

  Ralf

---

From: [email-masked]
Date: Fri, 17 Jul 1998 19:43:20 +0200

> That was 2 years ago so I ought to chase them again I guess

Do you have hardware?  I know an installation which might want to get rid
of their 486 based 32 processor Sequent system if they didn't already.

  Ralf

---

From: "Greg Chesson" <[email-masked]>
Date: Fri, 17 Jul 1998 10:47:02 -0700

Bill is, of course, quite correct.
In addition to the observations about the scale of the system,
realize also that a ccNUMA machine has a memory system for each
cpu node in the system.  The physical base addresses of these blocks
of memory are aligned on multi-gigabyte boundaries.  The high-order
bits of the address designate the cpu node, the rest address the physical
memory, etc etc.  What this means is that physical memory space has
many "holes"...  The idea of a simple buddy-system allocator as is
ingrained in the Linux kernel falls apart completely in the face of
this kind of architecture.   I suppose you could run a copy of Linux
on every node, but I consider that an excuse rather than a solution.

g

-- 
Greg Chesson

---

From: [email-masked] (Alan Cox)
Date: Fri, 17 Jul 1998 18:53:28 +0100 (BST)

> Do you have hardware?  I know an installation which might want to get rid

I dont. I had an offer at the time, and I did some digging on the older boxes
also recently on the new ones - did you know Sequent Numa-Q appears to use non
standard microcode ?

> of their 486 based 32 processor Sequent system if they didn't already.

I can just imagine the shipping cost ;)

---

From: "Greg Chesson" <[email-masked]>
Date: Fri, 17 Jul 1998 11:01:42 -0700

Ralf,

Beowolf machines don't have adequate IO for applications other than
things that resemble Linpack.  The teraflop machine at Sandia would like
to generate output every 5 or 10 minutes, but it takes an hour to get
the computation result out of the machine. Beowolf machines tend to not
have adequate bisection bandwidth more many kinds of codes.  Anything that
requires a cornerturn (transpose) is a good example.  Also, Beowolf machines
tend to run a single code for many hours: they don't multiplex well,
or sometimes at all.  But multiplexing and adapting to a dynamic load
is expected of many large scale machines and that requires some OS support
not found in the single-cpu or small-cpu-count OS or existing middleware.

 We cannot afford to build
machines that are good only for Linpack.... it is better
to build a Beowolf cluster if that is the best fit for the job.
It is rather ignorant to think that because something like Beowolf can do some
some types of applications well, that some software work can make it do
all jobs well.  That's like saying a Volkswagon could go 300 kph
if you drive carefully.

Linux has proven to be worthwhile as a node controller in an MPP architecture -
that's what a Beowolf is.  But that does not make it ready for SMP nodes
that scale to large numbers.  It seems wasteful to program a large scale
ccNUMA machine the same way as a Beowolf cluster: you'd be throwing away
most of the capabilities of the hardware.  That's why I don't
think it is interesting or particularly useful... unless a massive amount
of work went into rewriting the io and memory management subsystems,
not to mention scheduling, administration, etc.

g

-- 
Greg Chesson

---

From: [email-masked] (Alan Cox)
Date: Fri, 17 Jul 1998 19:14:04 +0100 (BST)

> on every node, but I consider that an excuse rather than a solution.

Actually the Linux buddy stuff is quite happy with holes. Its still
completely inappropriate. From the above I deduce we'd have to do
mips64 before we even considerd it anyway

---

From: "William J. Earl" <[email-masked]>
Date: Fri, 17 Jul 1998 11:19:54 -0700

Greg Chesson writes:
...
...

     I expect that much of the work will gradually happen in linux, once
the remaining small-CPU-count issues are resolved.  Right now, there
is no shortage of interesting problems to attack.  :-)

     One possible way to approach the large-CPU-count space with linux
is to indeed run multiple linux kernels, one per node in a ccNUMA
machine, and add a distributed OS layer at a fairly high level.  If it
is not underway already somewhere, I would expect someone to take up
the project as a graduate school project.  Some systems of this sort
have been built or attempted, with varying degrees of success.  Given
the linux bias toward small and simple, a linux-based distributed OS
might actually work.

     One important ingredient in such a system, which would be
valuable immediately for clusters, would be a efficient distributed
volume manager and file system.  

     In any case, trying to port linux straight to a large ccNUMA 
(or even a large SMP) system would be a lot of effort for limited
return at present.

---

From: "William J. Earl" <[email-masked]>
Date: Fri, 17 Jul 1998 11:21:18 -0700

> mips64 before we even considerd it anyway

     Yes, the address space is very large, even in a single rack.

---

From: Jeffrey Watts <[email-masked]>
Date: Fri, 17 Jul 1998 17:15:46 -0500 (CDT)

> > and thousands of disk).  It expect such a linux will happen eventually,

Boy would it be fun to work on that port.  Origin2000s are seriously cool. 
I am getting 9 new Origin2000 systems in about 6 weeks.  I wonder if
Sprint would mind if I used one for Linux porting...  :-) 

J.

o-----------------------------------o
| Jeffrey Watts                     |
| [email-masked]           o-------------------------------------o
| Systems Analyst               | "Proprietary system advocates       |
| Sprint - Systems Management   |  aren't evil or stupid.  They are   |
o-------------------------------|  the victims.  They have a disease  |
                                |  and they need help."               |
                                |  -- Donald B. Marti Jr.             |
                                o-------------------------------------o

---

From: [email-masked]
Date: Sat, 18 Jul 1998 03:37:59 +0200

> mips64 before we even considerd it anyway

At least in Vger CVS we alredy have code to efficiently deal with
non-dense memory architectures with buddy.

I think the memory allocator design as presented by Rik van Riel looks as
if it is a good base to deal much more efficiently with such an
memory architecture.  And then there is Sct with his big iron experience
working on something better than the buddy system.

I completly agree that we first have to go to MIPS64 before we really
can attack big iron problems.  The current Linux/MIPS kernel design limits
the memory to the size of KSEG0 which is 512mb.  Putting Linux into a
64bit universe we'd extend that design limit to the size of XKSEG0, or
1TB for current silicon.

  Ralf

---

From: [email-masked]
Date: Sat, 18 Jul 1998 03:57:15 +0200

> is no shortage of interesting problems to attack.  :-)

Actually I'd hate it if Linux'd be ``finished'' ...

> the linux bias toward small and simple, a linux-based distributed OS

I don't know the exact technical details but something similar has already
been done back in '95 (?) when some guys in Australia ported Linux to
Fujitsu's AP1000.  That's of course a different architecture and a
multikernel approach makes much more sense there.

> volume manager and file system.  

Hans Reiser is currently working on a new filesystem with alot of fresh
ideas.  In some aspects what he is aiming at is similar to XFS, in some
aspects not.  He basically started on a white sheet of paper with his design,
so his team's code isn't contaminated by old ideas.  His work looks pretty
promising.  Among his plans is also the implementation of a distributed
filesystem.  Current benchmarks are looking pretty good, in fact in
some cases extremly good.  The URL to checkout for interested people is
http://idiom.com/~beverly/reiserfs.html.

> (or even a large SMP) system would be a lot of effort for limited

Explecitly not talking about the MIPS port - I think it makes sense in
working on porting to machines beyond what we currently scale to.  But
I agree about the ``large'' thing.

  Ralf

---

From: "Greg Chesson" <[email-masked]>
Date: Fri, 17 Jul 1998 18:58:07 -0700

I think everyone is suggesting the same general path, and I'm gratified
to know that the buddy system code anticipates non-contiguous physical
memory.

I think you'd want to extend the design limit (XKSEG0) beyond 1 TB
to handle the next rev of silicon.  I'd suggest 64 TB (48 bits) as
the appropriate goal.

g

-- 
Greg Chesson

---

From: "Greg Chesson" <[email-masked]>
Date: Fri, 17 Jul 1998 19:00:30 -0700

> Actually I'd hate it if Linux'd be ``finished'' ...

Ancient Chinese proverb:  "House finish; man die."

g

-- 
Greg Chesson

---

From: [email-masked]
Date: Sat, 18 Jul 1998 04:44:03 +0200

> to handle the next rev of silicon.  I'd suggest 64 TB (48 bits) as

As far as the maximum amount of RAM goes, no problem.  The only assumption
which I make is that the entire available memory is visible in XKSEG0,
whatever it's size is.  That means the actual design limit is the maximum
possible size of XKSEG0 for the MIPS 64bit architecture which is 2^62 bytes.

I assume this also means the size limit of the XKUSEG has been extended
to 48 bits.  There we actually a somewhat more limiting design limit since
we only have three level page tables for 64 bit architectures which limits
us to a maximum mappable size of

  PAGE_SIZE * (PAGE_SIZE / sizeof(pte_t)) ^ 3

which for PAGE_SIZE = 4 kbytes and sizeof(pte_t) = 8 bytes is 512gb.

(And actually things are somewhat more complex.)

  Ralf

---

From: [email-masked]
Date: Sat, 18 Jul 1998 12:47:54 +0200

> possible size of XKSEG0 for the MIPS 64bit architecture which is 2^62 bytes.

Ooops, the real limit is of course how much memory can be presented in XKPHYS
which is a bit less, just 2^58 bytes or 4 exabytes.  XKSEG or XKSSEG would
be used for kernel virtual memory which usually isn't used very much under
Linux - it's virtual, but not swappable.

  Ralf