Subject: Re: Decstation 5000/150 2.3.21 Boot successs

From: "Kevin D. Kissell" <[email-masked]>
Date: Thu, 6 Jan 2000 10:12:21 +0100

The "setting flush to zero/Unimlemented exception"
problem is almost certainly due to the fact that the
DECstation binaries were compiled for R3000-based
platforms with R3K-style FPUs which have 32 single-precision
FP regs that can also be treated pairwise as 16
double-precision registers.  I didn't know that the 5000 line
had R4000 CPUs in them, but on the basis of your reported
CPU revision number, that's what you've got.

The default SGI/MIPS Linux kernel startup sets the
"FR" bit in the CP0.Status register, which enables
R4000-style FPU registers, which is to say a full
compliment of 32 double-precision registers.  This
has the side-effect of making the kernel incompatible
with the distributed mipsel binaries and the distributed
DECstation root file system, since those binaries which
do double-precision floating point load their initial values
from memory as two singles.  That works on an R3000
or an R4000-with-FR=0, but not on an R4000-with-FR=1.
If FR=1, the data in the registers is garbage, the FP
ops trap on unimplemented exceptions, which first
retry the instruction with flush-to-zero mode in a
desperate hope that that will solve the problem,
then call the fragmentatary SW emulator in the
kernel, which only handles a few cases, then
skip the instruction, printing the console message
that it "should" have sent a SIGFPE.   Hence the
output.  And hence the bizzare behaviour of awk,
ps, and a number of other basic programs that do
minimal FP as the system comes up (procps?).

It took me a while to figure this out when I ran into
it doing a little-endian port of Linux 2.2.12 to an R5260,
as you might imagine.   Unless things have been drastically
scrambled for 2.3.x,  the code controlling this is in
arch/mips/kernel/head.S.   What we did at MIPS was
to add a config option to determine which FPU model
is desired.   The truly elegant thing would be to examine
the a.out file and manage Status.FR dynamically, but
a quick inspection turned up no clean mechanism in
the Linux kernel for passing such architecture-dependent
information from the generic a.out file parsing to the
thread creation machinery, so that will be a later
exercise.   Thoward the end of the kernel entry point
(kernel_entry), our revised code looks a bit like this:

        /* Disable coprocessors */
        mfc0    t0, CP0_STATUS
#ifdef CONFIG_MIPS_32FR
        li      t1, ~(ST0_CU1|ST0_CU2|ST0_CU3|ST0_KX|ST0_SX)
#else
        li      t1, ~(ST0_CU1|ST0_CU2|ST0_CU3|ST0_KX|ST0_SX|ST0_FR)
#endif
        and     t0, t1
        or      t0, ST0_CU0
        mtc0    t0, CP0_STATUS

1:      jal     start_kernel
         nop

The "#ifdef/else/endif" sequence is our addition.  For a
quick hack, just add ST0_FR to the set of bits being
inverted in the existing code.

            Hope this helps,

            Kevin K.
__

Kevin D. Kissell
MIPS Technologies European Architecture Lab
[email-masked]
Tel. +33.4.78.38.70.67
FAX. +33.4.78.38.70.68

-----Original Message-----
From: Florian Lohoff <[email-masked]>
To: [email-masked] <[email-masked]>
Date: Wednesday, January 05, 2000 11:17 PM
Subject: Decstation 5000/150 2.3.21 Boot successs

>
>Hi,
>this is is a short output of the 2.3.21 booting on the Decstation
>5000/150 ... This is the current oss.sgi.com CVS.
>
>--------------------------------------------------------------
>This DECstation is a DS5000/1xx
>Loading R4000 MMU routines.
>CPU revision is: 00000430
>Primary instruction cache 8kb, linesize 16 bytes)
>Primary data cache 8kb, linesize 16 bytes)
>Secondary cache sized at 1024K linesize 32
>Linux version 2.3.21 (root@repeat) (gcc version egcs-2.90.27 980315 (egcs-1.0.2
release)) #1 Tue Jan 4 18:39:20 GMT 2000
>Calibrating delay loop... 49.81 BogoMIPS
>Memory: 62652k/65532k available (1068k kernel code, 1524k data)
>Buffer-cache hash table entries: 4096 (order: 2, 16384 bytes)
>Page-cache hash table entries: 16384 (order: 4, 65536 bytes)
>Checking for 'wait' instruction...  unavailable.
>POSIX conformance testing by UNIFIX
>TURBOchannel rev. 1 at 12.5 MHz (no parity)
>    slot 0: DEC      PMAZ-AA  V5.3d
>    slot 1: DEC      PMAZ-AA  V5.3b
>    slot 2: DEC      PMAF-FA  V1.1
>Linux NET4.0 for Linux 2.3
>Based upon Swansea University Computer Society NET3.039
>NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
>NET4: Linux TCP/IP 1.0 for NET4.0
>IP Protocols: ICMP, UDP, TCP
>TCP: Hash tables configured (established 4096 bind 8192)
>Starting kswapd v1.6
>DECstation Z8530 serial driver version 0.03
>tty00 at 0xbc100001 (irq = 4) is a Z85C30 SCC
>tty01 at 0xbc100009 (irq = 4) is a Z85C30 SCC
>tty02 at 0xbc180001 (irq = 4) is a Z85C30 SCC
>tty03 at 0xbc180009 (irq = 4) is a Z85C30 SCC
>SCSI ID 7 Clk 25MHz CCF=5 TOut 167 NCR53C9x(esp236)
>SCSI ID 7 Clk 12MHz CCF=3 TOut 139 NCR53C9x(esp236)
>SCSI ID 7 Clk 12MHz CCF=3 TOut 139 NCR53C9x(esp236)
>ESP: Total of 3 ESP hosts found, 3 actually in use.
>scsi0 : ESP236 (NCR53C9x)
>scsi1 : ESP236 (NCR53C9x)
>scsi2 : ESP236 (NCR53C9x)
>scsi : 3 hosts.
>  Vendor: Quantum   Model: XP34300           Rev: L912
>  Type:   Direct-Access                      ANSI SCSI revision: 02
>Detected scsi disk sda at scsi0, channel 0, id 0, lun 0
>  Vendor: SEAGATE   Model: ST15150N          Rev: 8902
>  Type:   Direct-Access                      ANSI SCSI revision: 02
>Detected scsi disk sdb at scsi0, channel 0, id 2, lun 0
>scsi : detected 2 SCSI disks total.
>esp0: target 0 [period 200ns offset 15 5.00MHz synchronous SCSI]
>SCSI device sda: hdwr sector= 512 bytes. Sectors= 8399520 [4101 MB] [4.1 GB]
>esp0: target 2 [period 200ns offset 15 5.00MHz synchronous SCSI]
>SCSI device sdb: hdwr sector= 512 bytes. Sectors= 8388315 [4095 MB] [4.1 GB]
>declance.c: v0.008 by Linux Mips DECstation task force
>eth0: IOASIC onboard LANCE, addr = 08:00:2b:28:f0:a3, irq = 3
>Sending BOOTP requests.... OK
>IP-Config: Got BOOTP answer from 193.189.250.46, my address is 193.189.250.44
>-----------------------------------------------------
>
>I see some more ugly things:
>
>Setting flush to zero for dpkg-source.
>Unimplemented exception for insn 4620a0a4 at 0x0046f084 in dpkg-source.
>Should send SIGFPE to dpkg-source
>Setting flush to zero for dpkg-source.
>Unimplemented exception for insn 4620a0a4 at 0x0046f084 in dpkg-source.
>Should send SIGFPE to dpkg-source
>Setting flush to zero for dpkg-source.
>Unimplemented exception for insn 4620a0a4 at 0x0046f084 in dpkg-source.
>Should send SIGFPE to dpkg-source
>
>And this ...
>
>Bug in get_wchan
>
>This seems to be a result of buggy "procps" ...
>
>Flo
>--
>Florian Lohoff [email-masked]       +49-5241-470566
>  ...  The failure can be random; however, when it does occur, it is
>  catastrophic and is repeatable  ...             Cisco Field Notice
>

---

From: Ralf Baechle <[email-masked]>
Date: Thu, 6 Jan 2000 15:26:12 +0100

> (kernel_entry), our revised code looks a bit like this:

Linux _never_ changes the FR flag.  In fact it's living in the assumption
that once the firmware hands over the control to the kernel the FR flag
has been configured apropriately.  For a 32-bit kernel the binaries available
out there more or less conform to the MIPS ABI which uses the 16/32
register model, that is the kernel expects the FR flag to be cleared.

In firm assumption that due all the practical problems involved with
a non-standard execution model (i.e. 32-bit, o32-style ELF, 32/32
register model and 32-bit gprs) I decieded that in practice nobody will
use this and dumped all the support for it from later 2.3 kernels.  That
is the scheduler will no longer try to handle context switching for
the 32/32 fpr model correctly etc.

If that's desired, how about providing a syscall which allows to manipulate
this and possibly other bits?

Btw...  Thanks for posting.  You pointed my nose at the fact that this bug
actually exists for the 64-bit kernel - and there is actually a real world
bug because we can mix 32-bit and 64-bit binaries.

  Ralf

---

From: "William J. Earl" <[email-masked]>
Date: Thu, 6 Jan 2000 10:57:34 -0800 (PST)

Kevin D. Kissell writes:
...
 > or an R4000-with-FR=0, but not on an R4000-with-FR=1.
...

     Note that the SVR4 MIPS ABI assumes FR=0 (R3000-compatible), as
do SGI IRIX "-32" ("O32") binaries (and, I believe, default gcc
binaries).  SGI IRIX "-n32" and "-n64" binaries assumes FR=1 (R4000-compatible),
and also have a somewhat different register calling convention (which
affects where arguments to system calls reside).

---

From: "William J. Earl" <[email-masked]>
Date: Thu, 6 Jan 2000 11:10:43 -0800 (PST)

Ralf Baechle writes:
...
 > bug because we can mix 32-bit and 64-bit binaries.

      It also applies to -n32 (FR=1 32-bit binaries).  Basically, a complete
kernel implementation has to treat the two cases which different from its own
as "compatibility mode" targets.  That is, if the kernel is "-32" ("O32"),
it has to turn on FR=1 for those processes currently executing a "-n32" binary
and off for those executing a "-32" binary.  That is, exec has to change
the desired FR value, and context switching to and from user mode has to
save and restore it.

     One other issue is that UX should always be set, to allow use of
MIPS3 instructions, and that XX (bit 31) should be set on R5000 and
R10000 processors, to enable MIPS4 instructions.  This in turn means
that, to avoid various illegal address exceptions, the VM system
should not allow a 32-bit user program to map anything into the top 32
KB of the user address space.

     The problem has to do with some compilers using integer
arithmetic to compute a base for some variables in the current stack
frame, and then using negative displacements to address the variables,
for cases where the stack frame exceeds 32 KB, but is located near the
top of memory.  The 32-bit unsigned integer add to, say, 0x7fffff00
(64-bit address 0x000000007fffff00) produces a signed 32-bit value
such as 0x80000f00, which is the 64-bit value 0xffffffff80000f00,
since all 32-bit values, signed or unsigned, are stored as 32-bit
signed values sign-extended to 64 bits.  When you do a load with a
negative offset of, say, -0x1000, you get an address
0xffffffff7fffff00, not 0x000000007fffff00.  With UX=0, this would be
fine, but, with UX=1 (to enable MIPS3 instructions), the above address
is illegal.  If the $sp is always at least 32 KB below the top of the
address space, this problem does not arise, since any such intermediate
pointer generated by the compiler will always be below 0x80000000.

---

From: Harald Koerfgen <[email-masked]>
Date: Fri, 07 Jan 2000 20:09:05 +0100 (MET)

On 06-Jan-00 William J. Earl wrote:
> binaries).  SGI IRIX "-n32" and "-n64" binaries assumes FR=1
> (R4000-compatible), and also have a somewhat different register calling
> convention (which affects where arguments to system calls reside).

Wouldn't it make sense then if we made FR=0 the default for Linux/MIPS?

---
Regards,
Harald

---

From: Ralf Baechle <[email-masked]>
Date: Sat, 8 Jan 2000 00:29:11 +0100

> Wouldn't it make sense then if we made FR=0 the default for Linux/MIPS?

Obviously; you can clear that flag painless somewhere in
arch/mips/kernel/traps.c only for those CPUs that actually have it.

  Ralf

---

From: "William J. Earl" <[email-masked]>
Date: Fri, 7 Jan 2000 17:30:04 -0800 (PST)

> Wouldn't it make sense then if we made FR=0 the default for Linux/MIPS?

      FR should be tied to the compilation model, which is reflected
in the magic number of the executable.  For "-32" ("O32"), FR must be 0.
That is, there is no need for default; exec should set FR appropriately.

---

From: John Michael Clemens <[email-masked]>
Date: Sat, 8 Jan 2000 18:12:19 -0500 (EST)

So how insane would I be if I tried to compile binutils 2.9.5.0.22 and gcc
2.95.2 on my own to build a i386-hosted cross-compiler for playing with
Linux on my I2 R4400?

or should I just stick to the stuff on oss.sgi.com?

john.c
- --
/* John Clemens     http://www.rpi.edu/~clemej _/ "I Hate Quotes"       */
/* ICQ: 7175925     [email-masked]           _/    -- Samuel L. Clemens */ 
/* RPI Comp. Eng. 2000, Linux Parallel/Network/OS/Driver Specialist     */

---

From: Ralf Baechle <[email-masked]>
Date: Mon, 10 Jan 2000 05:01:12 +0100

> 2.95.2 on my own to build a i386-hosted cross-compiler for playing with

No chance with these versions.

> or should I just stick to the stuff on oss.sgi.com?

Yes.

  Ralf