Subject: Kernel crash on boot with current cvs (todays)

From: Florian Lohoff <[email-masked]>
Date: Mon, 11 Jun 2001 00:03:59 +0200

Hi,
i just tried to boot an Indy of mine with the current cvs (from this
morning) and it crashes 

ksymoops 2.4.1 on i686 2.2.18ext3.  Options used
     -v vmlinux-2.4.3 (specified)
     -K (specified)
     -L (specified)
     -o lib/modules/2.4.3 (specified)
     -m System.map-2.4.3 (specified)

No modules in ksyms, skipping objects
kernel BUG at semaphore.c:235!
Unable to handle kernel paging request at virtual address 00000000, epc == 8800f02c, ra == 8800f02c
$0 : 00000000 3004fc00 0000001f 89d0e000
$4 : 00000017 00000000 00000001 88669440
$8 : 0000000a 8814cc50 00000000 00000000
$12: 00000000 8814c870 fffffff9 0000000a
$16: 8866977c 00000000 89d0e000 88669784
$20: 89d0ff04 89d0ff00 89d0fea0 00000009
$24: 89d0fcf2 ffffffff
$28: 89d0e000 89d0fdd8 7fff7ba0 8800f02c
epc   : 8800f02c
Using defaults from ksymoops -t elf32-little -a unknown
Status: 3004fc03
Cause : 3000000c
Process start-stop-daem (pid: 211, stackpage=89d0e000)
Stack: 88106e30 88106e48 000000eb 00000000 8866977c 88669760 8800ed54 2aba3820
       00000000 885d1280 88064cf0 88065cd4 00000000 89d0e000 88669784 88669784
       fffffffe 88669760 89e24640 8866977c 89d0ff04 88064b50 89e24460 8bd1d8e0
       89d0ff00 89d0ff00 89e24640 89d0ff00 89e24640 8b6d400d 89d0ff00 89d0ff04
       880655d0 88065598 89e24640 88060cf4 89e24460 8b6d400d 8bd1d960 8bd1d960
       88053f88 ...
Call Trace: [<88106e30>] [<88106e48>] [<8800ed54>] [<88064cf0>] [<88065cd4>] [<88064b50>] [<880655d0>] [<88065598>] [<88060cf4>] [<88053f88>] [<88054074>] [<88054cc0>] [<88054c68>] [<88053794>] [<88050414>] [<8800fe08>] [<8800fe08>]
Code: 240600eb  0e007d99  00000000 <ae200000> 26040010  24050003  0e007085  24060001  0a003bf9
Error (Oops_bfd_perror): scan_arch for specified architecture Success

>>RA;  8800f02c <__rwsem_wake+b8/dc>
>>???; 8800f02c <__rwsem_wake+b8/dc>   <=====
Trace; 88106e30 <__gnu_compiled_c+d90/1204>
Trace; 88106e48 <__gnu_compiled_c+da8/1204>
Trace; 8800ed54 <__down_read+1dc/1e4>
Trace; 88064cf0 <proc_root_link+b4/e4>
Trace; 88065cd4 <proc_pid_make_inode+24/10c>
Trace; 88064b50 <proc_exe_link+1cc/1d4>
Trace; 880655d0 <proc_pid_follow_link+98/b0>
Trace; 88065598 <proc_pid_follow_link+60/b0>
Trace; 88060cf4 <update_atime+6c/78>
Trace; 88053f88 <path_walk+334/a94>
Trace; 88054074 <path_walk+420/a94>
Trace; 88054cc0 <__user_walk+78/94>
Trace; 88054c68 <__user_walk+20/94>
Trace; 88053794 <path_release+18/74>
Trace; 88050414 <sys_newstat+20/98>
Trace; 8800fe08 <stack_done+1c/38>
Trace; 8800fe08 <stack_done+1c/38>

1 error issued.  Results may not be reliable.

-- 
Florian Lohoff                  [email-masked]             +49-5201-669912
     Why is it called "common sense" when nobody seems to have any?

---

From: Keith Owens <[email-masked]>
Date: Mon, 11 Jun 2001 13:53:04 +1000

On Mon, 11 Jun 2001 00:03:59 +0200, 
>Error (Oops_bfd_perror): scan_arch for specified architecture Success

I cannot help with the oops but your ksymoops run has problems.  When
doing cross system oops debugging, you need to specify the target
machine and architecture with the -t and -a flags.  Otherwise ksymoops 
defaults to the current machine.  Adding -t and -a will fix the above
error and decode the code: line.

---

From: Ralf Baechle <[email-masked]>
Date: Mon, 11 Jun 2001 06:42:49 +0200

> Unable to handle kernel paging request at virtual address 00000000, epc == 8800f02c, ra == 8800f02c

This is a known and yet unresolved problem.  Yet I'm surprised - so far
I've only seen this problem on mips64.

  Ralf

---

From: [email-masked] (Raoul Borenius)
Date: Mon, 11 Jun 2001 16:50:19 +0200

Hi,

> I've only seen this problem on mips64.

Just FYI: the same happens on our Indy, see trace.txt and boot.txt.

Hope it helps...

Regards

Raoul

 ---------------------------------------------------------------------
 Raoul Gunnar Borenius		Deutsches Forschungsnetz e.V.
 WiNShuttle			Lindenspürstr.32, 70176 Stuttgart
 Phone  : +49 711 63314-206	FAX: +49 711 63314-133
 E-Mail : [email-masked]	http://www.shuttle.de
 ---------------------------------------------------------------------

---

From: Florian Lohoff <[email-masked]>
Date: Mon, 11 Jun 2001 21:12:51 +0200

> error and decode the code: line.

Nope - it wont as it strictly uses the /usr/bin/objdump - You have to
set the environment var "KSYMOOPS_OBJDUMP"

Flo

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Tue, 12 Jun 2001 03:01:32 +0200 (MET DST)

> set the environment var "KSYMOOPS_OBJDUMP"

 Good it has a way to override the default, but wouldn't looking for
objdump in the PATH variable (i.e. using execvp()) be more reasonable?
This way ksymoops would always work in a cross-compilation environment
(i.e. with $tooldir/bin preceding $bindir in PATH).

 Just an ad-hoc idea...

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: [email-masked], PGP key available        +

---

From: Keith M Wesolowski <[email-masked]>
Date: Mon, 11 Jun 2001 18:01:09 -0700

> (i.e. with $tooldir/bin preceding $bindir in PATH).

Actually, it would probably be preferable to try first arch-os-objdump
and maybe one or two other similar variants and then search the path.
Supposing I have multiple cross environments on my machine...

-- 
Keith M Wesolowski <[email-masked]> http://foobazco.org/~wesolows
------(( Project Foobazco Coordinator and Network Administrator ))------
	"Nothing motivates a man more than to see his boss put
	 in an honest day's work." -- The fortune file

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Tue, 12 Jun 2001 03:22:15 +0200 (MET DST)

> and maybe one or two other similar variants and then search the path.

 Sure, if you know how to guess the names (I don't).  The PATH approach
should work well if you choose one cross environment at a time.

> Supposing I have multiple cross environments on my machine...

 Does anyone have only a single one? ;-)  (Having a complete mipsel-linux,
a basic alpha-linux and a few obscure partial environments...)

---

From: Keith Owens <[email-masked]>
Date: Tue, 12 Jun 2001 12:05:19 +1000

> (i.e. with $tooldir/bin preceding $bindir in PATH).

I originally designed ksymoops that way but there are so many different
methods of naming the binutils for cross compile environments that I
gave up.  Using the path runs the risk of picking random versions of nm
and objdump and still not being correct.  Anybody running cross compile
has to specify parameters for System.map, ksyms and objects, so adding
paths for binutils is a small change and is the only way to guarantee
that the correct versions are used.

---

From: Florian Lohoff <[email-masked]>
Date: Tue, 12 Jun 2001 12:09:27 +0200

On Mon, Jun 11, 2001 at 04:50:19PM +0200, Raoul Borenius wrote:
> Linux version 2.4.3-cvs20010611 (raoul@bunny) (gcc version egcs-2.91.66 19990314 (egcs-1.1.2 release)) #12 Mon Jun 11 16:22:41 CEST 2001
> MC: SGI memory controller Revision 3

I got a hint that it might be the compile to produce this bug - I was
suggested to use some gcc 3.0 prerelease. I now checked again and i am
already using some gcc 3.0

Linux version 2.4.3 (flo@paradigm) (gcc version 3.0 20010303 (prerelease))
#1 Mon Jun 11 00:27:09 CEST 2001

(flo@paradigm)~# mips-linux-gcc -v
Reading specs from /usr/local/lib/gcc-lib/mips-linux/3.0/specs
Configured with: ./configure --prefix=/usr/local --target=mips-linux --enable-languages=c --with-newlib --disable-shared
gcc version 3.0 20010303 (prerelease)
(flo@paradigm)~# mips-linux-as -v
GNU assembler version 2.11.90 (mips-linux) using BFD version 2.11.90

Flo

---

From: Ralf Baechle <[email-masked]>
Date: Tue, 12 Jun 2001 13:53:06 +0200

> already using some gcc 3.0

It's not a tool related bug but a genuine kernel bug in our semaphore code.
Which - unfortunately is a bit of headache to fix but is more or less the #1
on the list of my instabilities right now.

  Ralf

---

From: "H . J . Lu" <[email-masked]>
Date: Tue, 12 Jun 2001 09:37:15 -0700

> on the list of my instabilities right now.

I have some kernel crashes which are cured by some gcc/binutils changes
which I don't believe should make any differences. I thought I could
take out the mips64 and -march patches. But wtithout them, the user
applications seem ok, but the kernel crashes in all different places.
I have to put them back in. They may be kernel bugs and my gcc/binutils
changes may just hide them.

BTW, what is the current 2.4 kernel patches for Algorithmics P6032?

Thanks.

H.J.

---

From: Florian Lohoff <[email-masked]>
Date: Wed, 13 Jun 2001 12:56:10 +0200

On Wed, Jun 13, 2001 at 10:06:02AM +0200, Raoul Borenius wrote:
> Thanx for the quick response. Just to confirm that, here is the same crash
> with the kernel from
> 
> ftp://ftp.rfc822.org/pub/local/debian-mips/kernel/kernel-image-2.4.3-ip22-r4k.tgz
> 
> Using ksymoops gave a lot of warnings this time. Don't know why, the
> System.map should be the right one (it's out of
> kernel-image-2.4.3-ip22-r4k.tgz).

This is because the system map has been generated with newer binutils
which always dump the addresses as 64Bit addresses. Load the system
map into an text editor and delete the first ffffffff from the
addresses 1,$ s/^ffffffff//

> Warning (compare_maps): mismatch on symbol EISA_bus  , ksyms_base says 88162b44, System.map says ffffffff88162b44.  Ignoring ksyms_base entry
> Warning (compare_maps): mismatch on symbol ROOT_DEV  , ksyms_base says 881711e8, System.map says ffffffff881711e8.  Ignoring ksyms_base entry
> Warning (compare_maps): mismatch on symbol ___strtok  , ksyms_base says 88194d68, System.map says ffffffff88194d68.  Ignoring ksyms_base entry
> Warning (compare_maps): mismatch on symbol ___wait_on_page  , ksyms_base says 88032a20, System.map says ffffffff88032a20.  Ignoring ksyms_base entry
> Warning (compare_maps): mismatch on symbol __alloc_pages  , ksyms_base says 8803de98, System.map says ffffffff8803de98.  Ignoring ksyms_base entry
> Warning (compare_maps): mismatch on symbol __bforget  , ksyms_base says 880472b8, System.map says ffffffff880472b8.  Ignoring ksyms_base entry

Flo

---

From: Keith Owens <[email-masked]>
Date: Wed, 13 Jun 2001 21:34:18 +1000

On Wed, 13 Jun 2001 12:56:10 +0200, 
Florian Lohoff <[email-masked]> wrote:
>which always dump the addresses as 64Bit addresses.

Looks like I need to add a new option to ksymoops.  -T <bits>, truncate
all addresses to this bit size.  Added to my list for the next ksymoops
release.

---

From: [email-masked] (Raoul Borenius)
Date: Wed, 13 Jun 2001 13:37:51 +0200

> addresses 1,$ s/^ffffffff//

Should have looked closer, sorry...

I've attached the output of ksymoops.

Regrds

Raoul

---

From: Keith Owens <[email-masked]>
Date: Wed, 13 Jun 2001 21:46:20 +1000

On Wed, 13 Jun 2001 13:37:51 +0200, 
[email-masked] (Raoul Borenius) wrote:
>Unable to handle kernel paging request at virtual address 00000000, epc == 8800e                               e5c, ra == 8800ee5c
>Call Trace: [<88124dc0>] [<88124dd8>] [<8800eb84>] [<88063810>] [<880647f4>] [<8                               8063670>] [<880640f0>] [<880640b8>] [<8805f7f4>] [<88052a88>] [<88052b74>] [<880
>537c0>] [<88053768>] [<88052294>] [<8804ef14>] [<8800fc28>] [<8800c638>] [<c013d                               686>]

The epc line and the call trace have embedded spaces and newlines which
are preventing ksymoops from doing a full conversion.  Try again with
the extra characters removed, i.e.

Unable to handle kernel paging request at virtual address 00000000, epc == 8800e5c, ra == 8800ee5c
....
Call Trace: [<88124dc0>] [<88124dd8>] [<8800eb84>] [<88063810>]
[<880647f4>] [<88063670>] [<880640f0>] [<880640b8>] [<8805f7f4>]
[<88052a88>] [<88052b74>] [<880537c0>] [<88053768>] [<88052294>]
[<8804ef14>] [<8800fc28>] [<8800c638>] [<c013d686>]
Code: ...

---

From: Ralf Baechle <[email-masked]>
Date: Wed, 13 Jun 2001 14:05:51 +0200

> all addresses to this bit size.  Added to my list for the next ksymoops

That can be done automatically.  For 32-bit ELF files mips*-linux binutils
dump some of the addresses as 32-bit addresses, some as sign-extended (!)
64-bit addresses.  So ksymops should just sign extend any 32-bit addresses
to 64-bit and then work on full lenght addresses.

Is ksymoops able to handle 64-bit addresses when running on a 32-bit host?
That is a common case for many people when decoding their MIPS oopses.

  Ralf

---

From: Keith Owens <[email-masked]>
Date: Wed, 13 Jun 2001 22:19:14 +1000

On Wed, 13 Jun 2001 14:05:51 +0200, 
>to 64-bit and then work on full lenght addresses.

Some utilities extend with leading 0, some with leading 1, some with
special values (alpha).  It is safer to truncate everything to the
specified width instead of guessing what the extension value is.

>That is a common case for many people when decoding their MIPS oopses.

Yes and no.  The framework is there to handle 32/64 splits, sparc
already uses it.  mips does not currently use the framework because the
oops report does not identify the machine type 32/64, MSB/LSB.  We
discussed this in January and I asked for a small change to mips64 oops
output (below), was that ever done?

------

From: Keith Owens <[email-masked]>
To: Ralf Baechle <[email-masked]>
Subject: Re: ksymoops on origin 
Date: Sat, 06 Jan 2001 18:21:35 +1100

ksymoops is designed to run on any build arch and debug an oops report
from any other target arch, as long as binutils supports the target
arch.  The presence or absence of __MIPSEL__ or __MIPSEB__ on the build
system says nothing about the type of the failing target, ksymoops
relies on text in the oops report to determine special cases like 32
bit userland and 64 bit kernel.

The best option is for a mips64 kernel to indicate that it is 64 bit
and its endianess.  Instead of printing

  "epc     : %016lx\n"

print

  "epc     : %016lx (64 "
#ifdef __MIPSEL__
		"LSB"
#else
		"MSB"
#endif
		")\n"

If you make that change to arch/mips64/mm/andes.c,
arch/mips64/mm/r4xx0.c and any other places that print epc on mips64
then I will change ksymoops to look for (64 [LM]SB) on the epc line and
decode accordingly.

While we are on the topic of 32 bit userland and 64 bit kernel, how
does mips64 handle modules?  modutils 2.4.0 has no support for mips64,
do you have any code?  Also I assume that you will want the same
ability as sparc, to compile modutils as a 32 bit program which can
handle both 32 and 64 bit modules.

---

From: [email-masked] (Raoul Borenius)
Date: Wed, 13 Jun 2001 15:16:44 +0200

> the extra characters removed, i.e.

Sorry about that. These things happen every time I try to do five or more
things at the same time...

Hope the trace is ok now.

Regards

Raoul

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Wed, 13 Jun 2001 15:44:28 +0200 (MET DST)

> That is a common case for many people when decoding their MIPS oopses.

 The point is whether addresses should be extended at all if a 32-bit
target is selected.  IMHO -- not, that's a limitation of BFD when
configured for both a 32-bit and a 64-bit target and it should be fixed
sooner or later (at least it's on my to-do list for some time).  BFD is
free to handle addresses as it likes internally, be it 64-bit or 32-bit,
but they should be truncated on final output to a 32-bit target, as they
already are for certain cases. 

 I'm not sure we need to work it around in ksymoops until BFD is fixed --
`cut -c8- < System.map' works as a charm.  It might be worth documenting,
though.

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Wed, 13 Jun 2001 15:44:28 +0200 (MET DST)

---

From: Keith Owens <[email-masked]>
Date: Thu, 14 Jun 2001 00:07:12 +1000

On Wed, 13 Jun 2001 15:44:28 +0200 (MET DST), 
"Maciej W. Rozycki" <[email-masked]> wrote:
>target is selected.  IMHO -- not ...
>`cut -c8- < System.map' works as a charm.

ksymoops also reads the output from nm and objdump internally, it is
just a little difficult to run cut -c8- on that data ;).

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Wed, 13 Jun 2001 17:08:20 +0200 (MET DST)

> just a little difficult to run cut -c8- on that data ;).

 I see.  So a workaround is likely needed for now.

---

From: Ralf Baechle <[email-masked]>
Date: Sat, 16 Jun 2001 11:24:18 +0200

> already are for certain cases. 

Agreed, that'd be much more readable for humans.  For processing by
software having some canonical form, that is 64-bit would be a bit more
handy though.

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Sat, 16 Jun 2001 11:22:20 +0200

> specified width instead of guessing what the extension value is.

That's a processor specific thing.  In case of MIPS only signed extension
is correct.  Some older binutils did unsigned extension and that's plain
wrong.

> output (below), was that ever done?

Nothing - ``Guilty as charged.  Filed in my "to do when I have time" folder,
which overflowed about 3 years ago :(.''

  Ralf

---

From: Keith Owens <[email-masked]>
Date: Wed, 27 Jun 2001 12:21:57 +1000

On Wed, 13 Jun 2001 22:19:14 +1000, 
>and its endianess.  Instead of printing

I have a new ksymoops release coming up.  Is it OK if I include code to
look for (64 LSB) and (64 MSB) in the oops and decode accordingly.  I
don't expect the kernel to produce this output immediately, I just want
agreement on the format that will be produced.

---

From: Thiemo Seufer <[email-masked]>
Date: Thu, 28 Jun 2001 16:32:51 +0200

Keith Owens wrote:
[snip]
> agreement on the format that will be produced.

The appended patch introduces the new format in mips64. Maybe this speeds
up agreement about it. :-)

Thiemo

diff -BurPX /bigdisk/dl/src/dontdiff linux-orig/arch/mips64/mm/andes.c linux/arch/mips64/mm/andes.c
--- linux-orig/arch/mips64/mm/andes.c	Sat May 26 00:25:39 2001
+++ linux/arch/mips64/mm/andes.c	Thu Jun 28 15:19:32 2001
@@ -332,7 +381,13 @@
 	printk("Lo      : %016lx\n", regs->lo);
 
 	/* Saved cp0 registers. */
-	printk("epc     : %016lx\nbadvaddr: %016lx\n",
+	printk("epc     : %016lx (64 "
+#ifdef __MIPSEL__
+	       "LSB"
+#else
+	       "MSB"
+#endif
+	       ")\nbadvaddr: %016lx\n",
 	       regs->cp0_epc, regs->cp0_badvaddr);
 	printk("Status  : %08x\nCause   : %08x\n",
 	       (unsigned int) regs->cp0_status, (unsigned int) regs->cp0_cause);
diff -BurPX /bigdisk/dl/src/dontdiff linux-orig/arch/mips64/mm/r4xx0.c linux/arch/mips64/mm/r4xx0.c
--- linux-orig/arch/mips64/mm/r4xx0.c	Thu Mar 29 17:11:46 2001
+++ linux/arch/mips64/mm/r4xx0.c	Thu Jun 28 14:51:20 2001
@@ -2125,7 +2118,13 @@
 	printk("Lo      : %016lx\n", regs->lo);
 
 	/* Saved cp0 registers. */
-	printk("epc     : %016lx\nbadvaddr: %016lx\n",
+	printk("epc     : %016lx (64 "
+#ifdef __MIPSEL__
+	       "LSB"
+#else
+	       "MSB"
+#endif
+	       ")\nbadvaddr: %016lx\n",
 	       regs->cp0_epc, regs->cp0_badvaddr);
 	printk("Status  : %08x\nCause   : %08x\n",
 	       (unsigned int) regs->cp0_status, (unsigned int) regs->cp0_cause);