Subject: Unable to handle kernel paging request at virtual address 04000460

From: <[email-masked]>
Date: Tue, 24 May 2005 21:02:06 +0530

Hi,
We are facing the following crash in custom Linux 2.4.26 kernel, when we
run a netperf TCP Stream (sizes varying from 64 to 32586 bytes) test
over an IPSEC tunnel created between a host and a VPN server through our
box. This is a Au1550 MIPS32 based board (DB1550 Cabernet board from
AMD). We observe that crash happens randomly (the PrId keeps changing at
each crash), because of burstiness in the netperf tool generated
traffic. Please look into the following capture below. I'd like some
help in debugging this issue. The same set of IPSEC drivers works fine
on a custom Linux 2.4.25 based kernel. Is there a patch that needs to be
applied for Linux 2.4.26 ? 
 
Unable to handle kernel paging request at virtual address 04000460, epc
== 802501cc, 8Oops in fault.c::do_page_fault, line 206:

$0 : 00000000 1000fc00 00000000 00000001 00000000 8b5f61b2 04000460
00000000

$8 : 00000000 00000000 1a000a08 0301590f 8b37d9d8 00000000 8b37dbd0
aa85e405

$16: 8b6d4780 00000001 8b6d4780 c01ce2a8 00000040 803251c0 00000001
8b37dc80

$24: 00000002 2afcee20 8b37c000 8b37dbb0 00004031 c01ce108

Hi : 00000000

Lo : 00000002

epc : 802501cc Not tainted

Status: 1000fc03

Cause : 00800008

PrId : 03030200

Process l2tpd (pid: 304, stackpage=8b37c000)

Stack: 1000fc03 00000002 00000000 8b5f61b2 80335808 c01ce108 1a000a08

0301590f 8b37d9d8 00000000 80255920 80255884 803d9400 803251cc 0000002c

803251ec 80352c88 8b6d4280 00000008 00000001 80255c94 2afcee20 00000006

80325080 8b37c000 8b37dc38 00004031 80255e48 803d9400 803251cc 0000002d

803251ec 80255e48 80255e60 8011b368 8011b368 8026a9e0 8025f444 803252b0

803251ec ...

Call Trace: [<c01ce108>] [<80255920>] [<80255884>] [<80255c94>]
[<80255e48>]

[<80255e48>] [<80255e60>] [<8011b368>] [<8011b368>] [<8026a9e0>]
[<8025f444>]

[<8025601c>] [<80116d90>] [<8011687c>] [<8010101c>] [<8010133c>]
[<801012ec>]

[<8026a7b8>] [<802ca7c8>] [<80255d08>] [<802c9f54>] [<80255e48>]
[<801aab24>]

[<801a3dc4>] [<801aab24>] [<801334c4>] [<801a1ee4>] [<8014fe9c>]
[<8014fe80>]

[<8024c918>] [<801a01a0>] [<801a22f4>] [<8015076c>] [<80150638>]
[<80105f0c>]

[<801046ac>] [<80107688>]

Code: 8e06009c 10c0000e 24030001 <8cc20000> c0450000 00a32023 e0440000
1080fffc 0

Kernel panic: Aiee, killing interrupt handler!

In interrupt handler - not syncing

Any help in debugging is welcome.

Regards,

Raghu

---

From: "Philippe De Swert" <[email-masked]>
Date: Tue, 24 May 2005 16:47:02 +0100

Decode this dump with ksymoops

See here:
http://freshmeat.net/projects/ksymoops/

This will help you discover in which function it is dying, on which offset in
your kernel code etc... Very helpful for debugging. (in 2.6 you can even
compile something alike in the kernel (kallsyms))

---------- Initial header -----------

> $0 : 00000000 1000fc00 00000000 00000001 00000000 8b5f61b2 04000460
<SNIP> KERNEL DUMP

CHEERS,

Philippe
 
| Philippe De Swert       
|      
| Stag developer http://stag.mind.be/  
| Emdebian developer: http://www.emdebian.org  
|   
| Please do not send me documents in a closed 
| format.(*.doc,*.xls,*.ppt)    
| Use the open alternatives. (*.pdf,*.ps,*.html,*.txt)    
| http://www.gnu.org/philosophy/no-word-attachments.html  

-------------------------------------------------------
NOTE! My email address is changing to ... @scarlet.be
Please make the necessary changes in your address book.

---

From: <[email-masked]>
Date: Wed, 25 May 2005 20:32:03 +0530

Hi Philippe,
We were able to build the cross ksymoops utility and converted couple of
Oops capture to analyze the dump. All the crashes were pointing to
skbuff issues. When packets are received, the __kfree_skb function is
crashing at some point and when packets are transmitted the
skb_drop_fraglist is crashing at some point.

Please find attached the oops decoded dumps. 
We would like to know are there any patches that need to be applied for
2.4.26 for it to handle fragmented packets when load is heavy ? We use
flood ping test on linux to get the crashes.

Thanks for your info.
Regards,
Raghu

-----Original Message-----
From: Philippe De Swert [mailto:[email-masked]] 
Sent: Tuesday, May 24, 2005 9:17 PM
To: Raghunathan Venkatesan (WT01 - EMBEDDED & PRODUCT ENGINEERING
SOLUTIONS)
Cc: linux-mips
Subject: Re:Unable to handle kernel paging request at virtual address
04000460

See here:

This will help you discover in which function it is dying, on which
offset in your kernel code etc... Very helpful for debugging. (in 2.6
you can even compile something alike in the kernel (kallsyms))

---------- Initial header -----------

> We are facing the following crash in custom Linux 2.4.26 kernel, when 
> we run a netperf TCP Stream (sizes varying from 64 to 32586 bytes) 
> test over an IPSEC tunnel created between a host and a VPN server 
> through our box. This is a Au1550 MIPS32 based board (DB1550 Cabernet 
> board from AMD). We observe that crash happens randomly (the PrId 
> keeps changing at each crash), because of burstiness in the netperf 
> tool generated traffic. Please look into the following capture below. 
> I'd like some help in debugging this issue. The same set of IPSEC 
> drivers works fine on a custom Linux 2.4.25 based kernel. Is there a 
> patch that needs to be applied for Linux 2.4.26 ?
>  
> Unable to handle kernel paging request at virtual address 04000460, 
> epc == 802501cc, 8Oops in fault.c::do_page_fault, line 206:
> 
<SNIP> KERNEL DUMP

CHEERS,

Philippe
 
| Philippe De Swert       
|      

-------------------------------------------------------
NOTE! My email address is changing to ... @scarlet.be Please make the
necessary changes in your address book.