Subject: RE: RM7k cache_flush_sigtramp

From: Adam Kiepul <[email-masked]>
Date: Fri, 1 Aug 2003 08:42:33 -0700

Hi Fuxin,

Could you please provide me with the _original_ Kernel code disassembly snippet around the point where your SYNC patch applies?
Also, can you check what RM7000 part revision is on your board? You can find it out by reading the PrID register.

I will check if there is an erratum that the code could trigger.

By the way, are you aware of any other ev64240 board that would exhibit the same behavior?

I would be quite careful drawing any conclusions at the moment since we can not preclude the possibility that it is simply a "bad CPU on the board" case. Please note that the SYNC instruction changes a lot in the manner things physically happen in the CPU so it can often mask off various problems, such as a bad part.

Thank you,

Adam

-----Original Message-----
From: Fuxin Zhang [mailto:[email-masked]]
Sent: Thursday, July 31, 2003 9:59 PM
To: Ralf Baechle
Cc: Adam Kiepul; MAKE FUN PRANK CALLS
Subject: Re: RM7k cache_flush_sigtramp

I am using a slightly modified 2.4.21-pre4,based on cvs of early this 
month(?).
We have merged with latest cvs, I will run it and report the result tonight.

Ralf Baechle wrote:

>Adam,
>
>On Fri, Aug 01, 2003 at 08:40:14AM +0800, Fuxin Zhang wrote:
>
>  
>
>>Current linux code does exactly this. But I was seeing all kinds of 
>>faults occuring around the
>>sigreturn point on the stack without a sync? And a sync does greatly 
>>improve the stablity.
>>
>>    
>>
>>>The ordering does matter however since the Hit_Invalidate_I makes sure the 
>>>write buffer is flushed.
>>>      
>>>
>
>could there be an errata explaining Fuxin's findings?
>
>Fuxin, what version are you running?
>
>  Ralf
>
>
>  
>

---

From: Fuxin Zhang <[email-masked]>
Date: Mon, 04 Aug 2003 11:38:28 +0800

Hi Adam,
   
   My cpu PRID:  0x2732, runs at freq 133x2MHz

disassemble code before patch:

ffffffff8010f2fc <r4k_flush_cache_sigtramp>:
ffffffff8010f2fc:	3c03802c 	lui	v1,0x802c
ffffffff8010f300:	246367ec 	addiu	v1,v1,26604
ffffffff8010f304:	94620010 	lhu	v0,16(v1)
ffffffff8010f308:	94650000 	lhu	a1,0(v1)
ffffffff8010f30c:	00021023 	negu	v0,v0
ffffffff8010f310:	00821024 	and	v0,a0,v0
ffffffff8010f314:	bc550000 	cache	0x15,0(v0)
ffffffff8010f318:	00052823 	negu	a1,a1
ffffffff8010f31c:	00852024 	and	a0,a0,a1
ffffffff8010f320:	bc900000 	cache	0x10,0(a0)
ffffffff8010f324:	03e00008 	jr	ra
ffffffff8010f328:	00000000 	nop

disassemble code after patch:
ffffffff8010ceb0 <r4k_flush_cache_sigtramp>:
ffffffff8010ceb0:	3c03802f 	lui	v1,0x802f
ffffffff8010ceb4:	2463e3ac 	addiu	v1,v1,-7252
ffffffff8010ceb8:	94620010 	lhu	v0,16(v1)
ffffffff8010cebc:	94650000 	lhu	a1,0(v1)
ffffffff8010cec0:	00021023 	negu	v0,v0
ffffffff8010cec4:	00821024 	and	v0,a0,v0
ffffffff8010cec8:	bc550000 	cache	0x15,0(v0)
ffffffff8010cecc:	0000000f 	sync
ffffffff8010ced0:	00052823 	negu	a1,a1
ffffffff8010ced4:	00852024 	and	a0,a0,a1
ffffffff8010ced8:	bc900000 	cache	0x10,0(a0)
ffffffff8010cedc:	03e00008 	jr	ra
ffffffff8010cee0:	00000000 	nop

We do have more than one set of ev64240 and RM7k cpu，but it will take 
some time for
me to get another one for test. I will tell you the result once i do it.

Thank you.

---

From: Fuxin Zhang <[email-masked]>
Date: Wed, 06 Aug 2003 19:00:07 +0800

hi,
   These days I have performed more experiments on our ev64240 board.

   Now it seems I get at least two problems: sigtramp flush and L3 cache.

   Let me descripe the phenomemena first.
      1.  fsck /dev/hda4(a 10G partition of 40G ide disk on a pci add-on 
card use
            intel piix4 chip) frequently fail with oops in various place:
                __remove_inode_queue, free_buffers, vmscan:359 etc.
       2. occasionally other apps may fail with segmentation fault or 
bus error.
       3. xwindow system is extremely unstable,both the applications and 
the
           Xserver may fail with sigill/sigsegv/sigbus etc.

   To address the problems, I modified arch/mips/signal.c to let kernel 
dump core
unconditionally(even if there are use handler installed) for 
sigill/sigsegv/sigbus.
By this way I get many core files for XFree86,then I find that they all 
look quite
similiar--all around the point of kernel generated sigreturn code(Two 
example are
 attached). Days ago i added a 'sync' after writeback and the situation 
was much better.
But then i still see this kinds of failure even with the 'sync'.  I have 
to go further back
to use 'Writeback_SD', so far no more such fault. But just as Adam 
pointed out,it may
just mask over another error.  I have tried to add code in 
r4k_flush_sigtramp and
sigreturn,and when xserver fails,I do observe that there are flush for 
the faulting point,
but no sigreturn executed. So it is at my wit's end:(. Maybe some 
complex schedule or
reentry problem? Or even a potential bug of context management(e.g.,we 
are using the
other's stack)?

   Using Writeback_SD only help xserver problem, the other problems look 
like
cache related. So I try to run with L3 cache disabled. That helps 
greatly, no oops
now. With a little tweak on ide code,the 'lost interrupt' problem seems 
gone too.
But with only L3 disabled, the Xserver problem remains.

  I am doing stress test now. Hope it won't give me more surprise.

  And here I have a question for Mr. Adam: original linux code use 
'Writeback_Inv_D"
and "Hit_Invalidate_I",not "Writeback_D" and "Hit_Invalidate_I",could it 
lead to the
problem?

 BTW:
   a silly question: how can i make my email show up pretier? I find 
that the mailing list
often break my lines very badly. I feel guilty for that:) I am using 
mozilla composer,the
original linebreaks are manually inserted(hit enter when i feel it is 
long enough).

Thank you for any help.

>>Fuxin, what version are you running?
GNU gdb 2002-04-01-cvs
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "mipsel-linux"...(no debugging symbols found)...
Core was generated by `/usr/bin/X11/X -dpi 100 -nolisten tcp'.
Program terminated with signal 4, Illegal instruction.
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/ld.so.1
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...
done.
Loaded symbols for /lib/libnss_files.so.2

    GDB is unable to find the start of the function at 0x7fff7600
and thus can't determine the size of that function's stack frame.
This means that GDB may be unable to access that stack frame, or
the frames below it.
    This problem is most likely caused by an invalid program counter or
stack pointer.
    However, if you think GDB should simply search farther back
from 0x7fff7600 for code which looks like the beginning of a
function, you can increase the range of the search using the `set
heuristic-fence-post' command.
#0  0x7fff7600 in ?? ()
(gdb) where
#0  0x7fff7600 in ?? ()
(gdb) disass 0x7fff7580 0x7fff7680
Dump of assembler code from 0x7fff7580 to 0x7fff7680:
0x7fff7580:	nop
0x7fff7584:	nop
0x7fff7588:	nop
0x7fff758c:	nop
0x7fff7590:	nop
0x7fff7594:	nop
0x7fff7598:	nop
0x7fff759c:	nop
0x7fff75a0:	nop
0x7fff75a4:	nop
0x7fff75a8:	nop
0x7fff75ac:	nop
0x7fff75b0:	nop
0x7fff75b4:	nop
0x7fff75b8:	nop
0x7fff75bc:	nop
0x7fff75c0:	nop
0x7fff75c4:	nop
0x7fff75c8:	nop
0x7fff75cc:	nop
0x7fff75d0:	nop
0x7fff75d4:	beq	at,t3,0x80003ef8
0x7fff75d8:	sllv	zero,zero,zero
0x7fff75dc:	0xe
0x7fff75e0:	beq	zero,gp,0x7ffef244
0x7fff75e4:	beq	zero,t0,0x800012e8
0x7fff75e8:	0x7fff7600
0x7fff75ec:	beq	zero,t2,0x7ffd98b0
0x7fff75f0:	sd	ra,-1(ra)
0x7fff75f4:	sd	ra,-1(ra)
0x7fff75f8:	slti	sp,s6,-25040
0x7fff75fc:	beq	zero,t2,0x7ffd9cc0
0x7fff7600:	li	v0,4119
0x7fff7604:	syscall
0x7fff7608:	0x12c
0x7fff760c:	lb	a0,-19437(zero)
0x7fff7610:	slti	s1,s6,17812
0x7fff7614:	nop
0x7fff7618:	nop
0x7fff761c:	nop
0x7fff7620:	0xcf9210
0x7fff7624:	nop
0x7fff7628:	mfhi	zero
0x7fff762c:	nop
0x7fff7630:	beq	zero,at,0x8000caa4
0x7fff7634:	nop
0x7fff7638:	0xe
0x7fff763c:	nop
0x7fff7640:	slti	v0,k1,28680
0x7fff7644:	nop
0x7fff7648:	0x1228
0x7fff764c:	nop
0x7fff7650:	nop
0x7fff7654:	nop
0x7fff7658:	multu	zero,zero
0x7fff765c:	nop
0x7fff7660:	nop
0x7fff7664:	nop
0x7fff7668:	slti	t9,s7,17756
0x7fff766c:	nop
0x7fff7670:	beq	at,t3,0x7fff6f04
0x7fff7674:	nop
0x7fff7678:	0x12c
0x7fff767c:	nop
End of assembler dump.
(gdb) info regs
(gdb) info regs 
          zero       at       v0       v1       a0       a1       a2       a3
 R0   00000000 b004b400 ffffffff ffffffff 00000001 7fff73f8 00000000 00000000 
            t0       t1       t2       t3       t4       t5       t6       t7
 R8   0000b400 00000000 00000000 00000000 00000000 822b2880 822b2900 00000000 
            s0       s1       s2       s3       s4       s5       s6       s7
 R16  00000000 102b3248 00000004 0000000e 101cdf18 102b1820 00000001 00000000 
            t8       t9       k0       k1       gp       sp       s8       ra
 R24  00000000 006c86ec 00000000 00000000 10082740 7fff75f0 7fff7d08 7fff7600 
            sr       lo       hi      bad    cause       pc
      a004b413 00000002 00000000 8009c6a0 00000028 7fff7600 
           fsr      fir       fp
      00800004 00000000 00000000 
(gdb) quit
GNU gdb 2002-04-01-cvs
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "mipsel-linux"...(no debugging symbols found)...
Core was generated by `/bin/sh /usr/bin/X11/startx'.
Program terminated with signal 4, Illegal instruction.

    GDB is unable to find the start of the function at 0x7fff75b8
and thus can't determine the size of that function's stack frame.
This means that GDB may be unable to access that stack frame, or
the frames below it.
    This problem is most likely caused by an invalid program counter or
stack pointer.
    However, if you think GDB should simply search farther back
from 0x7fff75b8 for code which looks like the beginning of a
function, you can increase the range of the search using the `set
heuristic-fence-post' command.
#0  0x7fff75b8 in ?? ()
(gdb) info reg
          zero       at       v0       v1       a0       a1       a2       a3
 R0   00000000 2ad918f0 2ad918f0 0000000a 00000012 7fff7538 00000001 00000001 
            t0       t1       t2       t3       t4       t5       t6       t7
 R8   0000000a 2aca6394 00000000 00000004 00000000 00000000 00000000 07200720 
            s0       s1       s2       s3       s4       s5       s6       s7
 R16  00000000 00000004 00000080 7fff7878 00000003 ffffffff 1000f0f8 00000001 
            t8       t9       k0       k1       gp       sp       s8       ra
 R24  00000000 00000000 00000000 00000000 1000d880 7fff7590 00000003 7fff75a0 
            sr       lo       hi      bad    cause       pc
      a004f413 000001b0 00000000 8009c6a0 80000028 7fff75b8 
           fsr      fir       fp
      00000000 00000000 00000000 
(gdb) disass 0x7fff7500 0x7fff7600
Dump of assembler code from 0x7fff7500 to 0x7fff7600:
0x7fff7500:	0xc2009d
0x7fff7504:	0x10000e8
0x7fff7508:	0x11a0110
0x7fff750c:	0x990121
0x7fff7510:	slti	t9,s6,32304
0x7fff7514:	tltu	a0,t9,0x2
0x7fff7518:	slti	t9,s6,32304
0x7fff751c:	0x442c88
0x7fff7520:	nop
0x7fff7524:	nop
0x7fff7528:	nop
0x7fff752c:	nop
0x7fff7530:	b	0x7ffed734
0x7fff7534:	nop
0x7fff7538:	nop
0x7fff753c:	slti	t8,s6,-8108
0x7fff7540:	nop
0x7fff7544:	sllv	zero,zero,zero
0x7fff7548:	sll	zero,zero,0x2
0x7fff754c:	0x7fff7878
0x7fff7550:	sra	zero,zero,0x0
0x7fff7554:	sd	ra,-1(ra)
0x7fff7558:	b	0x7fff393c
0x7fff755c:	b	0x7ffed760
0x7fff7560:	teq	v0,a0,0xa9
0x7fff7564:	nop
0x7fff7568:	nop
0x7fff756c:	nop
0x7fff7570:	nop
0x7fff7574:	nop
0x7fff7578:	b	0x7ffed77c
0x7fff757c:	nop
0x7fff7580:	nop
0x7fff7584:	b	0x7ffed788
0x7fff7588:	0x7fff75a0
0x7fff758c:	0x1
0x7fff7590:	b	0x7fff2174
0x7fff7594:	0x7fff7804
0x7fff7598:	slti	t9,s6,32304
0x7fff759c:	0x475718
0x7fff75a0:	li	v0,4119
0x7fff75a4:	syscall
0x7fff75a8:	slti	t8,s6,-8108
0x7fff75ac:	lb	a0,-3053(zero)
0x7fff75b0:	slti	t5,s6,9620
0x7fff75b4:	nop
0x7fff75b8:	nop
0x7fff75bc:	nop
0x7fff75c0:	b	0x7fff8cb4
0x7fff75c4:	nop
0x7fff75c8:	sllv	zero,zero,zero
0x7fff75cc:	nop
0x7fff75d0:	nop
0x7fff75d4:	nop
0x7fff75d8:	sra	zero,zero,0x0
0x7fff75dc:	nop
0x7fff75e0:	0x7fff7878
0x7fff75e4:	nop
0x7fff75e8:	sll	zero,zero,0x2
0x7fff75ec:	nop
0x7fff75f0:	0x1
0x7fff75f4:	nop
0x7fff75f8:	mult	zero,zero
0x7fff75fc:	nop
End of assembler dump.
(gdb) quit

---

From: Ralf Baechle <[email-masked]>
Date: Wed, 6 Aug 2003 13:55:31 +0200

> and "Hit_Invalidate_I",not "Writeback_D" and "Hit_Invalidate_I",could it 

No.  To synchronize the D-cache and I-cache it's irrelevant if you
invalidate the D-cache or not.

> original linebreaks are manually inserted(hit enter when i feel it is 

Format your email with hard breaks to about 75 columns.  75 columns
because god made vt100 with 80 columns so that leaves a bit of space for
quoting your mail nicely.

> (gdb) info reg
[...]
>      a004f413 000001b0 00000000 8009c6a0 80000028 7fff75b8
[...]

> 0x7fff75a4:     syscall

So the pc is pointing just after the trampoline which suspiciously looks
like the return of an old bug.  Could your application be doing something
unusual such as forking from a signal handler or similar?  The scenario
is about

 - kernel installs signal trampoline on stack
 - kernel forks.  Now the signal trampoline installed in the first step
   resides on a copy-on-write page.
 - newly created process touches the cow page, thereby resulting in
   breaking of the cow page.  Now parent and child have their own copy
   of the page.  BUT: flush_cache_page() doesn't properly flush this page.
 - Parent executes again on the copy of the page for which caches have
   not been flushed proplerly in the previous step, thereby failing to
   execute the trampoline - crash.

  Ralf

---

From: Fuxin Zhang <[email-masked]>
Date: Wed, 06 Aug 2003 20:52:46 +0800

>invalidate the D-cache or not.
I think so. Just in case the hardware is doing something strange:)

>quoting your mail nicely.
Thanks:)

>unusual such as forking from a signal handler or similar?  The scenario
I am not sure. It is stardard X distribution from debian-woody. Fairly 
easy to reproduce,just move the mouse
around and click here and there then it would die. Will check this 
later,but I think such a giant as Xserver
won't fork frequently.

>   of the page.  BUT: flush_cache_page() doesn't properly flush this page
>  
>
If the new process touch the cow page first,shouldn't it get a new page 
and leave the original page for parent?
If so,the parent should be able to see the trampoline content from 
icache anyway(either L2 or memory should
have the value),though the child may not?

>   execute the trampoline - crash.
RM7000 has 16k 4-way set-associated primary caches,which are supposed to 
have no cache aliasing problem

Bad news:

oops again:(  while true; do fsck -y -f /dev/hda4 ; done 

after about 5 succeeded run.
So still some  problems lurking somewhere.

It seems I have to switch some hardware...

---

From: Ralf Baechle <[email-masked]>
Date: Wed, 6 Aug 2003 16:45:14 +0200

On Wed, Aug 06, 2003 at 08:52:46PM +0800, Fuxin Zhang wrote:

> later,but I think such a giant as Xserver won't fork frequently.

The scenario I was describing was just how we did originally discover the
bug.  Supposedly that was fixed but your register dump and dissassembly
show the exact fingerprint of that old problem, so I though I should
describe it in the hope it's going to help you.

> have the value),though the child may not?

RM7000 has a physically indexed cache.  That means if the copy of the
page wasn't explicitly or implicitly written back to L2 the process
whichever ends up with the copy of the page might fetch stale instructions
from memory - boom.

> have no cache aliasing problem

The described scenario is not an aliasing problem; it's the case where the
copy of the cow page hasn't properly been flushed at all.  When we
isolated the bug was that neither flush_page_to_ram() nor flush_cache_page()
were flushing the cache.  I suspect your case must be something fairly
similar.

  Ralf

---

From: Fuxin Zhang <[email-masked]>
Date: Wed, 06 Aug 2003 23:04:19 +0800

>were flushing the cache.  I suspect your case must be something fairly
After cache rewrite,flush_page_to_ram is null; and in this case 
flush_cache_page
 do nothing for a stack page. (It flushes only when has_dc_aliases or 
exec set).
So  the  one use  the new copy will  have problem ?!  Am I missing 
something?

Thank you very much, great Ralf:).

---

From: Ralf Baechle <[email-masked]>
Date: Thu, 7 Aug 2003 00:30:09 +0200

> So  the  one use  the new copy will  have problem ?!  Am I missing 

The stack page contains the trampoline so it must be marked executable,
so on an RM7000 flush_dcache_page must flush both the D-cache and
I-cache.

  Ralf