Subject: RE: do_ri failure in cache flushing routines

From: G H <[email-masked]>
Date: Thu, 5 Aug 2004 11:04:27 -0700 (PDT)

I've not had much response to this question so I would like to rephrase it :
 
Can anyone think of any possible scenario where do_ri could occur in blast_icache32() ??
 
Is this possibly a cache synchronisation problem ??
 
TIA
 
>While testing out an amd au1500 based board I have been getting " do_ri " exceptions >that always occur in the cache flushing routines. More often than not in >blast_icache_32().
 
>So far this has mainly happened after running the board for days on end while running >multiple telnet sessions to it. It has sometimes ( quite rarely ) happened after a few >hours to a day of multiple telnet session use.

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

---

From: Pete Popov <[email-masked]>
Date: Thu, 05 Aug 2004 11:09:01 -0700

> Is this possibly a cache synchronisation problem ??

Could be a hardware memory glitch. I would use kgdb to put a breakpoint 
there and see what the data in memory looks like when this happens -- 
look for memory corruption, etc.

Pete

> TIA
>  
> >While testing out an amd au1500 based board I have been getting " 
> do_ri " exceptions >that always occur in the cache flushing routines. 
> More often than not in >blast_icache_32().
>  
> >So far this has mainly happened after running the board for days on 
> end while running >multiple telnet sessions to it. It has sometimes ( 
> quite rarely ) happened after a few >hours to a day of multiple telnet 
> session use.
>

---

From: Jun Sun <[email-masked]>
Date: Thu, 5 Aug 2004 11:11:33 -0700

> Can anyone think of any possible scenario where do_ri could occur in blast_icache32() ??

One possibility _could_ be the "instruction flushing itself" problem on
MIPS32.  However, as far as I know au1x00 CPUs don't suffer from this problem.
Anybody knows for sure?

You could try to use the two phase cache flushing (such as the one used
by tx47xx and also see an earlier related discussion thread) and see if
the problem goes away.

Jun

---

From: Pete Popov <[email-masked]>
Date: Thu, 05 Aug 2004 11:13:16 -0700

Pete Popov wrote:

> Could be a hardware memory glitch. I would use kgdb to put a 
> breakpoint there and see what the data in memory looks like when this 
> happens -- look for memory corruption, etc.

Another thought, what other stress testing have you done on your board? 
Does it complete a full native kernel compile without crashing?

Pete

>> quite rarely ) happened after a few >hours to a day of multiple 
>> telnet session use.
>>

---

From: G H <[email-masked]>
Date: Thu, 5 Aug 2004 13:16:43 -0700 (PDT)

At the moment I don't have the board set up for using kgdb and it's complicated by the fact that we only have one serial console port. But I am looking into setting it up for kgdb now.
 
As far as stressing the system, it doesn't have enough resources ( disk space ) to be able to compile the kernel, but we did write a simple program that would stress the system by spawning multiple threads, each one performing floating point calculations. With this test, top reported a load average of over 400 and we have seen no failure so far.

Pete Popov <[email-masked]> wrote:

Pete

> quite rarely ) happened after a few >hours to a day of multiple telnet 

---------------------------------
Do you Yahoo!?
Yahoo! Mail Address AutoComplete - You start. We finish.

---

From: G H <[email-masked]>
Date: Thu, 5 Aug 2004 13:25:30 -0700 (PDT)

I was also thinking that maybe it could be related to the MIPS32 instruction cache flushing routine, so I tried applying the patch Ralf posted. The system still functioned OK for me, but one other in my group had their board lock up hard ( no oops produced ), and when I asked Ralf if that patch was ready for applying to CVS, he said it needed to be reworked before doing that. As a result we didn't follow up too closely on that avenue of investigation.
 
So basically what I am concluding from the responses so far , is that do_ri should NEVER occur in blast_icache32() and for it to do so, it could be either a hardware problem, or possibly the MIPS32 icache flushing problem.
Anyone agree / disagree ?
 
Jun Sun <[email-masked]> wrote:

		
---------------------------------
Do you Yahoo!?
Yahoo! Mail - You care about security. So do we.

---

From: Pete Popov <[email-masked]>
Date: Thu, 05 Aug 2004 15:31:33 -0700

G H wrote:

> At the moment I don't have the board set up for using kgdb and it's 
> complicated by the fact that we only have one serial console port. But 
> I am looking into setting it up for kgdb now.

A single serial port will work fine.

>  
> As far as stressing the system, it doesn't have enough resources ( 
> disk space ) to be able to compile the kernel, but we did write a 
> simple program that would stress the system by spawning multiple 
> threads, each one performing floating point calculations. With this 
> test, top reported a load average of over 400 and we have 
> seen no failure so far.

You can use an NFS mounted root fs to do native builds, assuming you 
have a native toolchain. That's an excellent stress test.

Pete

>
> */Pete Popov <[email-masked]>/* wrote:
>
>     G H wrote:
>
>     > Can anyone think of any possible scenario where do_ri could
>     occur in
>     > blast_icache32() ??
>     >
>     breakpoint
>     > end while running >multiple telnet sessions t! o it. It has
>     sometimes (
>     telnet
> ------------------------------------------------------------------------
> Do you Yahoo!?
> Yahoo! Mail Address AutoComplete 
> <http://us.rd.yahoo.com/mail_us/taglines/aac/*http://promotions.yahoo.com/new_mail/static/ease.html> 
> - You start. We finish.