Subject: SB1250 locking up in init on current 2.6.16 kernel

From: Larry Stefani <[email-masked]>
Date: Mon, 24 Mar 2008 07:00:15 -0700 (PDT)

Hi,

I've been trying to upgrade from 2.6.16.18 to
2.6.16.60, but am seeing a hard lockup right before
"INIT: version 2.78 booting" on my SB1250-based board.

I found a related discussion on the Debian mailing
list:

http://groups.google.com/group/linux.debian.bugs.dist/browse_thread/thread/b7159ee25106c7f9

However, after applying Thiemo's patch to mark pages
tainted by PIO IDE as dirty, the lockup still occurs.

I narrowed the file changes to

     arch/mips/mm/c-sb1.c
     arch/mips/mm/cache.c
     arch/mips/mm/init.c
     include/asm-mips/cache-flush.h
     include/asm-mips/page.h

between 2.6.16.27 and 2.6.16.29.  There was no
2.6.16.28 tarball posted on linux-mips.org, so I
basically brought .27 to .29 until I found the
offending files.

Is anyone running a 2.6.16 kernel (after 2.6.16.27) on
a SB1250-based board?

Thanks,
Larry Stefani
[email-masked]

      ____________________________________________________________________________________
Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

---

From: Ralf Baechle <[email-masked]>
Date: Mon, 24 Mar 2008 20:33:11 +0000

> tainted by PIO IDE as dirty, the lockup still occurs.

It's a bug which should be fixed but nevertheless I can highly recommend
something like a SiliconImage SATA controller - the onboard PIO PATA
controller is so slow.

> basically brought .27 to .29 until I found the

I've pushed the tag again so now there is a tarball.

If you need to track something like this you're probably best with
git bisect which should bring you right to the offending commit.

> Is anyone running a 2.6.16 kernel (after 2.6.16.27) on

Later kernels do run on bcm1480 which is close enough.

  Ralf

---

From: Thiemo Seufer <[email-masked]>
Date: Tue, 25 Mar 2008 17:12:29 +0000

> basically brought .27 to .29 until I found the

Please learn about git bisect (git-bisect start; git-bisect good;
git-bisect bad) and use it to isolate the offending commit.

Thiemo

---

From: Larry Stefani <[email-masked]>
Date: Tue, 25 Mar 2008 06:05:35 -0700 (PDT)

Hello Ralf,

Thank you very much for the quick response.  I know
you must be terribly busy with the server move, so I
won't take up too much more of your time.

I did find an old (possibly related) discussion you
were involved in:

http://www.linux-mips.org/archives/linux-mips/2006-09/msg00194.html

I can't tell from the thread whether it was a problem
seen on 2.6.16.29, but that might have been tip at the
time.

> something like a SiliconImage SATA controller - the

I understand, but changing that is not an option for
me today.

> I've pushed the tag again so now there is a tarball.

Thanks.  I thought something was terribly wrong with
.28 for it to be skipped.  

> git bisect which should bring you right to the

I probably should have used that approach instead of
diffing .27 and .29 and narrowing the file list by
hand.  As it was, there were changes to non-MIPS
platforms and devices I'm not using so those were easy
to apply to .27.  Also, there were many file changes
for MT SMP support, and *most* (but not all) of those
changes were wrapped with conditional compiles, so
those were also easy to apply.  I knew once I got to
these five files I was in some interesting code that
could point to the problems I'm seeing.

> Later kernels do run on bcm1480 which is close

By "later kernels", do you mean 2.6.16.60 or different
kernel branches?

Perhaps, but I'm seeing identical failures in .29 and
.60, although that could be misleading.  I do see
significant SMP-related changes in c-sb1.c between .29
and .60, and I am running in SMP.  I wish there was an
easy way to know whether it's in the same code.

Anyway, in the interest of time I may revert to .27
which appears to work.  I don't need any of the MT SMP
related changes that followed, and perhaps I can live
without the enhancements between .27 and .60 for now.

Thanks,
Larry Stefani
[email-masked]

      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs

---

From: Thiemo Seufer <[email-masked]>
Date: Tue, 25 Mar 2008 15:37:47 +0000

Thiemo

---

From: Larry Stefani <[email-masked]>
Date: Wed, 26 Mar 2008 13:52:42 -0700 (PDT)

Hi Ralf,

I used git bisect and narrowed the lockup to the
"[MIPS] Retire flush_icache_page from mm use." patch
(see git results below).  This is consistent with my
earlier testing and what Thiemo reported March 3 on
the linux.debian.kernel list.  I tried his patch (mark
pages tainted by PIO IDE as dirty) on 2.6.16.60, but
it didn't prevent the lockup.

Regards,
Larry Stefani
[email-masked]

======================================================
/testkernels/linux> git bisect bad
Bisecting: 0 revisions left to test after this
[3a57f2ad7436d27dfa90717921b921fc3a168504] [MIPS]
Retire flush_icache_page from mm use.
/testkernels/linux> git bisect bad
3a57f2ad7436d27dfa90717921b921fc3a168504 is first bad
commit
commit 3a57f2ad7436d27dfa90717921b921fc3a168504
Author: Ralf Baechle <[email-masked]>
Date:   Sat Aug 12 16:40:08 2006 +0100

    [MIPS] Retire flush_icache_page from mm use.

    On the 34K the redundant cache operations were
causing excessive stalls
    resulting in realtime code running on the second
VPE missing its deadline.
    For all other platforms this patch is just a
significant performance
    improvment as illustrated by below benchmark
numbers.

    Processor, Processes - times in microseconds -
smaller is better
   
------------------------------------------------------------------------------
    Host                 OS  Mhz null null      open
slct sig  sig  fork exec sh
                                 call  I/O stat clos
TCP  inst hndl proc proc proc
    --------- ------------- ---- ---- ---- ---- ----
---- ---- ---- ---- ---- ----
    25Kf      2.6.18-rc4     533 0.49 1.16 7.57 33.4
30.5 1.34 12.4 5497 17.K 54.K
    25Kf      2.6.18-rc4-p   533 0.49 1.16 6.68 23.0
30.7 1.36 8.55 5030 16.K 48.K
    4Kc       2.6.18-rc4      80 4.21 15.0 131. 289.
261. 16.5 258. 18.K 70.K 227K
    4Kc       2.6.18-rc4-p    80 4.34 13.1 128. 285.
262. 18.2 258. 12.K 52.K 176K
    34Kc      2.6.18-rc4      40 5.01 14.0 61.6 90.0
477. 17.9 94.7 29.K 108K 342K
    34Kc      2.6.18-rc4-p    40 4.98 13.9 61.2 89.7
475. 17.6 93.7 8758 44.K 158K
    BCM1480   2.6.18-rc4     700 0.28 0.60 3.68 5.92
16.0 0.78 5.08 931. 3163 15.K
    BCM1480   2.6.18-rc4-p   700 0.28 0.61 3.65 5.85
16.0 0.79 5.20 395. 1464 8385
    TX49-16K  2.6.18-rc3     197 0.73 2.41 19.0 37.8
82.9 2.94 17.5 4438 14.K 56.K
    TX49-16K  2.6.18-rc3-p   197 0.73 2.40 19.9 36.3
82.9 2.94 23.4 2577 9103 38.K
    TX49-32K  2.6.18-rc3     396 0.36 1.19 6.80 11.8
41.0 1.46 8.17 2738 8465 32.K
    TX49-32K  2.6.18-rc3-p   396 0.36 1.19 6.82 10.2
41.0 1.46 8.18 1330 4638 18.K

    Original patch by me with enhancements by Atsushi
Nemoto.

    Signed-off-by: Ralf Baechle <[email-masked]>
    Signed-off-by: Atsushi Nemoto
<[email-masked]>
    (cherry picked from
4bbd62a93a1ab4b7d8a5b402b0c78ac265b35661 commit)

:040000 040000
6b4aeaca51e1f8974394f5c85c36cfcbf40984c9
2af41ed07f6c83962f9a4871812adb6f7c0dbee7 M      arch
:040000 040000
c2fb4be66f03aed2a1746d30f0e17922086eefdd
ab8cb20f7da9b97e72742946adfb5ab927dc777a M     
include
=====================================================

---

From: Thiemo Seufer <[email-masked]>
Date: Fri, 28 Mar 2008 13:43:17 +0000

> it didn't prevent the lockup.

ISTR I got 2.6.16.60 to work by always enabling the cache flush in
ide.h (it is currently only run to clean out aliases). Keep in mind
that this is a crude workaround on top of other cache code hacks for
the SB-1.

Thiemo

---

From: Larry Stefani <[email-masked]>
Date: Fri, 28 Mar 2008 09:38:11 -0700 (PDT)

Hi Thiemo,

I applied your patch (from
http://www.linux-mips.org/archives/linux-mips/2008-03/msg00001.html)
on 2.6.16.60, and also patched arch/mips/mm/c-sb1.c to
remove:

          local_flush_data_cache_page = (void *)
sb1_nop;

in order to compile after your changes to cache.c and
cacheflush.h.  However, this did not work on my board,
and I experienced the same lockup as before.

>>Keep in mind that this is a crude workaround on top
of other cache code hacks for the SB-1.

What other "cache code hacks for SB-1"?  Are there
additional changes required to 2.6.16.60 to make SB1
work properly?  Did you post those hacks somewhere?

Thanks,
Larry

---

From: Thiemo Seufer <[email-masked]>
Date: Fri, 28 Mar 2008 17:11:46 +0000

> and I experienced the same lockup as before.

Stick with the original 2.6.16.60 code but try to remove the

   if (cpu_has_dc_aliases) {

conditional in ide.h _without_ using my patch. This is what made
it boot for me.

> work properly?  Did you post those hacks somewhere?

No, what I meant to say is that the old sb-1 cache code isn't quite
the most trustibe code, it had some holes which were papered over
by doing more cache flushes than necessary.

Thiemo

---

From: Larry Stefani <[email-masked]>
Date: Sun, 30 Mar 2008 19:02:12 -0700 (PDT)

Hi Thiemo,

I went back to 2.6.16.60, reverted your patch changes,
then removed the

conditional expression in three places in
include/asm-mips/mach-generic/ide.h.

Unfortunately, I get the same lockup as before.

- Larry