Subject: semaphore woes in 2.6, 32bit

From: Jun Sun <[email-masked]>
Date: Fri, 7 May 2004 18:10:31 -0700

I got a bunch of segfaults which are due to HAS_LLSCD cpu operating
on a semaphore which is aligned along 4-byte boundary instead of the
desired 8-byte boundary.

I traced down one such place at serial/serial_core.c:

int uart_register_driver(struct uart_driver *drv)
{
        struct tty_driver *normal = NULL;
        int i, retval;

        BUG_ON(drv->state);

        /*
         * Maybe we should be using a slab cache for this, especially if
         * we have a large number of ports to handle.
         */
        drv->state = kmalloc(sizeof(struct uart_state) * drv->nr, GFP_KERNEL);
...

where drv->state contains a semaphore variable, but apparently kmalloc() only
give 4-byte boundary alignment.

There are many other faults, which I did not bother to trace down.
Simply removing CPU_HAS_LLSCD makes the problem go away, which probably indicates
they are all of the same nature.

I wonder why this problem only shows up now while it did not show up
earlier when we introduced the new up().  Perhaps kmalloc() always
returns 8-byte aligned blocks?

I can't think of an immediate and good fix.  Hopefully someone else smarter
than me can find a solution before I come back to it on Monday.  :)

Jun

---

From: Ralf Baechle <[email-masked]>
Date: Sat, 8 May 2004 09:18:22 +0200

> on a semaphore which is aligned along 4-byte boundary instead of the

Dare to give a complete version number?  I've dumped 2.4 on all my systems
months ago and never have seen this problem except with slab debugging
enabled - but that side effect of slab debugging is known since years.

  Ralf

---

From: Jun Sun <[email-masked]>
Date: Sat, 8 May 2004 22:48:06 -0700

> enabled - but that side effect of slab debugging is known since years.

Kernel is yesterday's CVS. gcc is 3.3.1.  config is ddb5477.  No 
additional patch.  See below.

In any case if you look at the uart code you should see there
is a problem already.  'state' is allocated through kmalloc() which only
gives 4-byte alignement.  The only puzzling thing is that why this
did not show up before.  Maybe kmalloc() was giving 8-byte aligned block?

VERSION = 2
PATCHLEVEL = 6
SUBLEVEL = 6
EXTRAVERSION =-rc3
NAME=Zonked Quokka

[jsun@orion linux]$ mipsel-linux-gcc -v
Reading specs from /opt/mvl-installs/pe040201/montavista/pro/devkit/mips/fp_le/bin/../lib/gcc-lib/mipsel-hardhat-linux/3.3.1/specs
Configured with: ../configure --host=i686-pc-linux-gnu --target=mipsel-hardhat-linux --prefix=/opt/montavista/devkit/mips/fp_le --exec-prefix=/opt/montavista/devkit/mips/fp_le --bindir=/opt/montavista/devkit/mips/fp_le/bin --sbindir=/opt/montavista/devkit/mips/fp_le/sbin --sysconfdir=/opt/montavista/devkit/mips/fp_le/etc --datadir=/opt/montavista/devkit/mips/fp_le/share --includedir=/opt/montavista/devkit/mips/fp_le/include --libdir=/opt/montavista/devkit/mips/fp_le/lib --libexecdir=/opt/montavista/devkit/mips/fp_le/libexec --localstatedir=/opt/montavista/devkit/mips/fp_le/var --sharedstatedir=/opt/montavista/devkit/mips/fp_le/share --mandir=/opt/montavista/devkit/mips/fp_le/man --infodir=/opt/montavista/devkit/mips/fp_le/info --program-transform-name=s,^,mips_fp_le-, --enable-cross --with-sysroot=/opt/montavista/devkit/mips/fp_le/target --enable-shared --enable-languages=c,c++ --enable-__cxa_atexit --enable-threads=posix --disable-multilib --with-gxx-include-dir='$'{gcc_tooldir}/../target/usr/include/c++/3.3.1
Thread model: posix
gcc version 3.3.1 (MontaVista 3.3.1-3.0.10.0300532 2003-12-24)
 
Jun

---

From: Geert Uytterhoeven <[email-masked]>
Date: Sun, 9 May 2004 11:09:29 +0200 (MEST)

> did not show up before.  Maybe kmalloc() was giving 8-byte aligned block?

AFAIK, kmaloc() always[*] returns 8-byte (or higher, for archs that need it)
aligned blocks.

[*] But I did see reports of this being false if slab debugging was enabled.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email-masked]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

---

From: Ralf Baechle <[email-masked]>
Date: Sun, 9 May 2004 14:57:50 +0200

> AFAIK, kmaloc() always[*] returns 8-byte (or higher, for archs that need it)

We got tripped by a change in 2.6.6-rc2.  Before that change the kmalloc
slab caches were being created with SLAB_HWCACHE_ALIGN which results in
L1_CACHE_SHIFT alignment for allocations of L1_CACHE_SHIFT for slab caches
that are at least that size.  For the sake of S390 this behaviour was
changed; new it defaults to BYTES_PER_WORD alignment which is four bytes.

Fixed by defining ARCH_KMALLOC_MINALIGN as 8.

  Ralf

Index: include/asm-mips/cache.h
===================================================================
RCS file: /home/cvs/linux/include/asm-mips/cache.h,v
retrieving revision 1.16
diff -u -r1.16 cache.h
--- include/asm-mips/cache.h	10 Oct 2003 20:37:35 -0000	1.16
+++ include/asm-mips/cache.h	9 May 2004 12:57:38 -0000
@@ -18,4 +18,6 @@
 #define SMP_CACHE_SHIFT		L1_CACHE_SHIFT
 #define SMP_CACHE_BYTES		L1_CACHE_BYTES
 
+#define ARCH_KMALLOC_MINALIGN	8
+
 #endif /* _ASM_CACHE_H */

---

From: Atsushi Nemoto <[email-masked]>
Date: Sun, 09 May 2004 22:56:37 +0900 (JST)

>>>>> On Sun, 9 May 2004 14:57:50 +0200, Ralf Baechle <[email-masked]> said:

ralf> We got tripped by a change in 2.6.6-rc2.  Before that change the
ralf> kmalloc slab caches were being created with SLAB_HWCACHE_ALIGN
ralf> which results in L1_CACHE_SHIFT alignment for allocations of
ralf> L1_CACHE_SHIFT for slab caches that are at least that size.  For
ralf> the sake of S390 this behaviour was changed; new it defaults to
ralf> BYTES_PER_WORD alignment which is four bytes.

ralf> Fixed by defining ARCH_KMALLOC_MINALIGN as 8.

Hmm, many drivers use kmalloc and pci_map_single for DMA buffer.  So
ARCH_KMALLOC_MINALIGN should be L1_CACHE_BYTES for non-coherent
system?

---
Atsushi Nemoto

---

From: Ralf Baechle <[email-masked]>
Date: Sun, 9 May 2004 18:48:35 +0200

> ARCH_KMALLOC_MINALIGN should be L1_CACHE_BYTES for non-coherent

No, those drivers are simply broken.  dma_get_cache_alignment() gives the
mimimum alignment and width for DMA mappings and that value is larger
than kmalloc alignment in almost all cases.

  Ralf

---

From: Atsushi Nemoto <[email-masked]>
Date: Mon, 10 May 2004 22:28:45 +0900 (JST)

>>>>> On Sun, 9 May 2004 18:48:35 +0200, Ralf Baechle <[email-masked]> said:
>> Hmm, many drivers use kmalloc and pci_map_single for DMA buffer.
>> So ARCH_KMALLOC_MINALIGN should be L1_CACHE_BYTES for non-coherent
>> system?

ralf> No, those drivers are simply broken.  dma_get_cache_alignment()
ralf> gives the mimimum alignment and width for DMA mappings and that
ralf> value is larger than kmalloc alignment in almost all cases.

I see.  Thank you for pointing out it.  I must learn 2.6 DMA API
quickly ...

---
Atsushi Nemoto

---

From: Ralf Baechle <[email-masked]>
Date: Mon, 10 May 2004 16:06:06 +0200

> I see.  Thank you for pointing out it.  I must learn 2.6 DMA API

This also applies to the 2.4 PCI DMA API.

  Ralf

---

From: Atsushi Nemoto <[email-masked]>
Date: Tue, 11 May 2004 22:53:05 +0900 (JST)

>>>>> On Mon, 10 May 2004 16:06:06 +0200, Ralf Baechle <[email-masked]> said:

>> I see.  Thank you for pointing out it.  I must learn 2.6 DMA API

ralf> This also applies to the 2.4 PCI DMA API.

But 2.4 PCI DMA API does not have dma_get_cache_alignment() (or
equivalent), or am I missing something?

---
Atsushi Nemoto

---

From: Ralf Baechle <[email-masked]>
Date: Tue, 11 May 2004 16:05:07 +0200

> equivalent), or am I missing something?

No, you don't; that was a problem in the 2.4 DMA API which got fixed
in 2.5.  For 2.4 using L1_CACHE_BYTES from <asm/cache.h> is a reasonable
alternative.

  Ralf