Subject: Kernel freezes in r4k_flush_icache_range() with CONFIG_CPU_MIPS32_R2

From: Maxime Bizon <[email-masked]>
Date: Tue, 20 Dec 2005 03:57:18 +0100

Hello all,

I'm porting linux for a board with a R4KECr2. So far I was using
CONFIG_CPU_MIPS32_R1 and the kernel (2.6.14) was running well.

But I'm unable to get it working with CONFIG_CPU_MIPS32_R2, it freezes
somewhere in trap_init, more exactly in r4k_flush_icache_range().

Here is a snippet from the generated assembly code:

00002624 <r4k_flush_icache_range>:
    2624:       27bdffd0        addiu   sp,sp,-48
    2628:       3c020000        lui     v0,0x0
    262c:       afb30024        sw      s3,36(sp)
    2630:       afb20020        sw      s2,32(sp)
    2634:       afbf0028        sw      ra,40(sp)
    2638:       afb1001c        sw      s1,28(sp)
    263c:       afb00018        sw      s0,24(sp)
[...]
    26a0:       bc900000        cache   0x10,0(a0)
    26a4:       1464fffd        bne     v1,a0,269c <r4k_flush_icache_range+0x78>
    26a8:       3c020000        lui     v0,0x0
    26ac:       2442277c        addiu   v0,v0,10108   <----------
    26b0:       00400408        jr.hb   v0
    26b4:       00000000        nop
[...]
    272c:       8c43000c        lw      v1,12(v0)
    2730:       0060f809        jalr    v1
    2734:       00000000        nop
    2738:       3c020000        lui     v0,0x0
    273c:       2442277c        addiu   v0,v0,10108   <----------
    2740:       00400408        jr.hb   v0
    2744:       00000000        nop
    2748:       8fbf0028        lw      ra,40(sp)
    274c:       8fb30024        lw      s3,36(sp)
    2750:       8fb20020        lw      s2,32(sp)
    2754:       8fb1001c        lw      s1,28(sp)
    2758:       8fb00018        lw      s0,24(sp)
    275c:       03e00008        jr      ra
    2760:       27bd0030        addiu   sp,sp,48
    2764:       3c020000        lui     v0,0x0
    2768:       8c430018        lw      v1,24(v0)
    276c:       0060f809        jalr    v1
    2770:       00000000        nop
    2774:       0800099c        j       2670 <r4k_flush_icache_range+0x4c>
    2778:       3c030000        lui     v1,0x0

0000277c <r4k_dma_cache_inv>:
[...]

At offset 0x26ac and 0x273c, we can see that instruction_hazard() got
duplicated due to inlining, and that the jr.hb is going to send us to
10108 (0x277C), outside the function...

The only way I managed to get a good value in v0 was by using -O0 and
making r4k_flush_icache_range return int.

Now I'm really not familiar with gcc inline assembly so I don't know if
this is a compiler bug or if something is missing in
instruction_hazard().

# mipsel-linux-gcc -v
Using built-in specs.
Target: mipsel-linux-uclibc
Configured
with: /home/work/buildroot/toolchain_build_mipsel/gcc-4.0.2/configure
--prefix=/opt/toolchains/mipsel-uclibc-0.9.28-gcc-4.0.2
--build=i386-pc-linux-gnu --host=i386-pc-linux-gnu
--target=mipsel-linux-uclibc --enable-languages=c,c++ --enable-shared
--disable-__cxa_atexit --enable-target-optspace --with-gnu-ld
--disable-nls --enable-threads --enable-multilib
Thread model: posix
gcc version 4.0.2
#

Thanks,

-- 
Maxime

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Tue, 20 Dec 2005 11:47:42 +0000 (GMT)

> 10108 (0x277C), outside the function...

 FYI, GCC 3.4.4 produces the following code which is clearly wrong:

	.align	2
	.align	3
	.ent	r4k_flush_icache_range
	.type	r4k_flush_icache_range, @function
r4k_flush_icache_range:
	.set	nomips16
	.frame	$sp,56,$31		# vars= 16, regs= 5/0, args= 16, gp= 0
	.mask	0x800f0000,-8
	.fmask	0x00000000,0
[...]
	lui	$2,%hi($L506)	 # 239	movsi_internal/2	[length = 4]
	addiu	$2,$2,%lo($L506)	 # 240	*lowsi	[length = 4]
#APP
		jr.hb	$2					

#NO_APP
	lw	$31,48($sp)	 # 304	movsi_internal/4	[length = 4]
	lw	$19,44($sp)	 # 305	movsi_internal/4	[length = 4]
	lw	$18,40($sp)	 # 306	movsi_internal/4	[length = 4]
	lw	$17,36($sp)	 # 307	movsi_internal/4	[length = 4]
	lw	$16,32($sp)	 # 308	movsi_internal/4	[length = 4]
	.set	noreorder
	.set	nomacro
	j	$31	 # 310	return_internal	[length = 4]
	addiu	$sp,$sp,56	 # 309	addsi3_internal/2	[length = 4]
	.set	macro
	.set	reorder

$L510:
	lui	$2,%hi(r4k_blast_icache)	 # 181	movsi_internal/2	[length = 4]
	lw	$3,%lo(r4k_blast_icache)($2)	 # 182	movsi_internal/4	[length = 4]
	jal	$3	 # 183	call_internal/1	[length = 8]
	lui	$2,%hi($L506)	 # 313	movsi_internal/2	[length = 4]
	addiu	$2,$2,%lo($L506)	 # 314	*lowsi	[length = 4]
#APP
		jr.hb	$2					

#NO_APP
	lw	$31,48($sp)	 # 316	movsi_internal/4	[length = 4]
	lw	$19,44($sp)	 # 317	movsi_internal/4	[length = 4]
	lw	$18,40($sp)	 # 318	movsi_internal/4	[length = 4]
	lw	$17,36($sp)	 # 319	movsi_internal/4	[length = 4]
	lw	$16,32($sp)	 # 320	movsi_internal/4	[length = 4]
	.set	noreorder
	.set	nomacro
	j	$31	 # 322	return_internal	[length = 4]
	addiu	$sp,$sp,56	 # 321	addsi3_internal/2	[length = 4]
	.set	macro
	.set	reorder

$L508:
	lui	$2,%hi(r4k_blast_dcache)	 # 67	movsi_internal/2	[length = 4]
	lw	$3,%lo(r4k_blast_dcache)($2)	 # 68	movsi_internal/4	[length = 4]
	jal	$3	 # 69	call_internal/1	[length = 8]
	.set	noreorder
	.set	nomacro
	b	$L512	 # 334	jump	[length = 4]
	lui	$3,%hi(icache_size)	 # 170	movsi_internal/2	[length = 4]
	.set	macro
	.set	reorder

$L506:
	.end	r4k_flush_icache_range

Please file a bug report at: "http://gcc.gnu.org/bugzilla/".

  Maciej

---

From: Maxime Bizon <[email-masked]>
Date: Thu, 22 Dec 2005 03:31:31 +0100

> Please file a bug report at: "http://gcc.gnu.org/bugzilla/".

Same with 3.3 and 3.2...

I'm really not familiar with inline assembly so I would appreciate that
any gcc guru here confirm instruction_hazard() code is correct before I
(or he) submit the bug report.

As the bug seems to be in all gcc versions, I guess we should find a
workaround... I changed the code to use an asm label instead of the C
label and the bug disappeared. But I'm not sure my changes are correct
for any platform other than mine...

Could anyone with the right skills help me to write a valid workaround
please ?

Here is what I have:

__asm__ __volatile__(
        "lui $2,1f\n"
        "addiu $2,1f\n"
        "jr.hb $2\n1:":: );

Thanks,

-- 
Maxime

---

From: Ralf Baechle <[email-masked]>
Date: Thu, 22 Dec 2005 12:10:23 +0100

> for any platform other than mine...

We ran into similar problems in the past.  It only seems to be a matter
of the code nearby and the exact options to trigger it.

The alternative is manually loading the address using la / dla which
defeats gcc's splitting of address loading.  And having to use different
code for 32-bit and 64-bit sucks.

> Could anyone with the right skills help me to write a valid workaround

Wrong, breaks 64-bit and uses $2 without telling gcc about it.

  Ralf