Subject: Re: Strange, strange occurence

From: "S C" <[email-masked]>
Date: Mon, 12 Jul 2004 21:23:20 +0000

And thinking about it a little more, I might as well clarfy my understanding 
while we're on the topic.

Here's a code snippet from r4k_tlb_init() in arch/mips/mm/c-r4k.c

memcpy((void *)KSEG0, &except_vec0_r4000, 0x80);
flush_icache_range(KSEG0, KSEG0 + 0x80);

So my understanding is that the TLB exception handler is being copied to the 
right memory location, and just in case some other TLB exception handler 
(YAMON's presumably) is residing in I-cache, to flush ( hit invalidate) it. 
Is this correct?

Shouldn't there be code to writeback_invalidate the exception handler from 
the data cache ? Presumably the memcpy causes the TLB handler to lodge 
itself in the D cache till it is moved to RAM (either explicitly or when 
some other lines replace the cache lines where the handler is), right?

Thanks in advance for helping me understand the issue here.

Regards,
-Steve.

>From: Ralf Baechle <[email-masked]>
>To: S C <[email-masked]>
>CC: [email-masked]
>Subject: Re: Strange, strange occurence
>Date: Sat, 10 Jul 2004 12:04:12 +0200
>
>On Fri, Jul 09, 2004 at 06:50:00PM +0000, S C wrote:
>
> > Using MontaVista Linux 3.1 on a Toshiba RBTx4938 board. Using YAMON, 
>when I
> > download the kernel via the debug ethernet port it runs fine. If I 
>download
> > the kernel via the Tx4938 inbuilt ethernet controller, it crashes!
>
>If you're using a Montavista kernel you should report to Montavista.  We
>don't have the source so any comment here is speculation.
>
> > The crash is occuring inside the function r4k_flush_icache_range().
> >
> > I tried 'flush -i' and 'flush -d' on YAMON after the download but before
> > the 'go', but that didn't help. I also tried completely disabling caches
> > and loading/running uncached, but it gave the same error.
> >
> > Now, the final twist! Using an ICE, I set a breakpoint at the
> > r4k_flush_icache_range function. Then I loaded the kernel as usual, ran 
>it
> > with the ICE, stepped through a few instructions inside the
> > r4k_flush_icache_range function and then did a 'cont'. The kernel now
> > booted fine!
>
>As already pointed out by the other poster Niels Sterrenburg using a
>debugger unavoidably changes the state of the system to be debugged.
>
>For at least some of the TX49xx processors there is a problem under certain
>circumstances if a flush of an I-cache line flushes that cache instruction
>itself.  Make sure you're not getting hit by that one.
>
>   Ralf

_________________________________________________________________
Check out the latest news, polls and tools in the MSN 2004 Election Guide! 
http://special.msn.com/msn/election2004.armx

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Mon, 12 Jul 2004 23:48:31 +0200

> Thanks in advance for helping me understand the issue here.

Your intuition is correct, and the code in r4k_tlb_init() does look scary.
But at least in the linux-mips CVS tree, flush_icache_range() tests to see
if "cpu_has_ic_fills_f_dc" (CPU has Icache fills from Dcache, I presume)
is set, and if it isn't, it pushes the specified range out of the Dcache before
flushing the Icache.  I would speculate that either your c-r4k.c is out of
sync with yout tlb-r4k.c, or you erroneously have cpu_has_ic_fills_f_dc
set.

            Regards,

            Kevin K.

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Tue, 13 Jul 2004 00:25:37 +0200

> sync with yout tlb-r4k.c, or you erroneously have cpu_has_ic_fills_f_dc

Hmm.  On closer examination, there *is* a bug in the current r4k_flush_icache_range(),
in that it computes its cache flush loop for the I-cache based on the D-cache line size.
Those line sizes are *usually* the same.  By any chance are they different for the
TX49 family?  If the icache line is longer than the dcache line, there should be no
functional problem, just some wasted cycles.  But if the dcache line were, say, 
twice the length of the Icache line, only half of the icache lines would be invalidated...

            Regards,

            Kevin K.

---

From: Ralf Baechle <[email-masked]>
Date: Tue, 13 Jul 2004 01:00:46 +0200

On Mon, Jul 12, 2004 at 09:23:20PM +0000, S C wrote:

> And thinking about it a little more, I might as well clarfy my 
> understanding while we're on the topic.
> 
> So my understanding is that the TLB exception handler is being copied to 
> the right memory location, and just in case some other TLB exception 
> handler (YAMON's presumably) is residing in I-cache, to flush ( hit 
> invalidate) it. Is this correct?
> 
> the data cache ?

flush_icache_range() does that where necessary.

> Presumably the memcpy causes the TLB handler to lodge 
> itself in the D cache till it is moved to RAM

Correct.  All MIPS processors [1] do their primary I-cache refills from
second or third level cache or memory - whatever hits first.  If code
was changed a flush of the data cache is required so the I-cache can
actually fetch the new data and because old stale code might still be
in the I-cache an I-cache flush is also required.

> (either explicitly or when 

  Ralf

[1] Exceptions are the R4000/R4400 SC and MC versions in a split S-cache
    configuration where the primary I-cache is refilled from the secondary
    I-cache which mean this flush has to flush the secondary data cache
    back to memory also.  Linux doesn't support this configuration because
    no known system uses it iow it's only a theoretically possible config.

    The second exception are the AMD MIPS32 processors where the I-cache
    is snooping the D-cache and therefore the D-cache flush can is
    unnecessary.

    The third exception are R1x000 in SMP configurations where I-caches
    snoop remote stores so coherency doesn't need any maintenance in
    software at all.  Only the I-cache of the CPU that did modify the code
    needs flushing.

---

From: Ralf Baechle <[email-masked]>
Date: Tue, 13 Jul 2004 01:11:38 +0200

> Your intuition is correct, and the code in r4k_tlb_init() does look scary.

Not scarry at all.  flush_icache_range() has to do whatever is needed to
maintain I-cache coherency for the range passed in as the argument.  And
I don't think we should really have to deal with all the complicated
details of cache maintenance in a function like r4k_tlb_init().

> if "cpu_has_ic_fills_f_dc" (CPU has Icache fills from Dcache, I presume)

Right.  Cpu_has_ic_fills_f_dc is only non-zero for the AMD processors
where the I-cache is refilled from the D-cache.  For typical kernel
configurations The definition of cpu_has_ic_fills_f_dc is a constant so
the compiler can optimize this further.

  Ralf

---

From: Ralf Baechle <[email-masked]>
Date: Tue, 13 Jul 2004 01:13:59 +0200

> twice the length of the Icache line, only half of the icache lines would be invalidated...

Whoops.  Fixing ...

  Ralf