Subject: NTP, once more

From: [email-masked]
Date: Sat, 7 Mar 1998 01:03:49 +0100

Sigh,

it's sometimes useful not to cleanup mailfolders.  The following patch
provided by Mark Salter ages ago should fix the problem.  It definately
fixes a serious problem and makes the kernel print warnings in case of
another; I just haven't tested if this fixes xntpd.

  Ralf

>From [email-masked]  Thu Aug 14 00:15:32 1997
Date: Wed, 13 Aug 1997 17:09:32 -0500
From: Mark Salter <[email-masked]>
To: [email-masked]
Subject: clock skew and ethernet timeouts
Sender: [email-masked]
Status: RO

I've been taking a look at the problem with frequent ethernet
transmit timeouts. In addition to the timeouts, I've also seen
an occasional duplicate packet being sent in response to pings
coming from another machine. I'm debugging somewhat in the dark
because I don't have documentation on the indy's DMA controller
although the sgihpc.h file has been helpful.

I also noticed that the linux time of day clock falls behind
real time whenever these timeouts occur. I decided to take a
look at this side of the problem and discovered that interrupts
are being turned off for extended periods of time. I modified
the timer interrupt handler to print a message if it detects
a missed system tick. Sure enough, every ethernet timeout is
accompanied by a message coming from the timer interrupt. The
message indicates that the timer interrupt was held off for
as much as 45ms!

Here's the change I made to indy_timer_interrupt() in indy_timers.c:

	/* Ack timer and compute new compare. */
#if 0
	r4k_cur = (read_32bit_cp0_register(CP0_COUNT) + r4k_offset);
#else
	count = read_32bit_cp0_register(CP0_COUNT);
	if ((count - r4k_cur) >= r4k_offset) {
		printk("missed heartbeat: r4k_cur[0x%x] count[0x%x]\n",
		       r4k_cur, count);
		r4k_cur = count + r4k_offset;
	}
	else
	    r4k_cur += r4k_offset;
#endif

The original code which calculates the next value for the CP0_COMPARE
register introduces skew by basing it on the current value of the 
CP0_COUNT register rather than the previous CP0_COMPARE value. But as
it turns out, that little bit of skew is preferable to the large 
skew that would result whenever the timer interrupt is held off too
long.

Anyway, I'm going to be away from the office until 19 August, so I'll
pick it up then unless someone else finds it first. It seems likely
that the enet timeouts are the result of interrupts being turned off
too long, but if that's not the case, can someone point me to some
documentation for the indy's dma controller? I have some suspicions
of race conditions when setting up new DMA buffers, but I'd like to
have a document before I try various fixes.

--
Mark Salter
[email-masked]

---

From: "William J. Earl" <[email-masked]>
Date: Fri, 6 Mar 1998 16:18:11 -0800

[email-masked] writes:
...
 > Subject: clock skew and ethernet timeouts
...
 > 		r4k_cur = count + r4k_offset;
...

     I have not checked the source, but is the workaround for the R4000
count/compare bug in read_32bit_cp0_register(CP0_COUNT)?  If not,
every once in a while you will lose count/compare interrupts for about
84 seconds on a 100 MHZ R4000.  Note that this bug does not apply
to any other processor.  The bug is that the processor fails to
signal an interrupt if $count is read at exactly the cycle that
$count == $compare.  Since $compare is commonly not that far ahead
of $count when one is using it for timer interrupts, this is far more
likely than it might seem at first thought.  

      The workaround I did in IRIX, for R4000 only, was to keep a
shadow copy of $compare, and, with interrupts disabled, fetch $count
and see if it was very close to the shadow value of $compare (to allow
for the possibility that "close" counted in R4000 bugs as well as in
horseshoes :-) ), and, if so, whether the timer interrupt was pending
in $sr.  If the timer was not pending, I would then enter a loop where
it would set $compare a little ahead of the current $count, and then
wait a few cycles for the interrupt to turn on in $sr, retrying with a
larger window if I missed the window on the first try.  It is
important to have interrupts off (mfc0 $a0,C0_SR; mtc0 zero,C0_SR)
while doing this workaround to avoid all sorts of races with higher
level code and also minimize unexpected cache misses (which greatly
affect the timing).