Subject: another 4kc machine check.

From: Greg Weeks <[email-masked]>
Date: Thu, 07 Apr 2005 10:06:01 -0400

This is the malta branch with the tlb patch from Ralf applied. The LTP test:

-bash-2.05b# shmem_test_06
shmem_test_06: IPC Shared Memory TestSuite program
 
        mykey to uniquely identify the shared memory segment 0x330b035a
 
...
       
        Detach from the segment using the shmdt subroutine
 
        Release shared memory

causes:

timesys-bsp login: Got mcheck at 802199bc
Cpu 0
$ 0   : 00000000 1000fc00 00000004 00000000
$ 4   : 80401d30 00020004 c0002034 803e5964
$ 8   : 803e598c 00000000 fffffffa 8299f470
$12   : 00000001 000000ff 00000000 00000000
$16   : 80400000 00020004 82d15720 82d15724
$20   : 60100000 803e5964 8337dd38 00000000
$24   : 00000000 8014e6a4
$28   : 82d3e000 82d3fe38 10021a80 8021d314
Hi    : d3c9c18f
Lo    : 7263fc03
epc   : 802199bc ipc_lock+0x14/0x54     Not tainted
ra    : 8021d314 shm_close+0x5c/0x11c
Status: 1020fc03    KERNEL EXL IE
Cause : 00800060
PrId  : 00018001
                
Kernel panic - not syncing: Caught Machine Check exception - caused by 
multiple matching entries in the TLB.
                            
To run this test I have a different config from the default malta.
[gweeks@tanith linux]$ diff -du arch/mips/configs/malta_defconfig .config
--- arch/mips/configs/malta_defconfig   2005-03-18 12:36:52.000000000 -0500
+++ .config     2005-04-07 09:42:47.000000000 -0400
@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
 # Linux kernel version: 2.6.12-rc1
-# Fri Mar 18 15:41:40 2005
+# Thu Apr  7 09:42:47 2005
 #
 CONFIG_MIPS=y
 
@@ -18,7 +18,7 @@
 CONFIG_LOCALVERSION=""
 CONFIG_SWAP=y
 CONFIG_SYSVIPC=y
-# CONFIG_POSIX_MQUEUE is not set
+CONFIG_POSIX_MQUEUE=y
 # CONFIG_BSD_PROCESS_ACCT is not set
 CONFIG_SYSCTL=y
 # CONFIG_AUDIT is not set
@@ -158,6 +158,7 @@
 # CONFIG_PAGE_SIZE_16KB is not set
 # CONFIG_PAGE_SIZE_64KB is not set
 CONFIG_CPU_HAS_PREFETCH=y
+CONFIG_BOARD_HAS_MEMCPY_PREFETCH_BUG=y
 # CONFIG_64BIT_PHYS_ADDR is not set
 # CONFIG_CPU_ADVANCED is not set
 CONFIG_CPU_HAS_LLSC=y
@@ -187,7 +188,7 @@
 # Executable file formats
 #
 CONFIG_BINFMT_ELF=y
-# CONFIG_BINFMT_MISC is not set
+CONFIG_BINFMT_MISC=y
 CONFIG_TRAD_SIGNALS=y
 
 #
@@ -394,8 +395,7 @@
 CONFIG_DM_SNAPSHOT=m
 CONFIG_DM_MIRROR=m
 CONFIG_DM_ZERO=m
-CONFIG_DM_MULTIPATH=m
-CONFIG_DM_MULTIPATH_EMC=m
+# CONFIG_DM_MULTIPATH is not set
 
 #
 # Fusion MPT device support
@@ -663,7 +663,7 @@
 CONFIG_NET_QOS=y
 CONFIG_NET_ESTIMATOR=y
 CONFIG_NET_CLS=y
-CONFIG_NET_CLS_BASIC=m
+# CONFIG_NET_CLS_BASIC is not set
 CONFIG_NET_CLS_TCINDEX=m
 CONFIG_NET_CLS_ROUTE4=m
 CONFIG_NET_CLS_ROUTE=y
@@ -1006,10 +1006,13 @@
 CONFIG_PROC_FS=y
 CONFIG_PROC_KCORE=y
 CONFIG_SYSFS=y
-# CONFIG_DEVFS_FS is not set
+CONFIG_DEVFS_FS=y
+# CONFIG_DEVFS_MOUNT is not set
+# CONFIG_DEVFS_DEBUG is not set
 CONFIG_DEVPTS_FS_XATTR=y
 CONFIG_DEVPTS_FS_SECURITY=y
-# CONFIG_TMPFS is not set
+CONFIG_TMPFS=y
+# CONFIG_TMPFS_XATTR is not set
 # CONFIG_HUGETLB_PAGE is not set
 CONFIG_RAMFS=y
 
@@ -1138,7 +1141,7 @@
 CONFIG_CRYPTO_SHA256=m
 CONFIG_CRYPTO_SHA512=m
 CONFIG_CRYPTO_WP512=m
-CONFIG_CRYPTO_TGR192=m
+# CONFIG_CRYPTO_TGR192 is not set
 CONFIG_CRYPTO_DES=m
 CONFIG_CRYPTO_BLOWFISH=m
 CONFIG_CRYPTO_TWOFISH=m
[gweeks@tanith linux]$

---

From: Greg Weeks <[email-masked]>
Date: Fri, 08 Apr 2005 10:45:48 -0400

> -bash-2.05b# shmem_test_06

This test does some mucking with shmat at fixed addresses that might not 
be correct for mips. I still wouldn't expect to see a machine check when 
it goes to free the shared memory areas. I'm going to try it on a 24K 
malta as soon as I get yamon on it and can get it booted.

Greg Weeks

/*
 *
 *   Copyright (c) International Business Machines  Corp., 2001
 *
 *   This program is free software;  you can redistribute it and/or modify
 *   it under the terms of the GNU General Public License as published by
 *   the Free Software Foundation; either version 2 of the License, or
 *   (at your option) any later version.
 *
 *   This program is distributed in the hope that it will be useful,
 *   but WITHOUT ANY WARRANTY;  without even the implied warranty of
 *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
 *   the GNU General Public License for more details.
 *
 *   You should have received a copy of the GNU General Public License
 *   along with this program;  if not, write to the Free Software
 *   Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 */

/*
 * Copyright (C) Bull S.A. 1996
 * Level 1,5 Years Bull Confidential and Proprietary Information
 */

/*---------------------------------------------------------------------+
|                           shmem_test_06.c                            |
| ==================================================================== |
| Title:        Segment Register 0xE                                   |
|                                                                      |
| Purpose:      simultaneous attachment of more than ten shared        |
|               memory regions to a process. Using segment registers   |
|               0x3 to 0xC and 0xE .                                   |
|                                                                      |
| Description:  Simplistic test to verify that a process can obtain    |
|               11 shared memory regions in the adress space from      |
|               0x30000000 to 0xCFFFFFFF and 0xE0000000 to 0xEFFFFFFF. |
|                                                                      |
|                                                                      |
| Algorithm:    *  from 1 up to 11                                     |
|               {                                                      |
|               o  Create a key to uniquely identify the shared segment|
|		   using ftok()	subroutine.			       |
|               o  Obtain a unique shared memory identifier with       |
|                  shmget ()                                           |
|               o  Map the shared memory segment to the current        |
|                  process with shmat ()                               |
|               o  Index through the shared memory segment             |
|               }                                                      |
|               *  from 1 up to 11                                     |
|               {                                                      |
|               o  Detach from the segment with shmdt()                |
|               o  Release the shared memory segment with shmctl ()    |
|               }                                                      |
|                                                                      |
| System calls: The following system calls are tested:                 |
|                                                                      |
|               ftok()                                                 |
|               shmget ()                                              |
|               shmat ()                                               |
|               shmdt ()                                               |
|               shmctl ()                                              |
|                                                                      |
| Usage:        shmem_test_06                                          |
|                                                                      |
| To compile:   cc -O -g  shmem_test_06.c -o shmem_test_06 -lbsd       |
|                                                                      |
| Last update:                                                         |
|                                                                      |
| Change Activity                                                      |
|                                                                      |
|   Version  Date    Name  Reason                                      |
|    0.1     010797  J.Malcles  initial version for 4.2.G              |
|                                                                      |
|                                                                      |
+---------------------------------------------------------------------*/

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <sys/shm.h>

#include <sys/types.h>
#include <sys/ipc.h>

#include <sys/stat.h>
/*
off_t offsets[] = {
0x20000000,
0x30000000,
0x48000000,
0x50000000,
0x60000000,
0x70000000,
0x80000000,
0x90000000,
0xA0000000,
0xB0000000,
0xBFE00000,
};
*/
off_t offsets[] = {
0x20000000,
0x00000000,
0x48000000,
0x50000000,
0x60000000,
0x60200000,
0x60400000,
0x70200000,
0x70400000,
0x70600000,
0x70800000,
};
int offsets_cnt = sizeof (offsets) /sizeof (offsets[0]);
/* Defines
 *
 * MAX_SHMEM_SIZE: maximum shared memory segment size of 256MB 
 *
 * MAX_SHMEM_NUMBER: maximum number of simultaneous attached shared memory 
 * regions
 *
 * DEFAULT_SHMEM_SIZE: default shared memory size, unless specified with
 * -s command line option
 * 
 * SHMEM_MODE: shared memory access permissions (permit process to read
 * and write access)
 * 
 * USAGE: usage statement
 */
#define MAX_SHMEM_SIZE		256*1024*1024
#define MAX_SHMEM_NUMBER	11
#define DEFAULT_SHMEM_SIZE	1024*1024
#define	SHMEM_MODE		(SHM_R | SHM_W)
#define USAGE	"\nUsage: %s [-s shmem_size]\n\n" \
		"\t-s shmem_size  size of shared memory segment (bytes)\n" \
		"\t               (must be less than 256MB!)\n\n"

#define GOTHERE printf("got to line %d\n", __LINE__);

/*
 * Function prototypes
 *
 * parse_args (): Parse command line arguments
 * sys_error (): System error message function
 * error (): Error message function
 */
void parse_args (int, char **);
void sys_error (const char *, int);
void error (const char *, int);

/*
 * Global variables
 * 
 * shmem_size: shared memory segment size (in bytes)
 */
int shmem_size = DEFAULT_SHMEM_SIZE;

/*---------------------------------------------------------------------+
|                               main                                   |
| ==================================================================== |
|                                                                      |
|                                                                      |
| Function:  Main program  (see prolog for more details)               |
|                                                                      |
| Returns:   (0)  Successful completion                                |
|            (-1) Error occurred                                       |
|                                                                      |
+---------------------------------------------------------------------*/
int main (int argc, char **argv)
{
  off_t offset;
  int i;
  
  int shmid[MAX_SHMEM_NUMBER];	/* (Unique) Shared memory identifier */
  
  char *shmptr[MAX_SHMEM_NUMBER];	/* Shared memory segment address */
  char	*ptr;		/* Index into shared memory segment */
  
  int value = 0;	/* Value written into shared memory segment */
  
  key_t mykey[MAX_SHMEM_NUMBER];
  char * null_file = "/dev/null";
  char  proj[MAX_SHMEM_NUMBER] = {
    '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'E'
  };

  /*
   * Parse command line arguments and print out program header
   */
  parse_args (argc, argv);
  printf ("%s: IPC Shared Memory TestSuite program\n", *argv);
  
  for (i=0; i<offsets_cnt; i++) 
    {
      char tmpstr[256];
      /*
       * Create a key to uniquely identify the shared segment
       */
      
      mykey[i] = ftok(null_file,proj[i]);
      
      printf ("\n\tmykey to uniquely identify the shared memory segment 0x%x\n",mykey[i]);
      
      printf ("\n\tGet shared memory segment (%d bytes)\n", shmem_size);
      /*
       * Obtain a unique shared memory identifier with shmget ().
       */
      
      if ((long)(shmid[i]= shmget (mykey[i], shmem_size, IPC_CREAT | 0666 )) < 0)
	sys_error ("shmget failed", __LINE__);

      printf ("\n\tAttach shared memory segment to process\n");
      /*
       * Attach the shared memory segment to the process
       */
      offset = offsets[i];

      if ((long)(shmptr[i] = (char *) shmat (shmid[i], (const void*)offset, 0)) == -1)
	{
	  sprintf(tmpstr, "shmat failed - return: %p", shmptr[i]);
	  sys_error (tmpstr, __LINE__);
	}

      printf ("\n\tShared memory segment address : 0x%p \n",shmptr[i]);
      
      printf ("\n\tIndex through shared memory segment ...\n");
      
      /*
       * Index through the shared memory segment
       */
      
      for (ptr=shmptr[i]; ptr < (shmptr[i] + shmem_size); ptr++)
	*ptr = value++;
      
    } // for 1..11

  printf ("\n\tDetach from the segment using the shmdt subroutine\n");
  
  printf ("\n\tRelease shared memory\n");
  
  for (i=0; i<offsets_cnt; i++) 
    {
      // Detach from the segment
      
      shmdt( shmptr[i] );
      
      // Release shared memory
      
      if (shmctl (shmid[i], IPC_RMID, 0) < 0)
	sys_error ("shmctl failed", __LINE__);
      
    } // 2nd for 1..11
  /* 
   * Program completed successfully -- exit
   */
  
  printf ("\nsuccessful!\n");
  
  return (0);
  
}

/*---------------------------------------------------------------------+
|                             parse_args ()                            |
| ==================================================================== |
|                                                                      |
| Function:  Parse the command line arguments & initialize global      |
|            variables.                                                |
|                                                                      |
| Updates:   (command line options)                                    |
|                                                                      |
|            [-s] size: shared memory segment size                     |
|                                                                      |
+---------------------------------------------------------------------*/
void parse_args (int argc, char **argv)
{
  int	i;
  int	errflag = 0;
  char	*program_name = *argv;
  extern char 	*optarg;	/* Command line option */
  
  while ((i = getopt(argc, argv, "s:?")) != EOF) {
    switch (i) {
    case 's':
      shmem_size = atoi (optarg);
      break;
    case '?':
      errflag++;
      break;
    }
  }
  
  if (shmem_size < 1 || shmem_size > MAX_SHMEM_SIZE)
    errflag++;
  
  if (errflag) {
    fprintf (stderr, USAGE, program_name);
    exit (2);
  }
}

/*---------------------------------------------------------------------+
  |                             sys_error ()                             |
  | ==================================================================== |
  |                                                                      |
  | Function:  Creates system error message and calls error ()           |
  |                                                                      |
  +---------------------------------------------------------------------*/
void sys_error (const char *msg, int line)
{
  char syserr_msg [256];
  
  sprintf (syserr_msg, "%s: %s\n", msg, strerror (errno));
  error (syserr_msg, line);
}

/*---------------------------------------------------------------------+
|                               error ()                               |
| ==================================================================== |
|                                                                      |
| Function:  Prints out message and exits...                           |
|                                                                      |
+---------------------------------------------------------------------*/
void error (const char *msg, int line)
{
  fprintf (stderr, "ERROR [line: %d] %s\n", line, msg);
  exit (-1);
}

---

From: Ralf Baechle <[email-masked]>
Date: Fri, 8 Apr 2005 17:13:57 +0100

> >This is the malta branch with the tlb patch from Ralf applied. The LTP 

Credits for this patch go to Maciej who actually went through the pain
of debugging this problem that did show it's ugly head long, long after
the affected revision has been obsoleted by newer revisions.

> malta as soon as I get yamon on it and can get it booted.

I've been able to reproduce it on on two 4Kc boards of different revisions
so this seems to be something else.  It does not hit 5Kc and 24K.

  Ralf

---

From: Greg Weeks <[email-masked]>
Date: Fri, 08 Apr 2005 12:45:24 -0400

>so this seems to be something else.  It does not hit 5Kc and 24K.
Have you tried a 4kec? That's the only other mips board I have available.

Greg Weeks

---

From: Greg Weeks <[email-masked]>
Date: Mon, 11 Apr 2005 15:47:12 -0400

This patch appears to fix my machine check problem on the 4kc. The 4kc 
shouldn't need an ssnop here, but this appears to fix it.

Greg Weeks
--- mips-malta4kcle-basic/arch/mips/mm/tlbex.c-orig
+++ mips-malta4kcle-basic/arch/mips/mm/tlbex.c
@@ -847,7 +847,6 @@
 
 	case CPU_R10000:
 	case CPU_R12000:
-	case CPU_4KC:
 	case CPU_SB1:
 	case CPU_4KSC:
 	case CPU_20KC:
@@ -874,6 +873,7 @@
 		tlbw(p);
 		break;
 
+	case CPU_4KC:
 	case CPU_4KEC:
 	case CPU_24K:
 		i_ehb(p);

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Mon, 11 Apr 2005 22:27:15 +0200

If the 4KC and 4KEC need it, so does the 4KSC (and 4KSD).

----- Original Message ----- 
From: "Greg Weeks" <[email-masked]>
To: "Ralf Baechle" <[email-masked]>
Cc: <[email-masked]>
Sent: Monday, April 11, 2005 21:47
Subject: Re: another 4kc machine check.

> shouldn't need an ssnop here, but this appears to fix it.

--------------------------------------------------------------------------------

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Mon, 11 Apr 2005 22:42:57 +0200

> shouldn't need an ssnop here, but this appears to fix it.

The 4Kc shouldn't need an ssnop/ehb for the purposes of a *load*, but
if the exception was on an instruction fetch, the documentation is reasonably
clear that there's a hazard of 3 cycles before the new entry can be used for
a fetch - which is what ERET will do in the case of a instruction fetch miss.

            Regards,

            Kevin K.

---

From: Greg Weeks <[email-masked]>
Date: Tue, 12 Apr 2005 10:52:38 -0400

>If the 4KC and 4KEC need it, so does the 4KSC (and 4KSD).
Is this a reasonable thing to do though. I always hate making changes 
that work, but that I don't understand why.

Greg Weeks

---

From: "Maciej W. Rozycki" <[email-masked]>
Date: Tue, 12 Apr 2005 16:22:34 +0100 (BST)

> If the 4KC and 4KEC need it, so does the 4KSC (and 4KSD).

 But that's weird in the first place as 4Kc implements the original 
revision of MIPS32 so it does not implement "ehb".  Therefore it acts just 
as an ordinary "nop", but according to the 4K manual there is no need for 
one -- the hazard between a move to EntryLo0/EntryLo1 and tlbwi/tlbwr is 
explicitly listed as 0 instructions.

  Maciej

---

From: "Kevin D. Kissell" <[email-masked]>
Date: Tue, 12 Apr 2005 18:38:14 +0200

> explicitly listed as 0 instructions.

Oops.  Maybe I misread the patch.  I thought the added NOP was between
the TLBWR and the ERET.

            Kevin K.

---

From: Greg Weeks <[email-masked]>
Date: Tue, 12 Apr 2005 12:52:18 -0400

>Oops.  Maybe I misread the patch.  I thought the added NOP was between
No, it adds it before the TLBWR where there shouldn't be a hazard.

Greg Weeks

---

From: Greg Weeks <[email-masked]>
Date: Tue, 12 Apr 2005 15:12:29 -0400

> No, it adds it before the TLBWR where there shouldn't be a hazard.

On the off chance that the hazard between the TLBWR and the ERET might 
be coming into play I tried a kernel with 3 nops between the TLBWR and 
ERET and no EHB before the TLBWR and it still machine checks for me.

Greg Weeks