Subject: NTools ENewsFlash -- Report: NT 3.5x Faster Than Linux

From: [email-masked] (Ariel Faigon)
Date: Wed, 14 Apr 1999 11:20:47 -0700 (PDT)

[I'm forwarding a message that bounced.  Just thought some people
 may be interested.  To add to what Eric wrote, another thing that
 I find strange is that they are comparing NT 3.5 rather than the
 latest NT 4.x to Linux.  I also believe some of the results
 are missing crucial details, like how the two very different
 web servers were configured (e.g. standalone vs from inetd,
 ssi turned on etc.) who paid for the report, full disclosures...
 -- Ariel]

---------- Forwarded message:
:Sender: [email-masked]
:Date: Wed, 14 Apr 1999 10:22:27 -0700
:From: Eric Kimminau <[email-masked]>
:Reply-To: [email-masked]
:Organization: Electronic Support Tools
:To: [email-masked]
:Subject: [Fwd: NTools ENewsFlash -- Report: NT 3.5x Faster Than Linux]
:
:I honestly don't believe this. The primary key being that they made no
:kernel rebuild and they were using all modules rather than building
:things like raid support directly into the kernel.
:
:Their bandwidth utilization also stinks of using 10Mbit vs. 100Mbit
:during testing.
:
:They also give no indication of the kinds of data they were delivering
:(CGI, DHTML, java/javascript) nor the version/vendor for the
:webserver.
:
:This is Microsoft FUD.
:
:EAK.
:
:-- 
:---------1---------2---------3---------4---------5---------6---------7
:   Eric Kimminau        [email-masked]        Electronic Support Tools
:            Voice: (248) 848-4455    VNet:  6-327-4455
:                 "I speak my mind and no one else's."
: "I am a bomb technician. If you see me running, try to keep up..."
:                      http://support.sgi.com
:
:To: "Latest Win NT News" <[email-masked]>
:From: [email-masked]
:List-Unsubscribe: <mailto:[email-masked]>
:Reply-To: [email-masked]
:Message-Id: <[email-masked]>
:Precedence: bulk
:
:*****************************************************
:NTools E-NewsFlash: Report: NT 3.5x Faster Than Linux
:*****************************************************
:                    April 14, 1999
:
:Hi All,
:
:As most of you know, when some important news hits the
:wires we will inform you immediately. This morning I
:found something in my in-box that will definitely throw
:some more gasoline on the raging 'NT vs Linux' fire.
:
:I quickly read through the report and could not find
:anything wrong with it at first observation. The numbers
:seem to be correct, as they are using an industry standard
:benchmark that I have used myself as well, the ZD bench.
:
:Oh, before I forget, Novell actually wrote a rebuttal 
:against that last report that SMS is better than ZEN.
:Interesting reading, and this gives the Novell POV:
:http://www.novell.com/products/nds/zenworks/ms2.html
:
:But here comes today's bomb in the NT vs Linux battle.
:One wonders who pays for these tests but I will ask the
:CEO of MindCraft and report on that in the next coming
:newsletter. Here goes!
:
:
:LOS GATOS, Calif., April 13. Today, Mindcraft released the results 
:of a study comparing the performance of Red Hat Linux 5.2 (updated 
:to the Linux 2.2.2 kernel) and Microsoft Windows NT Server 4.0 
:operating systems. According to the report, Windows NT Server 
:provides over three and a half times the performance of Linux as a
:Web server.  Furthermore, the report shows that when testing Windows 
:NT Server and Linux as file servers, Windows NT Server provides over 
:two and a half times the performance of Linux.  The full report,
:including all of the details needed to reproduce the tests, is on
:Mindcraft's Web site at:
:
:http://www.mindcraft.com/whitepapers/nts4rhlinux.html.
:
:Using benchmarks from Ziff-Davis Benchmark Operation (ZDBOp), the 
:report compares the peak performance levels of both Windows NT 
:Server and Linux configured both as a file server and a Web server.   
:
:All tests were performed on a standard Dell PowerEdge 6300/400 server 
:with four 400-MHz Xeon CPUs, 1GB RAM (960MB for Linux -- this is the
:default maximum amount of RAM that Linux supports).  To simulate a 
:client load, Mindcraft used 144 physical client test systems; half 
:were running Windows 95 and the other half were running Windows 98.  
:
:Both Linux and Windows NT Server were tuned to perform optimally under
:each of the two workloads.  "We started the tests using standard Red Hat
:Linux 5.2 but had to update it because it does not support hardware RAID
:controllers and SMP at the same time," said Mindcraft's president, Bruce
:Weiner.  "Linux definitely takes more time and resources to tune and to
:configure than Windows NT Server.  You have to search the Net to find the
:latest kernel and driver versions to get the highest performance and most
:reliable modules. Then when you're done, Linux fails to deliver the same
:level of performance as Windows NT Server on enterprise-class servers."
:
:Mindcraft's report shows that using ZDBOp's WebBench 2.0 Web server
:benchmark, Windows NT Server and Internet Information Server (IIS) 4.0
:reach a peak of 3,771 requests/second and 22.4 Megabits/second(Mbps) of
:throughput. The report goes on to show that Linux and the Apache 1.3.4 
:Web server reach a peak of 1,000 request/second and 5.9 Mbps of
:throughput.  The WebBench 2.0 tests also revealed that there are problems
:with Linux/Apache at high client loads. "The Linux/Apache Web server
:performance collapsed with a WebBench load above 160 client test threads,
:while Windows NT Server/IIS continued to increase performance up through
:288 client test threads without reaching their peak performance," adds 
:Mindcraft's Bruce Weiner.
:
:To simulate a file server workload, Mindcraft used ZDBOp's NetBench 
:5.01 benchmark.  The testing revealed that Windows NT Server performance
:peaked at 286.7 Mbps with 112 clients, while Linux running Samba 2.0.1
:peaked at 114.6 Mbps with only 48 clients.  "The integration of the SMB
:file sharing protocol with the multi-processor kernel is a key performance
:win for Windows NT Server," said Weiner. "Customers benefit every day from
:the superior scalability of Windows NT Server, which delivers vital file
:and web services at two to three times the performance of Linux as shown
:in these benchmarks," said Edmund Muth, Group Product Manager, Microsoft
:Corporation. "Empirical data like this helps customers and planners make
:informed decisions, and showcases the industrial strength technology and
:mature engineering of the Windows NT Server operating system."
:
:About Mindcraft
:
:Mindcraft is a service-oriented, independent test lab. The company was
:founded in 1985 to provide high quality services and products to vendors
:and end users who want to test software, system, and network products.
:Mindcraft is committed to work to promote standards in our industry.
:Mindcraft is the only test lab to be a member of the Standard Performance
:Evaluation Corporation (SPEC).
:--------------------------------------
:
:That's all for this NewsFlash!
:
:Warm regards,
:
:Stu
:
:
:[[email-masked]] This is a posting from the
:nt-list, To unsubscribe, send a blank email 
:to [email-masked]
:For killer servers at unbelievable prices check out:
:http://www.dell.com/outlet/sunbelt.htm 
:
:

-- 
Peace, Ariel

---

From: Matthias Kleinschmidt <[email-masked]>
Date: Wed, 14 Apr 1999 15:24:21 -0400

Hi,

>  ssi turned on etc.) who paid for the report, full disclosures...

I think that this is not the right list to discuss this strange report 
but anyhow a few comments:

- The report was sponsored by Microsoft as you can read here: 
   http://www.mindcraft.com/whitepapers/nts4rhlinux.html#mccert

- Mindcraft compared Microsoft Windows NT Server 4.0 and Novell NetWare 5 in
  January. Here you can find Novells response:
   http://www.novell.com/advantage/nw5/nw5-mindcraftcheck.html

Matthias

-- 
Matthias Kleinschmidt
Cedar Dell 568B, Box 5398
UMass Dartmouth
285 Old Westport Rd.
North Dartmouth, MA 02747
Tel.:  +1 508 910 6851
email: [email-masked]

---

From: [email-masked] (Alan Cox)
Date: Wed, 14 Apr 1999 21:09:26 +0100 (BST)

Oh they did a very professional hatchet job. But then these are the people
who are responsible for a whole pile of such things.

They picked a kernel carefully chosen to have an NT TCP performance 
interoperability bug. They mistuned the linux box completely. They claim
to have checked on the net but mindscape never touched the linux.org.uk
web site since 1998

etc.

Whats very funny of course is Microsoft's Halloween II document is one of
the many things that contradicts this survey

---

From: [email-masked] (Gerald Henriksen)
Date: Wed, 14 Apr 1999 22:24:33 GMT

> ssi turned on etc.) who paid for the report, full disclosures...

A couple of clarifications (more info can be found by checking out
Slashdot, which carried the story Tuesday and at this point has over
700 comments)

1) they used NT Server 4

2) Microsoft paid for the testing (says so on the page that reports
the results)

3) they did compile a Linux kernel as they had to update Red Hat 5.2
to the 2.2.2 kernel, and they also mention the 3 changes they made to
the kernel config.  They didn't however do much to optimize Linux

4) someone even dug up the message they posted asking for help, but
they ignored at leat the response that told them to forget wasting 4
CPUs on a web server.  Note that they also goofed in their plea for
help, claiming that they were using Apache 2.* which doesn't exist.

-------------------------------------------------------------------------
 Gerald Henriksen                                        [email-masked]
 4th Year Computer Science                            [email-masked]
 Ryerson Polytechnic University                  [email-masked]

---

From: Raj Mathur <[email-masked]>
Date: Thu, 15 Apr 1999 10:36:04 +0530 (IST)

>>>>> "Ariel" == Ariel Faigon <[email-masked]> writes:

    Ariel> [I'm forwarding a message that bounced.  Just thought some
    Ariel> people may be interested.  To add to what Eric wrote,
    Ariel> another thing that I find strange is that they are
    Ariel> comparing NT 3.5 rather than the latest NT 4.x to Linux.  I
    Ariel> also believe some of the results are missing crucial
    Ariel> details, like how the two very different web servers were
    Ariel> configured (e.g. standalone vs from inetd, ssi turned on
    Ariel> etc.) who paid for the report, full disclosures...  --
    Ariel> Ariel]

>From http://www.mindcraft.com/whitepapers/nts4rhlinux.html

[About 3/4th way down the report]

Mindcraft, Inc. conducted the performance tests described in this
report between March 10 and March 13, 1999. Microsoft Corporation
sponsored the testing reported herein.

[Lookee who paid for it! ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^]

Mindcraft certifies that the results reported accurately represent the
file-server performance of Microsoft Windows NT Server 4.0 and Red Hat
Linux 5.2 upgraded to the Linux 2.2.2 kernel with Samba 2.0.1 running
on a Dell PowerEdge 6300/400 as measured by NetBench 5.01. Also, we
certify that the Web-server performance reported for Windows NT Server
4.0 with IIS 4 and for Red Hat Linux 5.2 upgraded to the Linux 2.2.2
kernel with Apache 1.3.4 accurately represent the WebBench 2.0
measurements we made on a Dell PowerEdge 6300/400.

Our test results should be reproducible by others using the same test
lab configuration, the same Dell computer, and the software
configurations and modifications documented in this report.

[End snippet]

    Ariel> April 14, 1999 : :Hi All, : :As most of you know, when some
    Ariel> important news hits the :wires we will inform you
    Ariel> immediately. This morning I :found something in my in-box
    Ariel> that will definitely throw :some more gasoline on the
    Ariel> raging 'NT vs Linux' fire.  : :I quickly read through the
    Ariel> report and could not find :anything wrong with it at first
    Ariel> observation. The numbers :seem to be correct, as they are
    Ariel> using an industry standard :benchmark that I have used
    Ariel> myself as well, the ZD bench.  : :Oh, before I forget,
    Ariel> Novell actually wrote a rebuttal :against that last report
    Ariel> that SMS is better than ZEN.  :Interesting reading, and
    Ariel> this gives the Novell POV:
    Ariel> :http://www.novell.com/products/nds/zenworks/ms2.html :
    Ariel> :But here comes today's bomb in the NT vs Linux battle.
    Ariel> :One wonders who pays for these tests but I will ask the
    Ariel> :CEO of MindCraft and report on that in the next coming
    Ariel> :newsletter. Here goes!  : : :LOS GATOS, Calif., April
    Ariel> 13. Today, Mindcraft released the results :of a study
    Ariel> comparing the performance of Red Hat Linux 5.2 (updated :to
    Ariel> the Linux 2.2.2 kernel) and Microsoft Windows NT Server 4.0
    Ariel> :operating systems. According to the report, Windows NT
    Ariel> Server :provides over three and a half times the
    Ariel> performance of Linux as a :Web server.  Furthermore, the
    Ariel> report shows that when testing Windows :NT Server and Linux
    Ariel> as file servers, Windows NT Server provides over :two and a
    Ariel> half times the performance of Linux.  The full report,
    Ariel> :including all of the details needed to reproduce the
    Ariel> tests, is on :Mindcraft's Web site at: :
    Ariel> :http://www.mindcraft.com/whitepapers/nts4rhlinux.html.  :
    Ariel> :Using benchmarks from Ziff-Davis Benchmark Operation
    Ariel> (ZDBOp), the :report compares the peak performance levels
    Ariel> of both Windows NT :Server and Linux configured both as a
    Ariel> file server and a Web server.  : :All tests were performed
    Ariel> on a standard Dell PowerEdge 6300/400 server :with four
    Ariel> 400-MHz Xeon CPUs, 1GB RAM (960MB for Linux -- this is the
    Ariel> :default maximum amount of RAM that Linux supports).  To
    Ariel> simulate a :client load, Mindcraft used 144 physical client
    Ariel> test systems; half :were running Windows 95 and the other
    Ariel> half were running Windows 98.  : :Both Linux and Windows NT
    Ariel> Server were tuned to perform optimally under :each of the
    Ariel> two workloads.  "We started the tests using standard Red
    Ariel> Hat :Linux 5.2 but had to update it because it does not
    Ariel> support hardware RAID :controllers and SMP at the same
    Ariel> time," said Mindcraft's president, Bruce :Weiner.  "Linux
    Ariel> definitely takes more time and resources to tune and to
    Ariel> :configure than Windows NT Server.  You have to search the
    Ariel> Net to find the :latest kernel and driver versions to get
    Ariel> the highest performance and most :reliable modules. Then
    Ariel> when you're done, Linux fails to deliver the same :level of
    Ariel> performance as Windows NT Server on enterprise-class
    Ariel> servers."  : :Mindcraft's report shows that using ZDBOp's
    Ariel> WebBench 2.0 Web server :benchmark, Windows NT Server and
    Ariel> Internet Information Server (IIS) 4.0 :reach a peak of
    Ariel> 3,771 requests/second and 22.4 Megabits/second(Mbps) of
    Ariel> :throughput. The report goes on to show that Linux and the
    Ariel> Apache 1.3.4 :Web server reach a peak of 1,000
    Ariel> request/second and 5.9 Mbps of :throughput.  The WebBench
    Ariel> 2.0 tests also revealed that there are problems :with
    Ariel> Linux/Apache at high client loads. "The Linux/Apache Web
    Ariel> server :performance collapsed with a WebBench load above
    Ariel> 160 client test threads, :while Windows NT Server/IIS
    Ariel> continued to increase performance up through :288 client
    Ariel> test threads without reaching their peak performance," adds
    Ariel> :Mindcraft's Bruce Weiner.  : :To simulate a file server
    Ariel> workload, Mindcraft used ZDBOp's NetBench :5.01 benchmark.
    Ariel> The testing revealed that Windows NT Server performance
    Ariel> :peaked at 286.7 Mbps with 112 clients, while Linux running
    Ariel> Samba 2.0.1 :peaked at 114.6 Mbps with only 48 clients.
    Ariel> "The integration of the SMB :file sharing protocol with the
    Ariel> multi-processor kernel is a key performance :win for
    Ariel> Windows NT Server," said Weiner. "Customers benefit every
    Ariel> day from :the superior scalability of Windows NT Server,
    Ariel> which delivers vital file :and web services at two to three
    Ariel> times the performance of Linux as shown :in these
    Ariel> benchmarks," said Edmund Muth, Group Product Manager,
    Ariel> Microsoft :Corporation. "Empirical data like this helps
    Ariel> customers and planners make :informed decisions, and
    Ariel> showcases the industrial strength technology and :mature
    Ariel> engineering of the Windows NT Server operating system."  :
    Ariel> :About Mindcraft : :Mindcraft is a service-oriented,
    Ariel> independent test lab. The company was :founded in 1985 to
    Ariel> provide high quality services and products to vendors :and
    Ariel> end users who want to test software, system, and network
    Ariel> products.  :Mindcraft is committed to work to promote
    Ariel> standards in our industry.  :Mindcraft is the only test lab
    Ariel> to be a member of the Standard Performance :Evaluation
    Ariel> Corporation (SPEC).
    Ariel> :-------------------------------------- : :That's all for
    Ariel> this NewsFlash!  : :Warm regards, : :Stu : : :[[email-masked]]
    Ariel> This is a posting from the :nt-list, To unsubscribe, send a
    Ariel> blank email :to
    Ariel> [email-masked] :For
    Ariel> killer servers at unbelievable prices check out:
    Ariel> :http://www.dell.com/outlet/sunbelt.htm : :

    Ariel> -- Peace, Ariel