OPINION
Incredulity, Reverse Bias and Mainframe Linux

And here are excerpts from the IBM sizing analysis for the 16-CPU 64-GB machine:

Cost and Performance

The Microsoft-sponsored report has cost information. IBM's does not. In summary, VeriTest's estimates for the annual cost of operating the mainframe partition as benchmarked are based on numbers from Gartner and range from about US$252,000 to about $480,000, depending on factors like customer licensing and alternate workload.

According to the report, these annual costs range from 10 to 20 times that of the more powerful 900-MHz dual Xeon machine running Windows Server 2003.

According to VeriTest, the best throughput results obtained from the 2-CPU z900 were 68.25 MB/second with one Linux instance running without zVM and 52.12 MB/second with four Linux instances running with zVM. Scale this linearly to the full 16-CPU system, and you would predict that the machine should offer about 546 MB/second native and 420 MB/second with perhaps 16 Linux instances under zVM.

IBM's results are nowhere near that. Using a single guest server in an eight-CPU partition, they were able to achieve only about 19.4 MB/second on a single OSA/Express GigaBit Ethernet card that maxed out at 105.8 MB/second when running zVM on the full machine with "between 50 and 70" concurrent Linux instances and all 10 OSA/GBE boards.

Scaling Is the Answer

The z900s max out at 64 GB of memory. For the Microsoft configuration to scale up linearly, you would need three times the system maximum.

Furthermore, the z900s used in the test have 24 bidirectional 1 Gbps ports on the main board, each of which connects to one or more external devices like disk controllers and the OSA/GBE boards. VeriTest used two dual-channel OSA/GBE cards to connect to the network and two fibre channel connectors running to a single 1.2 TB ESS/Shark array, while IBM used 10 OSA/GBE cards and 32 controllers on eight ports. For the Microsoft configuration to scale up linearly with unimpeded data flow on each device, you would need 32 ports -- eight more than the system maximum.

In effect, VeriTest got better than expected zVM/Linux results because the partition it used dramatically overstates the scalability of the machine. As a corollary, this also has the effect of seriously understating the cost of the fully scaled-up zVM system and thus hurts Microsoft's case on both cost and performance.

Blank or a Blink, Bill?

There's a more subtle question here. You don't have to be unusually cynical to expect that a report paid for by Microsoft and executed by a spin-off from an organization founded on hyping Microsoft products would tend to favor Microsoft. Why, then, did this report seem to somewhat exaggerate the real-world viability of mainframe Linux?

I have two ideas -- one related to beliefs about how operating systems and hardware condition tester behavior and one related more specifically to the report at hand. The results shown by Microsoft -- both the performance advantage a dual 900-MHz Xeon with Windows 2003 Server has over the "Turbo" mainframe and the 20-to-1 cost advantage to Wintel -- contradict the idea that enterprise-scale "big iron" must be as powerful as it is expensive.

It is possible, therefore, to guess that VeriTest produced some initial results that were simply disbelieved, resulting in the generosity shown in the revised document as ultimately published. An explanation based on a reverse bias arising from incredulity is supported by the fear, uncertainty and doubt that has to underlie this paragraph from the report:

The META Group, an independent consultancy, has audited the [test] plan, the facility, the tests and the final report. META Group was asked to verify that the benchmark configuration and procedures were appropriate but was not asked to endorse the results one way or the other. Also, neither VeriTest nor Ziff Davis Media, who provided the PC Magazine benchmark test suites, were approached about endorsing the results. Based on IBM's marketing, expectations going in to the project were that mainframe Linux would produce results at the higher end of Windows server performance. The results turned out quite the opposite.

Not only is this hesitancy uncharacteristic of Microsoft, but it's totally missing from the report's evil twin -- an earlier VeriTest report comparing Windows 2003 Server to Linux on that same dual 900-MHz Xeon.

That analysis was published in May, four months before this one, and pretty much conforms to cynical expectation with lots of questionable content and a probable wrong conclusion favoring Microsoft -- but no quavering quibbles about third-party consultants, tame or otherwise, reviewing test plans or authenticating processes.