VectorLinux Light vs. Zenwalk: Benchmarks [updated version 29 Sep 09]

Have you heard the news? There will be another Ubuntu derivative, this time based on LXDE and hence called Lubuntu. LXDE is a lightweight desktop environment, and this added to the generally high profile of *ubuntu distributions might make for an interesting lightweight distro indeed, and perhaps fresh competition to VectorLinux.

So I wandered over to the LXDE blog post and downloaded both ISOs, planning on testing them against my favourite VectorLinux Light 6.0. These are live CD isos and they proved to be un-installable, so they’re nothing I’m interested in for the moment. Will have to wait till the first installable ISO is out.

However, I was reading up about testing distros against one another for this purpose after being pointed at benchmarking on the VectorLinux forum. In the course of my search I found an interesting post on benchmarking Arch Linux vs. Ubuntu, and I figured I could do a little benchmarking myself to see if my choice of lightweight Linux distribution could be substantiated by hard figures. Originally I planned on testing four Linux distributions: Damn Small Linux, Puppy Linux, Zenwalk, and VectorLinux Light. As every distro can be tweaked to perform better, I wanted to test the clean installs as they would have been installed by a newbie.

Unfortunately, this is where I hit the first wall, as I’m indeed quite the newbie myself: I couldn’t for the life of me get Damn Small Linux installed to hard disk. Fair enough, it was meant as a live CD and that worked, but all the hints on their, erm, economic website didn’t enable me to install it, so I left it (as probably any other newbie would have). Puppy Linux proved more interesting. I ran into a few problems with GRUB, and installation wasn’t easy (despite the shiny install icon in the desktop), but I got it to run. Only to find that it had no compiler whatsoever installed by default. At least their community has found out about it, but as their solution was to install another .sfs file and make my clean install into a dev version, I declined.

Proves that some versions of Linux are more to my taste than others, as I was left with two distributions based on Slackware: VectorLinux Light 6.0 and Zenwalk 6.0. I’ve been trying to make my testing method as watertight as possible.

Method

Installation
Both distributions were installed on the same machine. The hard disk was partitioned into two partitions: primary bootable partition formatted with ext3, logical Linux swap partition of standardized, (RAM*2)+10% size: 570 MB. During installation, no advanced options were considered, if possible, I just “clicked though it”. I then checked the distributions with version and uname -r to make sure what I had before me.

Tools
Just like Michael Weingärtner, I used two test tools to run benchmarks tests: GtkPerf and the Phoronix Test Suite. While GtkPerf obviously tests GTK+ performance, The Phoronix Test Suite (PTS) includes a wide range of test tools, of which I included 3:

  • x11perf – a very basic performance/regression test for X.Org. I have used option 2: Scrolling 500 x 500px.
  • byte – This test runs the ANSI C version of SciMark 2.0, which is a benchmark for scientific and numerical computing developed by programmers at the National Institute of Standards and Technology. This test is made up of Fast Fourier Transform, Jacobi Successive Over-relaxation, Monte Carlo, Sparse Matrix Multiply, and dense LU matrix factorization benchmarks. I have used option 1: Dhrystone.
  • scimark2 – similar to byte. I have used the composite version of this test.

I compiled GtkPerf from source. PTS needed PHP ≥ 5.0, which I downloaded from php.net, configured it with –with-gd and then made and installed it. PTS itself only needs to be untarred, then run from the local directory (using ./phoronix-test-suite benchmark <name of test>). PTS then downloads and installs whatever it needs for the test (you are asked for your root password before doing this, obviously). In between each and every single test, I rebooted. This might have been a Windows-related knee-jerk reaction, but I wanted to have as few as possible processes running.

Results

Zenwalk is version 6.0 w/ kernel 2.6.28.7
VectorLinux Light is version 6.0 w/ kernel 2.6.27.12

  1. Boot time test: Time from press of power button to fully loaded desktop (including typing user name and password). Obviously not the most precise of tests, but timed with a digital stopwatch at least.

    Zenwalk: ~ 90 secs.
    VectorLinux Light: ~ 75 secs.

  2. GtkPerf I: 3 consecutive runs with 100 test rounds directly after reboot.
  3. Test Zenwalk VectorLinux Light
    GtkEntry 0.14 / 0.15 / 0.14 / 0.143 0.13 / 0.13 / 0.13 / 0.13
    GtkComboBox 5.20 / 5.20 / 5.30 / 5.23 5.60 / 5.62 / 5.59 / 5.603
    GtkComboBoxEntry 3.52 / 3.44 / 3.53 / 3.497 4.83 / 4.82 / 4.84 / 4.83
    GtkSpinButton 1.02 / 1.12 / 1.04 / 1.06 1.08 / 1.09 / 1.09 / 1.087
    GtkProgressBar 0.92 / 0.86 / 0.86 / 0.88 0.78 / 0.79 / 0.79 / 0.787
    GtkToggleButton 1.14 / 1.15 / 1.14 / 1.143 2.16 / 2.18 / 2.17 / 2.17
    GtkCheckButton 0.59 / 0.60 / 0.68 / 0.623 1.91 / 1.93 / 1.95 / 1.93
    GtkRadioButton 0.85 / 0.96 / 0.87 / 0.893 2.43 / 2.44 / 2.47 / 2.447
    GtkTextView – Add text 3.51 / 3.51 / 3.51 / 3.51 3.15 / 3.18 / 3.18 / 3.17
    GtkTextView – Scroll 0.91 / 0.91 / 0.92 / 0.913 1.81 / 1.83 / 1.80 / 1.813
    GtkDrawingArea – Lines 1.50 / 1.44 / 1.44 / 1.46 1.15 / 1.15 / 1.15 / 1.15
    GtkDrawingArea – Circles 1.86 / 1.95 / 1.96 / 1.923 1.55 / 1.54 / 1.54 / 1.543
    GtkDrawingArea – Text 3.36 / 3.26 / 3.35 / 3.323 12.27 / 12.25 / 12.30 / 12.273
    GtkDrawingArea – Pixbufs 0.58 / 0.68 / 0.61 / 0.623 0.68 / 0.69 / 0.68 / 0.683
  4. GtkPerf II: 3 consecutive runs with 100 test rounds directly after reboot with 3 standard applications running: word processor, image manipulation, web browser.
  5. Test Zenwalk VectorLinux Light
    GtkEntry 0.13 / 0.15 / 0.13 / 0.137 0.12 / 0.10 / 0.11 / 0.11
    GtkComboBox 5.98 / 4.91 / 4.98 / 5.29 5.66 / 5.65 / 5.67 / 5.66
    GtkComboBoxEntry 3.59 / 3.61 / 4.22 / 3.807 5.29 / 5.23 / 5.28 / 5.267
    GtkSpinButton 1.04 / 1.04 / 1.10 / 1.06 1.06 / 1.06 / 1.06 / 1.06
    GtkProgressBar 0.85 / 0.84 / 0.88 / 0.857 0.75 / 0.76 / 0.76 / 0.757
    GtkToggleButton 1.13 / 1.17 / 1.14 / 1.147 2.16 / 2.16 / 2.16 / 2.16
    GtkCheckButton 0.64 / 0.60 / 0.60 / 0.613 1.92 / 1.92 / 1.92 / 1.92
    GtkRadioButton 0.87 / 0.87 / 0.88 / 0.873 2.52 / 2.52 / 2.52 / 2.52
    GtkTextView – Add text 3.41 / 3.42 / 3.43 / 3.42 3.17 / 3.19 / 3.17 / 3.177
    GtkTextView – Scroll 0.93 / 0.91 / 0.91 / 0.917 1.83 / 1.82 / 1.82 / 1.823
    GtkDrawingArea – Lines 1.44 / 1.45 / 1.46 / 1.45 1.14 / 1.14 / 1.13 / 1.137
    GtkDrawingArea – Circles 2.28 / 1.89 / 1.89 / 2.02 1.53 / 1.51 / 1.51 / 1.517
    GtkDrawingArea – Text 3.65 / 3.27 / 3.26 / 3.393 12.32 / 12.33 / 12.28 / 12.31
    GtkDrawingArea – Pixbufs 0.60 / 0.59 / 0.56 / 0.583 0.69 / 0.68 / 0.68 / 0.683
  6. Phoronix Test Suite tests, run directly after boot, reboot after each test (for all tests: higher figures = better performance).
  7. Test Zenwalk VectorLinux Light
    x11perf 915.00 915.00
    byte 1607476.70 1596308.20
    scimark2 64.17 63.60

Discussion

Zenwalk 6.0 seems to be generally faster than VectorLinux Light 6.0, as it gets better figures for more test items than VL.

However, not all test items are equally important to the average user, and for a lot of test items, value differences are rather small. One value that seems of importance to me is obviously boot time (nobody likes waiting for his/her machine to start up).

One test item that strikes with significantly different values is “GtkDrawingArea – Text”. Now, according to the Gtk Class Reference, GtkDrawingArea is a widget that “is used for creating custom user interface elements”, in this case text. Newt’s comment on the old version of this post, has rightly pointed out that in order to rate significance of test items, one would have to find out what impact such a test item (in this case, the Gtk widget) has on desktop usage. And to be honest, I don’t really know (cf. below, “Note”). In retrospect, I think Gtk, as a library to build GUIs, isn’t exactly the perfect choice to test a distro for everyday use.

Unfortunately, the Phoronix Test Suite (which is more likely to include more relevant tests IMHO) ran into some problems during the install of a more universal testing suite, at which point I’ve simply given up and stuck with the 3 tests described above.

So, what’s the point to all of this? Firstly, I’ve had a first look at benchmarking, and I’ve come up with results. It proved that it’s probably more complicated to interpret those results than simply getting them. That’s why I’ve re-written this conclusion, which wasn’t much more than a late-night try to make VL Light look better than the figures would suggest, simply because I like it better (and it *feels* faster, and the handling’s really better, etc.).

Secondly,  it shows that you should never copy a testing method unreflectedly and without further testing of the method itself.

I will perhaps try to run another battery of benchmark  tests over both distros. Until then, please don’t be shy to comment and make suggestions.

Note

Finally, I’d like to stress that I have little experience with benchmarking. Please point me to any mistakes I might have made in the process (e.g., are there more relevant values I should have tested?), especially in the conclusion!

Advertisements

4 Comments

Filed under Linux distribution

4 responses to “VectorLinux Light vs. Zenwalk: Benchmarks [updated version 29 Sep 09]

  1. Pingback: Roy Schestowitz (schestowitz) 's status on Tuesday, 29-Sep-09 21:50:23 UTC - Identi.ca

  2. newt

    I am aware that benchmarking is a highly subjective process and that results will vary from system to system. Regardless, I always find it fun to review such results even though I take all reviews with much salt.

    I enjoyed reading your benchmarking test methodology as well as the test results. I was not necessarily surprised to see that Zenwalk demonstrates better performance than VectorLinux in most categories, as it is a very light-weight/stream-lined distro with emphasis on being “minimalist” (for lack of a better term). Keep up the research on benchmarking and I look forward to any future comparison-type benchmarking you may perform in the future.

    Criticism:
    My personal feeling is that your discussion/conclusion shows bias towards VectorLinux. The results seem to indicate overall better performance from Zenwalk yet you conclude that VectorLinux “wins”. Perhaps rather than try to produce a “winner” you could describe (in layman terms) some of the various tests that were performed and how they relate to the user experience, and just leave it up to the reader to create a conclusion.

    Regarding the results you present that make VL the winner (boot time and gtkentry): I know what boot time is and can relate, but how does gtkentry affect my desktop experience? Specifically, how does it slow down my Zenwalk experience and make my VL experience better?

    The percentages you present sure seem significant but looking at the raw data indicates that the gtkentry difference is very little, especially if those are values are in seconds. On a related note, I noticed the relatively large difference in “GtkDrawingArea – Text” with VL having an average 9.x slower “speed” than Zenwalk. If those are seconds then that could be a serious significance; plus, could point to a factor that VL devs should explore. But, again, what does “GtkDrawingArea – Text” mean and how does it affect my user experience?

    On the other hand, a 15-second boot time difference is significant as compared to the overall time of either distro, and it’s a result I can directly relate to. However, since I’m an “always on” computer user I am very rarely waiting for my system to boot up so this doesn’t much persuade me that VL has better performance.

    Please don’t take these criticisms too harsh, as they are not meant that way. More just thoughts I had.

    • S

      UPDATE NOTE:

      The original Discussion section (the one which, amongst other things, newt comments on) read:

      “Zenwalk 6.0 seems to be generally faster than VectorLinux Light 6.0, as it gets better figures for more test items than VL.

      However, if we look at which values are more important for general users and newbies with old hardware, we have to come to the conclusion that VectorLinux Light 6.0 “wins” over Zenwalk 6.0 with better figures for main values such as boot time (VL 16.7% faster than Z), and GtkEntry (VL 9% and 20% faster than Z).”

    • S

      Thanks, newt, for your comments. I think you are right about my bias towards VectorLinux. In everyday use, it *really* feels faster and I like it much better than Zenwalk for reasons other than mere speed.
      Some of the things about benchmarking you said were really interesting and in fact they rubbed under my nose that I didn’t scrutinise the test method before I employed it. Which I should have.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s