Have you heard the news? There will be another Ubuntu derivative, this time based on LXDE and hence called Lubuntu. LXDE is a lightweight desktop environment, and this added to the generally high profile of *ubuntu distributions might make for an interesting lightweight distro indeed, and perhaps fresh competition to VectorLinux.
So I wandered over to the LXDE blog post and downloaded both ISOs, planning on testing them against my favourite VectorLinux Light 6.0. These are live CD isos and they proved to be un-installable, so they’re nothing I’m interested in for the moment. Will have to wait till the first installable ISO is out.
However, I was reading up about testing distros against one another for this purpose after being pointed at benchmarking on the VectorLinux forum. In the course of my search I found an interesting post on benchmarking Arch Linux vs. Ubuntu, and I figured I could do a little benchmarking myself to see if my choice of lightweight Linux distribution could be substantiated by hard figures. Originally I planned on testing four Linux distributions: Damn Small Linux, Puppy Linux, Zenwalk, and VectorLinux Light. As every distro can be tweaked to perform better, I wanted to test the clean installs as they would have been installed by a newbie.
Unfortunately, this is where I hit the first wall, as I’m indeed quite the newbie myself: I couldn’t for the life of me get Damn Small Linux installed to hard disk. Fair enough, it was meant as a live CD and that worked, but all the hints on their, erm, economic website didn’t enable me to install it, so I left it (as probably any other newbie would have). Puppy Linux proved more interesting. I ran into a few problems with GRUB, and installation wasn’t easy (despite the shiny install icon in the desktop), but I got it to run. Only to find that it had no compiler whatsoever installed by default. At least their community has found out about it, but as their solution was to install another .sfs file and make my clean install into a dev version, I declined.
Proves that some versions of Linux are more to my taste than others, as I was left with two distributions based on Slackware: VectorLinux Light 6.0 and Zenwalk 6.0. I’ve been trying to make my testing method as watertight as possible.
Both distributions were installed on the same machine. The hard disk was partitioned into two partitions: primary bootable partition formatted with ext3, logical Linux swap partition of standardized, (RAM*2)+10% size: 570 MB. During installation, no advanced options were considered, if possible, I just “clicked though it”. I then checked the distributions with version and uname -r to make sure what I had before me.
Just like Michael Weingärtner, I used two test tools to run benchmarks tests: GtkPerf and the Phoronix Test Suite. While GtkPerf obviously tests GTK+ performance, The Phoronix Test Suite (PTS) includes a wide range of test tools, of which I included 3:
- x11perf – a very basic performance/regression test for X.Org. I have used option 2: Scrolling 500 x 500px.
- byte – This test runs the ANSI C version of SciMark 2.0, which is a benchmark for scientific and numerical computing developed by programmers at the National Institute of Standards and Technology. This test is made up of Fast Fourier Transform, Jacobi Successive Over-relaxation, Monte Carlo, Sparse Matrix Multiply, and dense LU matrix factorization benchmarks. I have used option 1: Dhrystone.
- scimark2 – similar to byte. I have used the composite version of this test.
I compiled GtkPerf from source. PTS needed PHP ≥ 5.0, which I downloaded from php.net, configured it with –with-gd and then made and installed it. PTS itself only needs to be untarred, then run from the local directory (using ./phoronix-test-suite benchmark <name of test>). PTS then downloads and installs whatever it needs for the test (you are asked for your root password before doing this, obviously). In between each and every single test, I rebooted. This might have been a Windows-related knee-jerk reaction, but I wanted to have as few as possible processes running.
Zenwalk is version 6.0 w/ kernel 22.214.171.124
VectorLinux Light is version 6.0 w/ kernel 126.96.36.199
- Boot time test: Time from press of power button to fully loaded desktop (including typing user name and password). Obviously not the most precise of tests, but timed with a digital stopwatch at least.
Zenwalk: ~ 90 secs.
VectorLinux Light: ~ 75 secs.
- GtkPerf I: 3 consecutive runs with 100 test rounds directly after reboot.
- GtkPerf II: 3 consecutive runs with 100 test rounds directly after reboot with 3 standard applications running: word processor, image manipulation, web browser.
- Phoronix Test Suite tests, run directly after boot, reboot after each test (for all tests: higher figures = better performance).
|GtkEntry||0.14 / 0.15 / 0.14 / 0.143||0.13 / 0.13 / 0.13 / 0.13|
|GtkComboBox||5.20 / 5.20 / 5.30 / 5.23||5.60 / 5.62 / 5.59 / 5.603|
|GtkComboBoxEntry||3.52 / 3.44 / 3.53 / 3.497||4.83 / 4.82 / 4.84 / 4.83|
|GtkSpinButton||1.02 / 1.12 / 1.04 / 1.06||1.08 / 1.09 / 1.09 / 1.087|
|GtkProgressBar||0.92 / 0.86 / 0.86 / 0.88||0.78 / 0.79 / 0.79 / 0.787|
|GtkToggleButton||1.14 / 1.15 / 1.14 / 1.143||2.16 / 2.18 / 2.17 / 2.17|
|GtkCheckButton||0.59 / 0.60 / 0.68 / 0.623||1.91 / 1.93 / 1.95 / 1.93|
|GtkRadioButton||0.85 / 0.96 / 0.87 / 0.893||2.43 / 2.44 / 2.47 / 2.447|
|GtkTextView – Add text||3.51 / 3.51 / 3.51 / 3.51||3.15 / 3.18 / 3.18 / 3.17|
|GtkTextView – Scroll||0.91 / 0.91 / 0.92 / 0.913||1.81 / 1.83 / 1.80 / 1.813|
|GtkDrawingArea – Lines||1.50 / 1.44 / 1.44 / 1.46||1.15 / 1.15 / 1.15 / 1.15|
|GtkDrawingArea – Circles||1.86 / 1.95 / 1.96 / 1.923||1.55 / 1.54 / 1.54 / 1.543|
|GtkDrawingArea – Text||3.36 / 3.26 / 3.35 / 3.323||12.27 / 12.25 / 12.30 / 12.273|
|GtkDrawingArea – Pixbufs||0.58 / 0.68 / 0.61 / 0.623||0.68 / 0.69 / 0.68 / 0.683|
|GtkEntry||0.13 / 0.15 / 0.13 / 0.137||0.12 / 0.10 / 0.11 / 0.11|
|GtkComboBox||5.98 / 4.91 / 4.98 / 5.29||5.66 / 5.65 / 5.67 / 5.66|
|GtkComboBoxEntry||3.59 / 3.61 / 4.22 / 3.807||5.29 / 5.23 / 5.28 / 5.267|
|GtkSpinButton||1.04 / 1.04 / 1.10 / 1.06||1.06 / 1.06 / 1.06 / 1.06|
|GtkProgressBar||0.85 / 0.84 / 0.88 / 0.857||0.75 / 0.76 / 0.76 / 0.757|
|GtkToggleButton||1.13 / 1.17 / 1.14 / 1.147||2.16 / 2.16 / 2.16 / 2.16|
|GtkCheckButton||0.64 / 0.60 / 0.60 / 0.613||1.92 / 1.92 / 1.92 / 1.92|
|GtkRadioButton||0.87 / 0.87 / 0.88 / 0.873||2.52 / 2.52 / 2.52 / 2.52|
|GtkTextView – Add text||3.41 / 3.42 / 3.43 / 3.42||3.17 / 3.19 / 3.17 / 3.177|
|GtkTextView – Scroll||0.93 / 0.91 / 0.91 / 0.917||1.83 / 1.82 / 1.82 / 1.823|
|GtkDrawingArea – Lines||1.44 / 1.45 / 1.46 / 1.45||1.14 / 1.14 / 1.13 / 1.137|
|GtkDrawingArea – Circles||2.28 / 1.89 / 1.89 / 2.02||1.53 / 1.51 / 1.51 / 1.517|
|GtkDrawingArea – Text||3.65 / 3.27 / 3.26 / 3.393||12.32 / 12.33 / 12.28 / 12.31|
|GtkDrawingArea – Pixbufs||0.60 / 0.59 / 0.56 / 0.583||0.69 / 0.68 / 0.68 / 0.683|
Zenwalk 6.0 seems to be generally faster than VectorLinux Light 6.0, as it gets better figures for more test items than VL.
However, not all test items are equally important to the average user, and for a lot of test items, value differences are rather small. One value that seems of importance to me is obviously boot time (nobody likes waiting for his/her machine to start up).
One test item that strikes with significantly different values is “GtkDrawingArea – Text”. Now, according to the Gtk Class Reference, GtkDrawingArea is a widget that “is used for creating custom user interface elements”, in this case text. Newt’s comment on the old version of this post, has rightly pointed out that in order to rate significance of test items, one would have to find out what impact such a test item (in this case, the Gtk widget) has on desktop usage. And to be honest, I don’t really know (cf. below, “Note”). In retrospect, I think Gtk, as a library to build GUIs, isn’t exactly the perfect choice to test a distro for everyday use.
Unfortunately, the Phoronix Test Suite (which is more likely to include more relevant tests IMHO) ran into some problems during the install of a more universal testing suite, at which point I’ve simply given up and stuck with the 3 tests described above.
So, what’s the point to all of this? Firstly, I’ve had a first look at benchmarking, and I’ve come up with results. It proved that it’s probably more complicated to interpret those results than simply getting them. That’s why I’ve re-written this conclusion, which wasn’t much more than a late-night try to make VL Light look better than the figures would suggest, simply because I like it better (and it *feels* faster, and the handling’s really better, etc.).
Secondly, it shows that you should never copy a testing method unreflectedly and without further testing of the method itself.
I will perhaps try to run another battery of benchmark tests over both distros. Until then, please don’t be shy to comment and make suggestions.
Finally, I’d like to stress that I have little experience with benchmarking. Please point me to any mistakes I might have made in the process (e.g., are there more relevant values I should have tested?), especially in the conclusion!