Browser benchmarks 2: even Wine beats Linux Firefox

Mozilla

We posted yesterday about Firefox having very different JavaScript performance on Windows and Linux, despite being the same version of the software.

Some people have said that we should have used a stock build from Mozilla. (We disagree, because we'd argue that most Linux users use software from their package manager rather than downloading bits and pieces from the web.) Others have said that Opera should be tested. And some people have said that it's Nvidia/AMD/Intel drivers that are slowing down Linux.

Anyway, we thought we would conduct a couple more quick benchmarks to see whether we can eliminate some of these variants. We don't have time to run the full benchmark suite and fiddle around from scratch, so we ran just a few quick tests to see what we could find.

This time the information you need to know is:

  • These benchmarks were run on the same computer as before, running the same Fedora 10 install.
  • We tried Mozilla Firefox for Linux as downloaded straight from Mozilla.
  • We also tried Mozilla Firefox for Windows as downloaded from Mozilla, but running it using Wine on Fedora.
  • We installed and tested Opera 9.63 for Fedora 10, as downloaded from Opera.com. Note: we were only able to find i386 builds on the Opera site; this isn't optimal so if someone can point us towards an i686 build for Fedora 10 we will happily update the article.
  • We ran the Google V8 Benchmark suite V3, as before.

To be absolutely clear: we took the Windows Firefox and ran it on Fedora Linux using Wine 1.1.12 as provided by Fedora:

  • "Firefox Windows" is Firefox running on Windows.
  • "Firefox Fedora" is Firefox running on Fedora using the Fedora package.
  • "Firefox Mozilla" is Firefox running on Fedora using the Mozilla build.
  • "Firefox Wine" is Firefox as compiled for Windows running on Fedora using Wine.
  • "Opera" is, well, Opera 9.63 running on Fedora.

With all that in mind, here's how the results look now:

Firefox on Windows, Linux and Linux/Wine

Firefox running on Windows, Linux and Linux/Wine, plus Opera.

The end result: Firefox from Mozilla or from Fedora has almost nil speed difference, and Firefox running on Wine is faster than native Firefox. Opera lags behind, but we're inclined to believe that number would increase if a better build was used.

Is this the end of the story? Probably not - we expect some commenters will come up with some other reason why the slowdown is clearly Nvidia/ATI/someone else's fault. Go on, folks: try the benchmark yourself. If you're running Linux, install Wine and try Windows Firefox on Wine and make up your own mind.

You should follow us on Identi.ca or Twitter


Your comments

:-(

This is amazing. I'd never thought that wine+firefox would be faster than the Linux version. This is getting silly.

And, have you thought about benchmarking wine+google chrome?

Javier
P.S. Congratulations for the distribution rise.

What wrong with Fedora?

I ran the same experiment on my T7500 (Dell XPS) laptop with Arch Linux. Linux's Firefox vs Wine's Firefox and they get similar scores on the V8 benchmark, ranging from 170 to 180 when repeating the test.

Do other people get the same results as yours?
Is it a Fedora problem (kernel/libc version)?

Scandalous... O_o Someone

Scandalous... O_o
Someone slowdown Linux?

On my Athlon 64 X2 5000+

Running Kubuntu 8.10: Wine Firefox gets 161, Ubuntu Firefox gets 136. Lame!

Just shows how much Mozilla

Just shows how much Mozilla are bothering with optimization of the Linux build, I guess.

It's the compiler, stupid.

Recompile Firefox with the Intel ICC compiler and rerun the test.

Re: It's the compiler, stupid.

NO! The whole *POINT* of this is that people *DON'T* recompile Firefox - the just use the one from their distro. If Linux performance is dragging behind because we use GCC, then this problem is much MUCH larger than just Firefox.

Re: It's the compiler, stupid.

I think he was just suggesting a likely cause of the effect and an experiment to investigate it.

My experience with other

My experience with other programs has usually been the opposite. I am a free software developer, and the programs I make/help out with are somewhat faster on Linux than they are in Windows. However, I notice through casual observation that Firefox is slower and buggier in Linux than it is in Windows.

I'm pretty sure that this is just a consequence of the Mozilla devs creating a half-assed port in the first place. Firefox in Linux is bigger than the Windows equivalent, which is also somewhat unusual.

Compiler, Firefox Memory Manager, timer API

I do remember at some point the Linux-build didn't have the same memory manager as the Windows version, I don't know what the differences are now. The compiler could also make a big difference. The windows timer API is quiet different from Linux/Unix, maybe Firefox works better with the Windows API.

compiler

MSVC is a better optimizing compiler than GCC. It's been that way for rather a while.

Please recompile it

I would like to see this ran with a re-compiled Firefox using different compilers. The reason is that Wine ultimately uses the same resources as firefox. For wine version to be faster, says it is compiler. If this is gcc, then it is time to revisit it and find out why the large difference.

Pointing to Mozilla

This test makes me think that the difference IS a problem of the Firefox Linux code and not so much of Linux itself.
Looks like Mozilla doesn't really bother optimizing for Linux.
I've always been an avid Firefox user, but maybe I should review my options.

Graphics Drivers

Is it possible to try out the benchmarks on a windows machine with the default windows drivers installed instead the Graphics Card provider's one?

What Desktop?

What desktop are you guys using? Gnome, KDE4 or KDE3(unlikely since Fedora 10 doesn't ship with it)?

Why not try an older version like openSUSE 11.0 with KDE3?(11.1's KDE3 isn't as fully KDE3 as 11.0's).

So many are complaining about video drivers, and most are running KDE4. I have no video driver issues running openSUSE 11.0/KDE3 under nVidia, ATI, or Intel, so that would be a good test IMO.

Swiftfox

Try swiftfox (if you'r lazy). It rox, i get almost same results on wine as on swiftfox, but, it should be better anyway. Take care.

Please keep trying :)

Yes, please do blame anything & keep benchmarking!! I'll love to install the final, quicker, Linux Firefox solution once its discovered :)

I do hope the Mozilla chaps are reading this, maybe they can locate the problem!

Food For Developer Thought

Why do you dismiss this by saying to recompile?

Why should the user compile anything?

Why didn't Mozilla choose to compile their own crown jewel in the "proper" manner?

Why did Mozilla choose to compile it one way for Windows and another way for Linux?

Why don't you place blame where it belongs?

The issue here is not that the user should recompile what is now a standard issue piece of software. The issue is why didn't the software manufacturer do it correctly in the first place.

If you replaced the word Firefox with Internet Explorer in this story, who would you blame? The user or the developer?

"Food for Developer Thought": Dude you need a cookie.

Nobody is asking the end user to recompile anything. People are suggesting a recompilation in an effort to narrow down the source of this issue collaboratively. Once we know whom or what to blame maybe we can work together to fix it too. But we need to figure it out first and the developers have to do this. And developers recompile; a lot.

Where does this difference come from?

I think it makes sense to think about where this difference comes from. Some thoughts about possible and impossible reasons:
* Anything used both by native Firefor and Firefox on Wine can't be responsible (e.g. the slowness clearly isn't due to X, because X is used by Wine as well).
* If the slowdown is graphics related, maybe the used toolkit (GTK I think) is responsible for the slowdown. If this is the case, every app using gtk on Linux and native GUI on Windows should be slower on Linux than its Windows equivalent. If that is the case, optimizing GTK (or switching to another, faster toolkit) would be the solution.
* If the slowdown is compiler related, the solution is to get better compiled executables. Maybe the solution would be as simple as getting the vendors to use the appropriate options; I thing many GCC users don't know that if they don't give appropriate options, GCC still creates 80386 compatible code, and therefore won't take advantage of any of the great features newer processors offer. I don't think there are many Firefox users who still use an 80386. But maybe GCC just doesn't produce as fast code even with the right options. In that case, one solution would be to compile with another compiler (ICC, LLVM). Another solution would be to make GCC optimize better.
* Another possibility is that the standard library supplied with GCC/Linux is slower than the standard library supplied with MSVC/Windows (this of course only applies if Firefox makes noticeable use of the standard library). In that case, the solution of course would be to improve the standard library.
* Yet another possibility would be that the Linux calling conventions are inherently slower than the Windows calling conventions. If that is the case, the only solution would be to get better calling conventions for Linux. Probably those different calling conventions would be enabled only through options, or by function attributes, in order to not break backwards compatibility.
* There's also the possibility that the Linux related Firefox code is just not as well hand-optimized as the Windows related one. The correct solution to this is obvious.
* Related, but different is the possibility that the general code is better tuned for Windows. This need not be deliberate; it may just be that most profiling is done on Windows, and changes which improve the speed on Windows actually degrade Linux performance. The solution to that would, of course, be to have more Mozilla profiling done on Linux.

OK, that's all possibilities which come to my mind. There's most probably some possibility I've forgotten, though.

In other news: How does IE6/7/8 compare?

So, I understand that this might be a tangent. But I compared results of these benchmarks on Firefox with IE6 both running natively under Windows. The result was that IE6 sucks balls. Firefox score was above 200 and IE6 score was below 50. Is this the general experience?

It's the compiler, stupid. Pt II

I am not asking you to recompile it and try the test again because you are a USER, I am asking because you are the one running the BENCHMARKS.

You know, part of being a user of open source software is contributing back. These benchmarks may help us make Firefox faster, but you have to be less arrogant and obnoxious towards those of us trying to DO something about it. I am asking you to recompile and retest so that we can try to narrow down where the problem lies.

Of course, if you are simply doing this to attack Linux developers without consideration for anything else, then why are you posting these benchmarks?

Surprised ?

Hello,

Thanks for those tests confirming what I felt as a multi boot user very linux enthousiast.
I have used almost every Ubuntu version (5.04-9.10) and every Firefox that came with it. Same on my gentoo and debian boxes and iceweasel. And I have always felt firefox being slower on linux than on windows XP, no Vista at home, windows 7 beta shows same feeling.
I do not have the sufficient knowledge to guess where it comes from. I doubt it does from the driver, firefox code (which is pretty much the same as on windows)...
I want to say to the compiler mob : Stop blaming the build or the compiling options, they do not belong to firefox but to the distribution. And accross distribution it seems there ain't much difference.
I want to say to the IE mob : What is IE ? Are there really serious people still using this crap ?

Java

Aren't all of these tests based on Java? Shouldn't you check what versions of Java each browser is using? Maybe the Linux-native Firefox is using the OpenJDK JRE or something.

Fix the code

Here's a thought .. Instead of going off on why people should recompile with this flag, defending open source, or making excuses because a particular library is bad - Fix the code!

Re: Java

its Javascript not Java, too many people get the two confused.

I guess what most people

I guess what most people saying recompile really want is to answer this question:

-Why is Firefox in linux slower?

That's just it! Stop saying it is or it isn't firefox's fault, just try other compiler / other distro / whatever other bright idea to zoom in on the issue. Unless this post was just to generate flame wars.

So please, stop with the hypotheticals and just help figure the answer out.

RE: It's the compiler, stupid. Pt II

"You know, part of being a user of open source software is contributing back."

Thanks to the guys from tuxradar for an insightful look at my favorite browser. I think I'm switching to competition though. Opera maybe. But before I do that, I'm recompiling and will run the benchmarks later today. I will post it in my bulletin board so that I can contribute back to the linux community in my country. Oh, and a reminder, those of us who are actually DOING something about it are not asking __other__ people to recompile or run benchmarks. Of course everybody has to contribute something back because, otherwise, newbies might think that there's a Linux customer service!

Could the person who scans

Could the person who scans the firefox source for security holes also please make sure the code is optimized during todays pass?

All too slow...

Ok, this is serious and not a troll comment to pit one OS off of another, it is just meant to show that there is still quite a bit of work to do in Firefox. My system is a MacBook Pro from almost 3 years ago (2.16GHz Core Duo (not C2D) with 2G RAM (667MHz) on Mac OS X 10.5.6) and running the latest dev version of Safari (Webkit) I get a score of 1090. Yes, one thousand ninety. Consistently. Running an optimized build of FF3.1 beta (Shiretoko) using the new tracemonkey I get a consistent score of 109. Yes, one hundred nine. That is pitiful. Come on Mozilla team. Let's get this fixed.

Looking at the SwiftFox build of FireFox...

I would tend to think it was some kind of optimization issues. While I have tried it, I notice that it is only at verion 3.0.4pre1, so any real comparison would require getting a "stock" build of FF 3.0.4pre1 (or waiting for SF to catch up).

other distro test

pls test, mozila on diferent distros, fedora is know to be slow distro. diferent companies compile their distros diferent way. when you run such test dont put "fedora = linux" mark pls.

Compiler

Good afternoon,

All the gents asking for ICC and GCC -03 recompilizations are asking so that the results can be submitted back to Mozilla. Mozilla will fix their linux compilation guidelines, and distros will inherit the changes.

We're not saying that linux 'is' faster. We're trying to "make linux faster", and submit the methodology upstream. This is how free software works.

A warning against ICC. It's not a full drop in replacement for GCC. Expect thar to be dragons. For easy experimentation with GCC cflags, ping any gentoo-er to do the work for you.

Frank

MUHAHAHA

I want to see how Chrome performs if and when Google bothers to release it for *nix...

Funny thing is...

My results:

Firefox Windows Parallels Desktop V4: 192
Firefox Windows VMWare Fusion V2: 188
Firefox Windows wine (Crossover V7.1): 162-189 (big run-to-run variation)
Firefox Linux (Ubuntu 8,10) VMWare Fusion V2: 172
Firefox OS X (10.5.6) native: 128

All on the same machine (2.4GHz C2D). All version 3.0.6. Will you still complain about Firefox performance under Linux? :-)

Oh, and BTW:

Safari OS X (10.5.6) native: 189

Other speed issues

I think the Mozilla team should focus in more serious speed issues (for example the *lame* scrolling).

A 20-30 points difference in JavaScript performance does not bother the end user so much, there are many more serious slowdowns/glitches I experience in Linux when comparing to Windows or even OS X.

Somebody benchmark Arora Browser.

I have no idea how this is possible but i am getting this result :-

Score: 1036
Richards: 2168
DeltaBlue: 1208
Crypto: 2005
RayTrace: 705
EarleyBoyer: 1954
RegExp: 171

Also slower on Gentoo AMD64

I get 288 with Wine and 251 with native Firefox in Gentoo AMD64, and on Gentoo everything is compiled from source on the user's machine.

Arora and that score...

If you had read my post above "All too slow..." you would see that I am getting that kind of score as well from WebKit (the dev version of Safari) on Mac OS X. Arora is based on WebKit. That is how much faster SquirrelFish (WebKit JavaScript engine) is than TraceMonkey. Really shows how much work Mozilla has to do on FF.

gentoo firefox: 205 wine

gentoo firefox: 205
wine firefox: 223

compiler (most likely) has nothing to do with it

I don't think the compiler can possibly have that huge of an effect. There is only so many ways you can write an optimizing compiler and gcc is already a very strong optimizing compiler. It has be something with the way Mozilla is optimizing their code on Windows.

Anyway to definitely rule out the compiler make sure you can compile with icc as others have said. Maybe also test Firefox 2 or Firefox 1.5, which are compiled with gcc on all platforms (including Windows).

What this test does that the other test didn't is mostly rule out the kernel. Very good. So lets get to the bottom of this.

Food for FanBoys

I hate these users. You guys make me want to take a shovel to your heads. You defend a brand as if it were a member of your family. DIE!!!

Fan boy bitches, get a life! You have no idea how damn annoying you guys are. I would love to meet one of you in real life and take my fist through your head.

compiler may have sth. to do with it

To the poster who said that "There is only so many ways you can write an optimizing compiler". It is exactly the opposite. It is actually a fun fact - it is proven that there are infinite number of ways to write an optimizing compiler.

is slower.. and.. what?

Considering the amount of resources used by firefox once you install 6 or 7 extensions.. do you really care about it?

Btw according to the topic it's a javascript performance issue, so what about slower linux calls, or worse compiler, or bad port, or.. .... ??
Mozilla people want better results, then focus on a native/optimized version for linux, upgrade the TK used by firefox (maybe you need to switch after all to QT, who knows), make a linux specific javascript engine, and stop using copy/paste from the windows-optimized version (of course it should work better on windows..)

Still slow? buy a better computer.. more ram, powerful cpu(s), raid0 disks,... that's it

Regards

It's easy to blame anyone...

What if the problem wasn't ATI/NVIDIA, but rather Linux/X/QT-GTK itself ?

... Think about it ...

Are you really suggesting

Are you really suggesting that one set of tests from Google is representative of Firefox's JS performance? Unless you also publish SunSpider and Dromaeo results, I'm gonna assume you cherrypicked the one set of results that gave you what you wanted and ignored the other two more credible test suites.

Would like someone to clarify..

.. where exactly is for sure, the point of failure in the performance drop between Firefox linux and Firefox Wine-on-linux.

Who cares? What does this

Who cares? What does this prove?

What are the units for the test?

architecture problems are obvious

X server? lose it or stay in the 90s.

Linux has some really crap legacy architecture (some take pride in saying that the x11 server is as old and even more complex than the kernel.. wtf?) and every sane programmer knows windows development is just so much faster because the platform is mature.

MATURE. and yes a lot of linux architecture is really basic and some also plainly WRONG.

but the low level hacko coders just dont see it.

and does who do face strong opposition from the whackos.

linuxhater was/is right, but learn to take criticism and find something good in it.

The Browser war..

that is going on now is all about speed.

Features across all the major browsers are pretty much in parity on a general level and apart from a few things speed is the final defining area of play left for browsers.

So when Mozilla release a browser for linux which renders content using linux system calls more slowly than emulating windows api calls in to linux system calls, then there is a big, big problem.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Post new comment

CAPTCHA
We can't accept links (unless you obfuscate them). You also need to negotiate the following CAPTCHA...

Username:   Password:
Create Account | About TuxRadar