Benchmarked: Firefox Javascript on Linux and Windows - and it's not pretty

TuxRadar

As you might have guessed from our domain name, TuxRadar.com, we're big fans of Linux. But being a fan of Linux doesn't necessarily make you a Linux fanboy - the kind of person who blindly ignores anything negative about their passion of choice as if that somehow made it better.

In fact, we think more Linux users need to admit there are some places where Linux isn't quite as good as its competitors. And one place where that's certainly true is in web browsing. Don't believe us? We did some simple JavaScript benchmarks of Firefox 3.0 using Windows and Linux to see how it performed across the platforms - and the results are pretty bleak for Linux.

You might be wondering, "why JavaScript?", and there's a simple reason: online apps such as Google's Gmail and Google Docs, Apple's Me.com, and of course ever-growing (and increasingly popular) JavaScript libraries such as jQuery, Prototype and Scriptaculous. These rely on browsers executing a huge amount of JavaScript, which means browsers have had to put a lot of working into optimising their JavaScript performance.

Proof of this can be seen by the number of JavaScript benchmarks that have appeared over the last few years - in fact, the WebKit team, the V8 team and the Mozilla team have all produced their own benchmarks, so no one can argue that JavaScript performance isn't important.

We wanted to find out just how well the Linux version of Firefox faces up against its Windows equivalent, so we ran the benchmarks from all three browser makers to make sure we got a full range of results. Some specifics you probably want to know:

  • We used Firefox 3.0.6 running on Windows XP SP3 and Fedora 10. Both were running in 32-bit mode.
  • Our test machine was a quad-core Intel Core 2 running at 2.66GHz, with 4GB of RAM and 1.5TB of hard drive space.
  • The three tests we used were: SunSpider, V8 Benchmark 3 and Dromaeo (using "Recommended Tests").
  • We have marked in red where Linux performance falls behind Windows for a given test, and vice versa. In short, red numbers are the ones that scored lower.

SunSpider

Uses the SunSpider benchmark from WebKit.org. Results in milliseconds; lower is better.

Firefox Windows vs Firefox Linux: SunSpider test

SunSpider (lower is better)

Windows 

Linux 

Difference 

Total 

2478.6 

2786.6 

308.00 

3d 

282.4 

344.4 

62.00 

Access 

410 

402.6 

7.40 

Bitops 

312.8 

266 

46.80 

Controlflow 

32.2 

34.4 

2.20 

Crypto 

168.6 

168.8 

0.20 

Date 

212.4 

326.4 

114.00 

Math 

288.8 

321.6 

32.80 

Regexp 

181.8 

226.6 

44.80 

String 

589.6 

695.8 

106.20 

V8 Benchmark Suite

Uses the V8 Benchmark Suite v3 benchmark from Google. Results are measured against a reference system with score 100; higher is better.

Firefox Windows vs Firefox Linux: V8 test

V8 (higher is better)

Windows 

Linux 

Difference 

Total 

241 

182 

59.00 

Richards 

243 

193 

50.00 

DeltaBlue 

298 

238 

60.00 

Crypto 

164 

166 

2.00 

RayTrace 

272 

158 

114.00 

EarleyBoyer 

317 

246 

71.00 

RegExp 

193 

122 

71.00 

Dromaeo

Uses the Dromaeo benchmark from Dromaeo.com. Results measured in test runs a second; higher is better.

Firefox Windows vs Firefox Linux: Dromaeo test

Dromaeo (higher is better)

Windows 

Linux 

Difference 

Total 

50.33 

36.97 

13.36 

3D Mesh Transformation 

94 

70.64 

23.36 

3D Raytrace 

56.52 

44.88 

11.64 

AES Encryption/Decryption 

29.44 

30.31 

0.87 

Arrays 

122.17 

86.72 

35.45 

Base 64 Encoding and Decoding 

40.42 

37.28 

3.14 

Bitwise And 

126.97 

101.82 

25.15 

Code Evaluation 

168.66 

115.89 

52.77 

Compute Bits in Byte 

32.07 

44.4 

12.33 

Compute Bits in Byte (2) 

38.02 

53.36 

15.34 

DNA Sequence Alignment 

83.23 

68.94 

14.29 

DNA Sequence Counting 

35.01 

27.28 

7.73 

DOM Attributes 

67.89 

39.93 

27.96 

DOM Attributes (Prototype) 

95.81 

60.28 

35.53 

DOM Attribute (jQuery) 

73.21 

46.25 

26.96 

DOM Events (Prototype) 

58.45 

36.57 

21.88 

DOM Events (jQuery) 

42.97 

27.11 

15.86 

DOM Modification 

77.79 

55.5 

22.29 

DOM Modification (Prototype) 

37.38 

24.31 

13.07 

DOM Modification (jQuery) 

55.21 

39.58 

15.63 

DOM Query 

283.52 

170.43 

113.09 

DOM Style (Prototype) 

59.7 

40.28 

19.42 

DOM Style (jQuery) 

40.94 

25.79 

15.15 

DOM Traversal 

15.92 

9.51 

6.41 

DOM Traversal (Prototype) 

56.27 

33.83 

22.44 

DOM Traversal (jQuery) 

28.37 

17.33 

11.04 

Date Formatting 

107.57 

72.26 

35.31 

Date Formating (2) 

76.08 

43.85 

32.23 

DeltaBlue Constraint Solving 

39.89 

33.73 

6.16 

Fannkuch 

45.14 

58.28 

13.14 

MD5 Hashing 

78.67 

75.85 

2.82 

N-Body Rotation and Gravity 

45.62 

31.68 

13.94 

Partial Sum Calculation 

139.91 

103.61 

36.30 

Prime Number Computation 

41.75 

45.57 

3.82 

Prime Number Computation (2) 

42.8 

46.27 

3.47 

RSA Encryption/Decryption 

8.11 

8.12 

0.01 

RayTracer 

2.58 

1.63 

0.95 

Recursive Number Calculation 

94.87 

89.82 

5.05 

Regular Expressions 

19.33 

15.12 

4.21 

Richards Benchmarks 

64.14 

51.77 

12.37 

Rotating 3D Cube 

72.84 

62.25 

10.59 

SHA1 Hashing 

74.52 

72.64 

1.88 

Script Unpacking 

7.74 

9.51 

1.77 

Spectral Norm of a Matrix 

69.99 

69.44 

0.55 

String Parsing and Searching 

4.12 

2.87 

1.25 

Strings 

130.57 

116.57 

14.00 

Tag Cloud Creation 

62.17 

43.17 

19.00 

Traversing Binary Trees 

57.03 

42.3 

14.73 

Trigonometric Calculation 

76.27 

83.65 

7.38 

Validate User Input 

51.26 

41.41 

9.85 

 

Answers to some predictable comments

Why didn't you use Firefox 3.1?

We tried using a nightly build of Firefox 3.1 to see how performance might change in the future, but it locked up while running the Dromaeo tests so we opted to leave it for now. To be fair, the browser is still in beta, so it wouldn't really be a good test.

 

What compile options did you use for Firefox?

We, almost certainly like the majority of Linux users, don't compile Firefox ourselves - we used the stock build from Fedora i686. You could probably squeeze more performance out of Firefox by compiling it yourself using some specific options, but then that would hardly be a real-world test, would it?

Seriously, if you think the excuse "but you didn't compile it yourself!" is worthy, STFU and GTFO.

 

Did you test Vista?

No. We used Vista in our last benchmark, and it wasn't terribly pleasant, so we're trying XP instead. Some Linux fans might complain that Vista is (reported to be) slower than XP and thus would yield lower results, but according to our site logs 67.48% of our visitors use XP so benchmarking on Vista would also not be a fair real-world test.

 

Why is Linux slower?

A quick Google can show you that Firefox/Linux performance problems are nothing new, but usually these lie in graphical areas such as slow page refreshes or scrolling. With infamous threads such as this one around, and OpenOffice.org still performing much better on Windows than it does on Linux, it seems more likely to us that the performance problems seen above are less Mozilla's fault and more down to general performance issues on Linux.

Clearly this is an area that's always under construction - people like Michael Meeks take performance seriously, and we're optimistic for the future. Still.

 

I ran those benchmarks myself and Linux was 1,000,000 times faster than Windows!

Uh-huh. Please don't post this kind of nonsense unless you provide the hard figures from the benchmarks and your system specs and your distro - something like the output from "uname -a" ought to do it.

 

These benchmarks are artificial and thus worthless - you eeediots!

Actually, they were designed to test the kinds of things that web pages really do. The Dromaeo test is particularly impressive, building upon Prototype, jQuery and more to produce tests that are designed to reflect real-world JavaScript usage.

 

Hah! This is final proof that LinSUX is a WASTE OF TIME and all LinSUX users are teh GHEY!!!1

Our other box is your Windows box. Now go away.

 

This is irrelevant - I don't care how fast JavaScript is!

That's fine - maybe you have a really fast computer, maybe you avoid JavaScript-heavy sites, or maybe you're just really patient. But performance matters to us, and it matters to a lot of other people too - particularly when we're talking about the same software on two competing operating systems.

Think about it: if Firefox is slow because Mozilla's Linux code is weak, then it's a small (if irritating) problem - perhaps the Mozilla developers are more interested in optimising for Windows users simply because there are more of them. But if it's slow because or GCC or some other more general reason, then it's likely to affect a much larger part of your desktop. If you really don't care about performance, why are you reading this in the first place?

Remember, It's hard to be sure whether it's Firefox or Linux that's to blame, but we're hopeful someone will pick up the baton and give Firefox on Linux a kick in the backside to get it moving faster regardless of what the cause is.

 

This is irrelevant - Linux is better because it's freeeeeeeee!

Yes, we know. There's a reason we use Linux, after all. But given how many computers ship with Windows pre-installed and without an option to get it Windows-less, Windows is effectively $free too.

And yes, we also know you can try declining the EULA and clawing cash back from the manufacturer, but very, very few people do that. The rest of us buy a computer with Windows, blank it, then put Linux/BSD/MikeOS on.

 

You must hate Linux to post this kind of FUD

We admit the results aren't pretty, but we invite you to reproduce the tests yourself and see how you do - we've provided all the links, so please feel free to report your results below.

Update: even Wine beats Linux Firefox - click here to read

You should follow us on Identi.ca or Twitter


Your comments

I have made a "test". I had

I have made a "test". I had to help to my boss on his laptop. He wanted to check his emails in his box. Its a javascript-fulfilled website. In the Fox on XP the loading time was 13-14 secs. On Ubuntu (basic install) running in VirtualBox on the same XP it was 6 secs. On my laptop in the Fox running on Ubuntu 8.10 it was 3 secs. Remarkable difference, isn't it?

Compiler, compiler, compiler

You did not compare apples to apples, but also means that we need to be fairly reasonable.

The compiler used in Firefox Winodws build have a bigger inherited level of optimizations. The same issue appears also in OpenOffice.

Anyway, your machine is a 64 bit capable, and for the sake of comparison, the 64 bit binaries in computational intensive tasks gain between 10-20% in real world applications. So you may get the same improvement using the 64 bit version of the distros. In real life there is no reason why you shouldn't use a 64 bit version. Also, the Firefox builds are 32 builds, so they will not give any optimizations.

What should be done in future to not make the difference so big? Mostly the distro makers should use Profile Guided Optimizations using the most used sites: gmail, ymail, and JS benchmarks. This will get a win of 5% overall (it may be as 15% in 2-3 cases) but will reduce the gap. As most Distro makers do not do that, if I would have your machine and I want Linux and speed, and no head hassle, for sure I would pick a Linux 64 distro!

Memory is the difference !

I find on Windows where btw, its faster ?, is because Windows up-front-loads the memory. Fact, XP, 512mb ram if you are real lucky after the "up-front-load" will retain maybe 128mb for whatever use you can find for that much memory today. Windows is faster if you are runnng with 2+gb ram, because 1gb or less slows down after "up-front-load" because the swap takes over while you continue to work. Linux while faster where it counts, Windows can have the 50+ms - 200+ms, because time I have for anything that works well as Linux. Comming from Windows I don't feel now after 6+ years, the need to go back.

Article rendered outdated by Ubuntu 9.04 final

I just tested FF 3.09 on the newly released Ubuntu 9.04 (32 bit) and on XP. I dual boot both OSes on a PC running an Athlon XP with 1GB Ram. I used Sunspider.

Results -

FF 3.09 on Ubuntu 9 - 7228.0ms +/- 2.4%

FF 3.09 on WinXP - 7491.8ms +/- 4.4%

FF 3.09 running natively on the new Ubuntu clearly beats the same version running natively on XP SP3.

Performance issues with Openoffice.

Try turning off java in Openoffice 3.0.
It's the first thing I do with any new openoffice install nowadays.
For some reason Java on linux is a big, ugly hulking beast. Ridding oneself of it leads to a much more pleasant user experience.
Obviously you can't do that with firefox and remain practically; it's why I look forward to 3.5's new speedier engine. Still slower than the windows version, but a lot faster than what we've got at the moment. Or chrome. Mmmm...webkit + process isolation = delicious.

Did You forgot something?

I hope that this test was made on the freshly installed windows and ubuntu. Well, You should wait till xp gets 5 to 10 Trojans, and mess up the registry and fragments the disk, than you should make this test once again.

ff is slow. period.

FF aren't very reliable. The browser is rather slow to begin with (a bit slower than IE8 in Windows, and MUCH slower than Opera). That said, I would rather see Opera performance test in Win vs Lin

Firefox still slow

I'm learning jQuery, using my Ubuntu system with a local Apache server, MySQL, etc. I have a very simple web site I'm working on, and I using a jQuery function to show/hide a section. Firefox is embarrassingly slow (3.0.14 from Ubuntu repository).

Opera is much faster, as is Epiphany. Using Windows, Firefox is fine.

Firebug is probably the only reason I even use Firefox on Ubuntu, as much as I hate to say it - I really like Firefox, and have used it for a long time, especially on Windows.

There was java benchmark at

There was java benchmark at Phoronix and Linux outperformed Windows. It is known Firefox is "broken" under Linux, because it isn't optimised same way as it is under Windows. The same about OOo. Graphic stack is another story. If you would compare apps which are identicaly optimised for both systems Linux should easily win.

include Ratio for comparison

While difference column does indicate slowness, it does not convey the notion of relative slowness independent of size of data.

Include Ratio as an additional column for more insight.

A benchmark with a larger payload, may show a larger difference, than a benchmark with a smaller payload that is less intensively tested.

ex
benchmark1 100s vs 150s difference is 50, ratio is 1.5
benchmark2 10 vs 50 difference is 40, but ratio is 4
the slow down is more severe for benchmark2

My slow Linux Firefox is watching your fast Winbot-Firefox

.. and with all the news, it only gets more exciting! Go bot-racer Go!

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Post new comment

CAPTCHA
We can't accept links (unless you obfuscate them). You also need to negotiate the following CAPTCHA...

Username:   Password:
Create Account | About TuxRadar