Viewing Story

Browser Benchmark Wars: How does Firefox 4 Beta 9 stand up?

Posted by on 17/01/2011

2
3
0

With Firefox 4 beta 9 impressing the crowds, we decided to do a benchmark test of all the major browsers on the market to see who comes out on top.

The contenders in this event include Firefox 4.0 B9, Firefox 3.6.13, Opera 11.00, Chrome 8.0.552.237, Safari 5.0.2 and Internet Explorer Beta 9. We did many tests on the JavaScript and memory management of each browser. We also initially decided to test CSS load times and site load times, but found the differences to be insignificant with too high a margin of error. At the end of the day JavaScript is responsible for a number of operations on any site including display of ads, call functions from other sites, some graphical management and much more. Programmed right, it can do pretty much anything these days. JavaScript can also be very heavy on a browser if the engine cannot properly manage it.

The system used to conduct these tests on was a Dell XPS 164 with a 2.8Ghz Duo-Core processor, 6GB DDR3 RAM and a 512MB ATI Radeon HD 3670 mobility card. The benchmark results will obviously vary from computer to computer depending on its overall computing and graphical power, but that does not take away from the validity of these tests. All non-essential plugins were turned off during the test. Finally no one test can can be compared with the other, since they test different aspects and may have opposing results.

With regard to IE beta 9, it still seems to be rather buggy with some functions just not working all together, so the results may need a bit of deviation (not sure how much) to compensate for the lack of processing of some JS functions.

SunSpider JavaScript Benchmark

This is the industry’s best JS benchmark tool because it tests a wide variety of JavaScript functions rather than focusing on one particular area. Also it tests on the JS platform and not other elements such as DOM, showing the browser’s true JavsScript potential. The lower the score, the better.

Firefox 4 beta 9 wins this category, showing the true potential of its JägerMonkey JavaScript engine. Not surprisingly, Opera and Chrome are just behind with a score of 307 and 309ms respectively. Firefox 3.6.13 seems to be the only browser that does not do well on this test, which explains why it sometimes can be laggy in everyday browsing scenarios.

Kraken JavaScript Benchmark

This test is a modified version of the SunSpider tool with enhancements. It is much more intensive as well. Once again, the lower the better.

Firefox 4 beta 9 was the clear winner here. All browsers seems to function well during the test but IE slightly froze a number of times, hence its poor results.

V8 JavaScript Benchmark, Version 6

This script comprises of a number of various JS test modules developed by various coders. The separate scores are then compiled and averaged (on a proportion basis) to give a final overall score. This script also focuses only on the JS aspect of the browser. The higher the score is, the better.

Chrome beats the competition coming out on top with score of 4594, while Firefox 4 beta 9 comes in second much lower at 3304. Shamefully, Firefox 3.6.13 comes in at a very low 493. Even Internet Explorer beats it at 1218. During this test, Firefox 3 basically became unresponsive at one point and displayed the JS force close option, but I continued with it till the test was done, which may explain why it crashes so often in real life. IE9 also displayed some sticky behaviour throughout the test.

Peacekeeper

Peacekeeper measures your browser’s performance by testing its JavaScript functionality. By measuring a browser’s ability to handle commonly used JavaScript functions Peacekeeper can evaluate its performance. Peacekeeper focuses more on the visual rendering aspect of JS, so the scores may go in a completely different direction here. The higher the score, the better.

Chrome comes out on top here followed behind by Opera 11. Not surprising. On the site, Opera 11 seemed to be the highest performer among all the aggregated scores. Note that this test also depends on the tester’s GPU and CPU capabilities, so the scores can go any way depending on the machine. IE9 unfortunately just didn’t work, I am guessing because of incompatibility with the test. Firefox 4 does not do too well, even though Mozilla pretty much boasted about its 3D and graphical capabilities.

Memory – 1 Tab Open

Moving on to a basic memory test, we wanted to see how much active memory it used after a 5 minute period with one tab open. We left it for 5 minutes so it could get past that cold start period and settle down. We loaded up our site for the test.

Disappointingly, Firefox 4 does not do well. It used 136MB of active RAM while Opera surprisingly came in at 111.2MB. IE9 used the lowest but some of the functions seem to just not work so we are not giving it full credit just yet. Otherwise, Safari seems to be the winner here based on it being a stable release. Chrome is not far behind.

Memory – 5 Tabs Open

We opened 5 tabs at let it sit for 5 minutes to stabilize. The sites we used were BlogTechnical, TechSpy, Newgrounds, BBC and Facebook (signed in home page).

Once again, Firefox 4 pretty much lost. Firefox 3 did pretty well in comparison to Chrome though. Browsers like Chrome and IE9 cannot be compared directly to Firefox and others because they split up their processes on a per tab basis. Each tab has its own process, so if one tab happens to crash, it doesn’t affect the others. In total, it may use a lot of RAM but by itself, it is very light.

Final Thoughts

Working backwards, Firefox 4 definitely doesn’t solve its RAM problems and can pose a problem for those who leave their browsers open for very extended periods of time. If it keeps adding more active memory over time like Firefox 3, then it may get very laggy at some point. Chrome and IE9 may use a lot of memory, but because of the split processes, the load per tab is significantly less thus preventing the chance of a crash and if it does crash, it protects the other tabs from fall prey to force closing. Firefox also needs to work on the 3D JS management because Chrome and Opera are still miles ahead.

Talking about the overall JS functions though, Firefox 4′s new engine definitely has shown its potential by winning the first 2 tests and coming in a good spot for the 3rd.

To say there is a clear winner here would be very difficult but the main battle stood between Chrome, Firefox and Opera with each of them excelling in various arenas. Firefox may be a very popular browser but if they don’t watch their backs, Chrome will easily be knocking on their doorstep. Firefox has always been popular because of its extensive list of add-ons but with most of them now being ported to Chrome it may be at risk of staggering on the market.

JavaScript has become quite an important tool over the past year with many sites using different forms of it, especially AJAX to improve users’ browsing experience. CSS is obviously important too but because the results fluctuated so much, it was hard to say how well they performed side by side.

The sad thing though is that users may get completely different experiences on their own systems, so it’s about finding which browser works best for you. All the browsers tested here seem to have a very similar visual interface, and the times have come now where the most popular browser will come down to which performs best over which offers more features.

Hope you all enjoyed.

2
3
0

Related Posts

No related photos.

  • http://www.facebook.com/people/Michael-Munnett/100001135990154 Michael Munnett

    Opera is far more efficient & loads far faster & crashes far less then the rest of the other competing web browsers from my own personal experiences running it on the dell dimension 3100 computer im using with a 3.2ghz pentium 4 ht processor with 504mb of ddr2 400mhz of ram with 71.1gb of harddrive capacity running windows xp home edition system pack 3.

  • http://twitter.com/dmandelin David

    Thanks for benchmarking us!

    I tried to duplicate your 1-tab test, and I got much lower memory usage numbers. Could you give me some more details on the methodology and exactly what value you are measuring? (Or just get in touch so we can chat about it.) We’d love to fix this, but we can’t do that unless we can really see the problem?

  • https://www.blogtechnical.com Bradley Wint

    I’d say my memory tests weren’t the most methodical… just killed all unnecessary processes on the machine, disabled unnecessary plugin and then launched each browser by itself, let it sit for 5 minutes and then measured it with Performance Monitor, nothing too complicated. Funny enough, when I ran it on two machines with much less RAM capacity, the total active memory was much lower :/ Honestly not sure why. I also wanted to do a cold start and warm start test but didn’t get around to it. The straight truth though is that even though it looked like it used a lot of active memory, it didn’t feel like that at all. It was very responsive and even on the crappiest of computers it ran just as smooth as chrome and opera. I was finally glad to switch it back so my parents could use it instead of Chrome.

  • Anonymous

    In my own js memory benchmark, FF4b9 is in leading place together with Chrome 9. Opera and Safari are a little bit slower, and FF3 and IE 8 hopeless. Have no IE 9 results yet:
    http://www.chr-breitkopf.de/comp/jsmem/results.html

  • https://www.blogtechnical.com Bradley Wint

    Thanks a lot for that as well. I wanted to test IE8 as well just for comparison sake, but when I installed IE9 beta, it replaced IE8. Nonetheless I knew it would perform horribly. I am glad Firefox’s JS engine is so super fast now. I honestly am not too concerned about the memory hogging because it never bothered me. Some days it would sometimes clock 1.5GB of active memory cause I leave it open for maybe 5 days at a time, and it only got slightly laggy. The JS engine in FF3 was slow though and loading sites with Google Adsense or other slow JS scripts kinda had its effect, even on a 10MB cable connection. Im on the beta now and still wont be heading to Chrome or Opera.

  • Anonymous

    It wouild have been a more interesting race if you had included Maxthon 3. I ran my own tests on an off-the-shelf HP with these results:

    “V8 Celtic Kane Sunspider “Kraken
    Chrome 6,124 803 216.8ms +/- 0.7 11,588.9ms +/- 0.2%

    Maxthon3 4,636 786 215.1ms +/- 1.1 1 3,513.9ms +/- 0.3%

    Firefox 4 Beta 4,613 575 227.0ms +/- 0.8% 5,858.3ms +/- 0.6%

    Opera 11 4,388 608 227.3ms +/- 0.8 10,732.0ms +/1- 0.8%

    Safari/Win, 817 628 405.0ms +/- 6.8% 14,964.5ms +/- 0.3%

    I.E. 9. beta 442 181 1,081.6ms +/- 1.4% 68,587.2ms +/- 0.3%

    Here’s the short version: Chrome was fastest in two of the tests. Maxthon and Firefox were each fastest in one benchmark each. Chrome got one 2nd place, Maxthon got two 2nd places and Opera took the remaining second place. This is starting to get boring, I know, so to cut to the chase, the browsers overall finished in this order: Chrome, Maxthon, Firefox 3 beta, Opera, Safari for Windows, and I.E.9 beta. Full disclosure: I work for Maxthon, and so I can understand if you don’t take my word for this. You shouldn’t. You should conduct your own benchmarks and invite Maxthon 3 to the party.

    Ron White
    Advocate
    Maxthon International

  • https://www.blogtechnical.com Bradley Wint

    I just included browsers that were popular on the market. If I did include Maxthon, then I’d have to test every single browser on the market.

  • Anonymous

    Hi, Bradley….

    As someone who also hates running benchmarks, I certainly understand your relunctance. So why don’t you simply do this: Include benchmarks on only the five most used Windows browsers. (Not to show any prejudice against Apple, but I’m not sure there are five browsers for Apple.) Do that, and you know what? You’d be including Maxthon, but not Safari. It takes a bit of digging through the information at NetApplications. But when the digging’s done, you’ll find that if you eliminate Safari for Apple, Maxthon is has a far bigger user base than Safari for Windows.

    The conventional line-up of the Big Five browsers has for so long included Safari, that I understand if you find this difficult to believe. I would be happy to take you through NetApplications’s statistics to demonstrate the truth. Just let me know. Then you wouldn’t have to test all the browsers, just the real top five, and you could bring some welcome news to your readers who are looking for a faster browser with more capabilities than Chrome.

    I’ll wait right here, by the email folder.

    Ron

  • https://www.blogtechnical.com Bradley Wint

    Good point on the Windows/Safari market share thing. I may do another benchmark when Firefox 4 goes gold, even though I don’t expect any drastic differences. I might just give your browser a try.

  • Anonymous

    Hi, Bradley….

    How’s the browser round-ups going?

    Ron White
    Maxthon

More in Computing, Featured, Policies/Ethics (19 of 70 articles)