Javascript Performance Thread

Discussion about official Mozilla Firefox builds
Post Reply
User avatar
cyko_01
Posts: 77
Joined: August 2nd, 2011, 6:52 pm

Re: Javascript Performance Thread

Post by cyko_01 »

I was joking. V8engine still shows up on the other machines - something broke
phuzi0n
Posts: 517
Joined: June 23rd, 2010, 5:48 pm

Re: Can you benchmark the browser binary's JS performance?

Post by phuzi0n »

rkl wrote:I know this might sound like a naive question, but can you benchmark the JS performance automatically using just the browser binaries (i.e. without having to build from source)? In other words, can all the data points from AWFY be gathered from shipped binary releases (whether they're nightly, alpha, beta or final)?

If that's the case, then I'd have thought that a) you wouldn't keep having to build releases to produce AWFY data points and b) you could easily benchmark nightlies of Firefox against, say, the latest finals of all the other major browsers (including IE!), which is probably what you should be doing in AWFY...trying to compare how your latest JS fares against what the public already has out there (since very few users relatively speaking use anything other than final versions of browsers).

(I'd also include the latest final of Firefox vs. nightly Firefox, since that's an important comparison (ideally, you want it the same or better - within error tolerances - between the two).

AWFY uses shell versions of just the JS engines (the engine is compiled as a shell app without any of the other browser code) which can have slightly different performance characteristics from the actual browsers but is generally very similar.

The main purpose of AWFY is so that the developers can see regressions in the engine and compare how they stack up against the other browsers. It wouldn't make any sense for them to compare against the current shipping engines in other browsers when they need to know how their upcoming engine will perform against the other guys' next engines.
rkl
Posts: 70
Joined: November 5th, 2002, 9:08 am

Re: Can you benchmark the browser binary's JS performance?

Post by rkl »

phuzi0n wrote:AWFY uses shell versions of just the JS engines

Yes, but this then has 2 issues that create a less than optimal situation.

Firstly, you need the source code for a browser's JS engine. Not only that, but you've got to actually make sure you can run it separately from the browser chrome itself. This rules out almost every major browser competitor except Chromium (which virtually no-one uses!) and that browser's JS engine is probably either already separated (or easily separable) because the Chrome/Chromium team are almost certainly running benchmarks in the same way as Mozilla does.

The second problem is that the way the JS engine is called from the main browser might have some significant impact on its performance. OK, that's just a theory. I assume someone's tested this out - i.e. Firefox's JS engine from the command line vs. Firefox running in a clean profile with a local copy of the JS test suite results in negligible or no performance difference (and ditto for Chrome)?

> The main purpose of AWFY is so that the developers can see regressions in the engine and compare how they stack up against the other browsers.

Yes, but they don't cover IE at all (sadly still the world's #1 browser) and forget Opera completely too. In fact, as it stands, it looks like all they test against is variants of Firefox's nightly JS engines vs. a nightly Chromium JS engine build. Arguably not too comprehensive, especially when most JS benchmarks published out there tend to involve more than 2 browsers (Tom's Hardware Grand Prix for one).

> It wouldn't make any sense for them to compare against the current shipping engines in other browsers when they need to know how their upcoming engine will perform against the other guys' next engines.

I don't agree with that (i.e. only test nightlies vs. nightlies), but if you really must do that, why not have two AWFY - one testing the latest builds against each other and another testing the last stable release?

I still haven't heard a technical reason why browser binaries can't be used for JS benchmark testing. On the Linux platform, you can use Xvfb to provide a virtual X display and either fire up a local home page that runs the JS tests (also locally) and saves the results somewhere (worst case is that you use Xnee to record a cut and paste to a local file, but that's tacky...surely some local save of the results must be scriptable?). Imagine a head-to-head of Firefox vs. IE vs. Chrome vs. Opera for both latest non-stable and latest stable releases? Would be a lot more informative than what we've got now.
User avatar
Omega X
Posts: 8225
Joined: October 18th, 2007, 2:38 pm
Location: A Parallel Dimension...

Re: Javascript Performance Thread

Post by Omega X »

IE and Opera's JS engine are closed source and makes them harder to test against using the system they have now.

And you're forgetting Webkit's SquirrlFish engine used in Safari. That makes 3 not two. Also, IE has almost no presence on Mobile which make it only #1 on desktop. Webkit dominates the mobile space.
User avatar
patrickjdempsey
Posts: 23686
Joined: October 23rd, 2008, 11:43 am
Location: Asheville NC
Contact:

Re: Javascript Performance Thread

Post by patrickjdempsey »

There's no point at all in testing stable releases. The only reason AWFY exists is so the guys landing the code patches can see and appreciate the changes as they happen. That's it. When they first started, seeing the comparisons to other browsers was important because that was the goal of the day. Ever since they crossed that line where they were testing faster than Chrome that aspect of the testing has been irrelevant and now they just want to better themselves. The viral PR element was unintended and a pleasant surprise for the people working on it.

For stable releases this graph simply wouldn't make any sense because it would only change once every year or so for IE, Safari and Opera, and only every 6 weeks (on a different schedule) for Chrome and Firefox. There's simply literally no reason to have a testing machine running continuously to track numbers that at the fastest only change once a month, and even then only ONE of those numbers may change every month. Tom's Hardware does a browser Grand Prix every 3-4 months which is more than enough data for the average user. And honestly, web browsers have been SO equally matched in this for so long now that the comparisons are academic at best and borderline obsessive.
Tip of the day: If it has "toolbar" in the name, it's crap.
What my avatar is about: https://addons.mozilla.org/en-US/seamonkey/addon/sea-fox/
rkl
Posts: 70
Joined: November 5th, 2002, 9:08 am

Re: Javascript Performance Thread

Post by rkl »

patrickjdempsey wrote:There's no point at all in testing stable releases.

Yes there is because it may be the only publicly available "recent" build available (think IE).

> For stable releases this graph simply wouldn't make any sense because it would only change once every year or so for IE, Safari and Opera

Funny, because IE gets regular cumulative patches as part of Windows Update and also has preview builds available leading up to a new major verison jump. I'm not sure about Safari's release schedule, but I'd be surprised if it was only yearly (particularly on a Mac, which is the only platform it has market share on). Opera gets stable updates every 1-2 months, not every year, BTW.

> There's simply literally no reason to have a testing machine running continuously to track numbers...

Er, it wouldn't be constantly re-running tests - the last test results should be cached until the version changes, surely? Technically, it's quite hard to set up - maybe a clean VM + checks for new releases + auto-install new releases + clean profile on every run + run tests in browser automatically + gather results automatically + finally plot them.

> And honestly, web browsers have been SO equally matched in this for so long now

I don't agree with that -- I still see many large gaps between Chrome and Firefox in the current AWFY tests (sadly, mostly in Chrome's favour). I suspect that gap is the being used as the JS performance target (which is fair enough), but it would be nice to know at any point in time where Firefox is relative to other major browsers too. I have no idea where Firefox and IE 9/10 are relative to each other in terms of JS performance for example and having to wait for 2 stable Firefox release cycles for Tom's Hardware to tell us seems bizarre to me.
Srap
Posts: 290
Joined: July 18th, 2012, 11:57 am

Re: Javascript Performance Thread

Post by Srap »

rkl wrote:[...]

> Yes there is because it may be the only publicly available "recent" build available (think IE).

As it was said, Tom's WBGP takes care of that, with many other things.

> Funny, because IE gets regular cumulative patches as part of Windows Update...

Do any of these actually bring significant performance improvement? If not, they change nothing, and mean nothing.

> Opera gets stable updates every 1-2 months, not every year, BTW.

Again, Tom's always use the latest stables, they do not stuck up with an outdated release for months.

> and having to wait for 2 stable Firefox release cycles for Tom's Hardware to tell us seems bizarre to me.

As I saw, Tom's always did the tests when something important landed on Stable: Memshrink's kick-in with FF7, TI With FF9, etc. Results won't really change if they run WBGP with a Fox did not brought anything important. Also, I am sure they will check out FF18 with IonMonkey, when it goes stable.

Edit: there is also LifeHacker's browser test, what is far less detailed, but it is still there regularly. And the last time I knew, AWFY's first and most important purpose was to be a dew tool: monitoring for regressions and measuring the effect of improvements.
Sorry for my bad English. Even if there wasn't a mistake.
User avatar
patrickjdempsey
Posts: 23686
Joined: October 23rd, 2008, 11:43 am
Location: Asheville NC
Contact:

Re: Javascript Performance Thread

Post by patrickjdempsey »

Srap wrote:And the last time I knew, AWFY's first and most important purpose was to be a dew tool: monitoring for regressions and measuring the effect of improvements.


This is in fact it's *only* purpose. It was never intended to be a marketing tool. Somehow it got posted to Reddit and took the world by storm, but that was just one of those happy accidents.
Tip of the day: If it has "toolbar" in the name, it's crap.
What my avatar is about: https://addons.mozilla.org/en-US/seamonkey/addon/sea-fox/
Tametomo
Posts: 42
Joined: April 1st, 2011, 10:03 am

Re: Javascript Performance Thread

Post by Tametomo »

AWFY's 64-bit machine hasn't been updating commits for a few days now, even though it's been running tests on the same commit number since then. The 32-bit one is running just fine though. Anyone know if something broke there (probably should be asking this on #jsapi instead)?

Probably not all that crucial at the moment, since other than the yarr update coming, I dunno anything which might be platform specific changes which might break things. Still would be good to get it fixed though, in case a commit does break on 64-bit builds.
User avatar
cyko_01
Posts: 77
Joined: August 2nd, 2011, 6:52 pm

Re: Javascript Performance Thread

Post by cyko_01 »

Looks like v8 engine is back
iamhere
Posts: 169
Joined: July 23rd, 2004, 11:48 am
Location: London, UK

Re: Javascript Performance Thread

Post by iamhere »

Seems like AWFY is not taken care of anymore. Is there any other place to follow what's up with improvements to the JS engines?
Srap
Posts: 290
Joined: July 18th, 2012, 11:57 am

Re: Javascript Performance Thread

Post by Srap »

iamhere wrote:Seems like AWFY is not taken care of anymore.

What makes you think this?
Sorry for my bad English. Even if there wasn't a mistake.
User avatar
Omega X
Posts: 8225
Joined: October 18th, 2007, 2:38 pm
Location: A Parallel Dimension...

Re: Javascript Performance Thread

Post by Omega X »

AFWY is the same as it always been.
iamhere
Posts: 169
Joined: July 23rd, 2004, 11:48 am
Location: London, UK

Re: Javascript Performance Thread

Post by iamhere »

Srap wrote:
iamhere wrote:Seems like AWFY is not taken care of anymore.

What makes you think this?


Of the 5 machines, only Mac OS X, 32-bit (Mac Pro) and Linux ARMv7 seem to be working. The Mac Mini machines have been broken for quite a while (and the comments here were that they would eventually be removed). The 64-bit Mac Pro has been stuck for about a week and before that was close to two weeks without showing anything for Chrome.

The second part of my previous comment ("another place to follow more up-to-date development") wasn't in relation to my comment about AWFY, though I can see that's how it looks like. I am now following the blog mentioned before in this thread (https://blog.mozilla.org/javascript/).
User avatar
bogas04
Posts: 977
Joined: May 18th, 2010, 1:14 am

Re: Javascript Performance Thread

Post by bogas04 »

So I read this article Chrome has gotten 26 percent faster this year , and wondered how fast has FF become over a year !

I ran the same Octane test , for FF8 , FF16 and the latest nightly FF 19 (The build is exactly a year after FF 8 was launched).

The results were :

FF 19 6025 (106%) <8 Nov 2012>
FF 16 5408 (85%) <9 Oct 2012>
FF 8 2921 (0%) <8 Nov 2011>

So that's all , FF has seen such drastic performance improvements over a year that it puts Chrome's improvements to dust.

PS : A very delightful and a happy birthday to Firefox for completing its 8 years ! :D :D :D
bogas04.github.io
MacBook Air Mid 2013 |@bogas04
Post Reply