Srap wrote:Chrome engine (V8, if I am not mistaken) seems to be missing from AWFY. Any reasons behind this?
While I don't know for sure, I can think of a couple of reasons why they might have done it (assuming that d8 didn't break, move repository locations, or that the local build system for d8 didn't break. Also going to call it d8 as well, like Mozilla does internally, so as to not confuse it with the benchmark of the same name):
1. If I am not mistaken, I could have sworn I've seen some accusations internally of d8 trying to game benchmarks a bit and getting increasingly frustrated with it, even to the point where while some code might work within a benchmark, and which might make them look good, they then aren't always doing full test coverage and end up broken with functionality or uses outside of benchmarks on newer javascript features. For instance, while I don't have a link handy, I think when Octane was first announced, there was a claim made that one of the benchmarks, if compiled with Emscripten instead of Mandreel, caused a compile failure when it was valid code. Don't remember which though, and could always be making something rather minor appear to be bigger than it was, and might even be remembering it wrong. I also wouldn't quote me on this as a source, as I'm not involved with any browser team, nor do I do a lot of work with javascript itself. If I'm wrong on this, then someone please let me know, and I'll gladly strike this (although I don't check here too often. Sorry about that).
2. Likewise, going along with number 1, but not requiring it to be true, there do seem to have been some concerns about their recent benchmarks seemingly being geared towards making
chrome look good, but not necessarily being representative of advancing the web as a whole. For instance, I remember seeing comments about how while some of the ideas in Octane probably weren't all that bad, it seemed too geared towards being a testing suite for marketing apps from the Chrome store, and that benchmarks based on bananabread or emscripten, for instance, might be better suited for real web performance purpose.
3. Dead code elimination being added to d8, which then removes some reproducability as it retains past information about what was used and what wasn't, and also brings some concerns into correctness issues for the code generated (not that any new compiler optimization technique doesn't already do this, but dead code elimination is somewhat controversial in some respects, since it can be rather easy to get it wrong).
4. An increasing frustration with d8's team relying on AWFY too much for their own testing resources, when the purpose is to help Mozilla better visualize and track their own regressions and improvements internally, even going as far as requesting that they remove V8 and replace it with Octane, so that they could track how they've been making progress on it a bit more easily, when the developers (at least from what I've seen) haven't been quite as enthusiastic about it, since there appear to be doubts about whether it's really designed for what they think they need for real web performance.
5. Other benchmarks like
Robohornet being developed, which then bring into question even more about whether the right things are even being tested at all, and whether or not a lot of these newer benchmarks being pushed by Google now are even measuring the right things any more (for instance, see the open bugs by
kevingadd). Then again, this is essentially an extension of #2, just for a different benchmark.
6. d8 becoming increasingly irrelevant to benchmark against, as it's not too different from JSC in performance now on V8, where Mozilla is the farthest behind at the moment. This was the original reason why JSC was originally removed IIRC (that, and it having been broken for a day or two at one point, making it so that it wasn't showing results), since at one point last year, they beat JSC by a rather large margin, and they didn't appear to be improving fast enough to warrant them being tracked any more. But then they landed quite a few changes to the compiler, and it started to get close to d8's performance and exceeded it on sunspider, which then prompted them to add it back again, as it became a relevant comparison again. In fact, it might be that last part which might have prompted dropping d8, since as far as I know, they also don't have a generational garbage collector yet, but are able to get close to d8 speeds on V8. And since JSC's compiler is now currently closer to what they are aiming to be, and that it's similar enough in benchmarking performance to be a good baseline to chase, then there's not much of a need for d8 to be profiled against at this time.
Could be any combination of these or none of these. Just my own wild mass speculations that I don't think are too far off the mark.
Also remember that AWFY was originally meant to be a developer tool to help them improve their own engine, and not so much a marketing tool, even though it has kind of turned into one. It should first suit their own internal needs rather than what the community might want. So if this is not an accident, and they decided internally that it's not helping them much any more to profile d8, then I don't see why they should continue to have to do so if the community asks for it. The code for the site is
open sourced, so if someone else decides that they want to track things differently, then it's completely possible to run your own private copy to display what you want to see.