Results from AS3 Development Task Contest #1
I have just completed running all of the performance tests for the ActionScript 3 Development Task Contest #1.
The result below are tentative until I have confirmed that the code follows the rules (and spirit) of the contest. If you see any issues with any of the top 5 submissions, please post them in the comments. You can view all of the entries in the contest’s GitHub repository.
Here are the results:
You can download the raw results in CSV format from here, or view them on google docs here.
All examples were compiled with MXMLC from the Flex SDK 3.4 (Version 3.4.0 build 9271), with the following command:
mxmlc --target-player=10.0.0 -compiler.source-path ~/src/PerformanceTest/ -- TestRunner.as
Results were obtained by running TestRunner.as. TestRunner.as ran each test 300 times per run. The TestRunner was run a total of 5 times (with the SWF reloaded each time).
Test were run in Safari Version 4.0.4 (6531.21.10) on Mac 10.6.2 in Flash Player 10,0,32,18 (regular). The Mac was a MacPro 2 x 3 GHZ Quad Core Intel Xeon with 8 GB RAM.
I will finalize results and announce the winner either late tonight, on Wednesday (I am traveling on Tuesday). Check back here for updated information, or follow my twitter account at @mesh. I will be awarding prizes to multiple submissions.
UPDATE : (November 17th, 2009) : I am declaring the entries final. Arnaud Gatouillat is the overall winner, and wins Flex Builder, a tshirt, etc…. In addition, given some of the great results that we got from the submissions, I am also going to award a prize (their choice of a tshirt from threadless.org) to the next 3 top entries, which include Grant Skinner, Jim Cheng, and Jean Phillipp Auclair. If you are one of the top four entries, please email me at mesh@adobe.com and we will work out the details.
Great stuff! I did a lot better than I thought and was not TOO far off from our contest host!
Made a little post about the whole thing on my company’s blog.
http://www.fiftyonereasons.com/2009/11/a51-actionscript-3-development-task-contest-1/
Thanks Mike, the whole thing was a blast.
Phil
16 Nov 09 at 2:01 pm
Great work everyone! I’m going to be picky because I came 6th ;-)
Arnaud’s is the only one in the top 5 that I consider valid.
My test is the same as Mike’s, except: bounds = new Rectangle(100,100, stage.stageWidth – 100, stage.stageHeight – 100);
3 of the top 5 throw an exception with this small change, because they don’t take into account that bounds.x/.y might be non-zero: GrantSkinner, JimCheng and JeanPhilippeAuclair – adding this check to those may or may not affect their standing.
yuku/2 doesn’t actually use the bounds size at all, which is pretty neat, but it’s fast because it doesn’t copy the objects vector passed to update but instead just keeps a reference – is that allowed?
Marcin Szczepanski
16 Nov 09 at 2:17 pm
@Marcin,
–
but it’s fast because it doesn’t copy the objects vector passed to update but instead just keeps a reference – is that allowed?
–
I just took a glance at it, and it looks like it is fine. He is not modifying the vector, and if the data changes, update would be called again.
http://github.com/mikechambers/ActionScript-3-Development-Task-Contests/blob/master/AS3DTC_1/entries/yuku/2/ProximityManager.as
mike chambers
mesh@adobe.com
mikechambers
16 Nov 09 at 2:36 pm
The fact that some entries don’t take bounds into account doesn’t matter that much.
I choose between codepaths in the contructor depending on the bounds. If @mesh didn’t change the test parameters, the zero-based codepath was chosen.
As the same trick could be done in 30sec for each class, it wouldn’t change the end results.
That contest was more about data structures and loop optimization.
Arnaud Gatouillat
16 Nov 09 at 2:53 pm
I dunno about it not changing the result much, I know it added about 0.2ms to my runtime to adjust for non-0 x/y.
Don’t get me wrong, of course I understand it’s about the optimisation and learning, I’m not denying that. I’ve certainly picked up a lot of interesting ideas from looking at all the other entries!
Marcin Szczepanski
16 Nov 09 at 2:58 pm
I’ve read your post again and you’re right that these classes and yours aren’t beeing jugdged on the same exact grounds.
Arnaud Gatouillat
16 Nov 09 at 3:13 pm
Good find, Marcin. Taking a closer look through the entries, it looks like a lot of us took a few shortcuts in error-checking and handling of edge conditions for the sake of speed. :(
In particular, many of the implementations (mine included) assume the bounds’ upper-left point is at the origin, do not verify that all the items passed in actually fall within the given bounds, don’t handle edge values nicely, and assume that the objects have x and y coordinates equal to or greater than zero.
It is very interesting to see the optimizations that various folks used. I’ve learned quite a bit while working on my implementation and then even more so seeing everyone else’s.
Interesting ideas I’ve picked out so far:
- Keep the logic simple
- Instantiation of new instances is expensive
- Minimize dereferences and boxing/unboxing
- Use integer and bitwise calculus for speed
- Pre-increment and multiply by reciprocal
- Vector.push(…) is slow
- The debug player is not the release player
Jim Cheng
16 Nov 09 at 3:22 pm
Out of curiosity, did anyone try unrolling their loops while coming up with their submission(s)? I tried this while coming up with mine and ran into some interesting results.
During testing, I had modified my update() method to unroll the loop a bit and saw some performance improvements in the debug player against a straight for-loop for iterating through the items vector.
I then proceeded to test the same code in the release player and, contrary to my expectation, saw that the unrolled loop ran _slower_ than the plain-vanilla version.
Needless to say, I abandoned the loop unrolling idea at this point, but found it rather interesting that the unrolled loop was faster in the debug player but slower in the release player when compared to the plain version. Maybe someone some greater insight into Flash’s bytecode generation and/or run-time JIT compiler could shed some light on this curious behavior?
Jim Cheng
16 Nov 09 at 3:33 pm
I tried to test the results, but I found there is a lot of influence from the randomness of the displayobject.
So I made a random function working from a seed in TestRunner, so it would always produce the same result.
// defining _prev and _random makes results different
_prev = 0.3;
_random = 0.5;
private function random():Number
{
_next = Math.abs(Math.cos(Math.tan(_prev + Math.sqrt(_random))));
_prev = _random;
_random = _next;
return _random;
}
I found some really different results :
standalone release version 10.0.32.18
compiled with Flex SDK in FlashDevelop
windows XP SP3
Intel Core 2 Duo 2.26 GHz / 2.96 Go Ram
tested 15 times each in 3 different situations, kept only the best and the worst result.
Arnaud Gatouillat 2.01 => 2.07
Grant Skinner 1.88 => 1.94
Jim Cheng 2.17 => 2.23
Jean Phillippe Auclair 1.97 => 2.03
yuku/2 2.22 => 2.29
Marcin Szczepanski 2.21 => 2.28
Jonas Monnier 2.36 => 2.41
Bertrand Larrieu 2.22 => 2.25
Andrew Traviss 2.78 => 2.83
I think there’s also a difference between mac flash player and pc flash player
lab9
16 Nov 09 at 4:15 pm
hehe, i really do like your test results Bertrand ;)
But still, from what i see, there is a LOT of variation between the tests depending on a lot of factor.
I ran the same tests (the first 4) on three computers and they were never in the same order.
on my old Toshiba Tecra S1 laptop, Gskinner is twice faster than my test. Pentium M 1.5Ghz 748Mo with FlashPlayerExternal.
jpauclair
16 Nov 09 at 7:08 pm
>>jpauclair
i agree. I try some different implementation in debug and standalone player. Some implementation will faster in debug player. And some implmentation will faster in stanalone player.
wailam
16 Nov 09 at 7:38 pm
On my current computer it was going faster with a “useless” variable.
There was a variable in my code that did nothing, and when i tryed to removed it, i was going from 3.4 to 3.6!!
but it was doing absolutly nothing!
jpauclair
16 Nov 09 at 8:11 pm
@jpauclair – re useless var – maybe some weird JIT related thing?? things getting aligned differently or something.
In general, I probably should have tested against the release player more, some of the optimisations I added that improved performance on the debug player might have had the opposite effect on release.
Still, I guess it’s a successful contest when everyone’s learned something they can apply to ASDTC#2! :)
Marcin Szczepanski
16 Nov 09 at 8:22 pm
I believe Grant’s could be sped up quite a bit by changing uint to int throughout his class. Try it out and see what happens.
Steven Sacks
16 Nov 09 at 9:28 pm
Personally I’d prefer Grant’s ProximityManager as it supports a more flexible way to find the neighbours of a cell.
Anyway, all did a great job and we now have a nice collection of alot of beautiful code. :)
Glassgiant
17 Nov 09 at 1:15 am
best if the next contest be held on http://wonderfl.net/ next time,
cause we can see all the entries WORKING!
mash
17 Nov 09 at 7:09 am
@mash
Feel free to post them there. All of the code is open source.
mike chambers
mesh@adobe.com
mikechambers
17 Nov 09 at 9:32 am
[...] Chambersa rozwi?zany Przez flexibleair Dodaj komentarz Kategorie: Adobe AIR Konkurs og?oszony przez Mike’a Chambersa zosta? rozwi?zany. Ku mojemu zdziwieniu okaza?o si? [...]
Konkurs Mike’a Chambersa rozwi?zany « Flexible AIR
17 Nov 09 at 3:13 pm
Hi lab9,
I think there is limitation of the random function that there are sometimes the random numbers are repeating.
boat accessories
18 Nov 09 at 4:21 am
“If you see any issues with any of the top 5 submissions”
I wished the prizes were for top 5! Just one more… :(
yuku
23 Nov 09 at 6:29 pm
I’ve read your post again and you’re right that these classes and yours aren’t beeing jugdged on the same exact grounds.
reiner
25 Jan 10 at 7:54 am
Nice. So sad to be late on this contest.. But I would definitely participate on a contest for a Flash CS5 Beta program!! ;)
Pablo
9 Feb 10 at 5:36 am