Android Portal

Here are some details about the prize money distribution which Dan Morill posted a few days ago. Wishing luck to all those who are participating!

ADC 1 == this $5,000,000 prize event going on now.
ADC 2 == the second $5,000,000 prize event that will begin later this year.
ADC 1 Round 1 == open participation with the deadline of 14 April, with 50 winners
ADC 1 Round 2 == participation limited to the winners of ADC 1 Round 1, with 20 “final” winners
ADC 1 Round 1 Phase 1 == reducing the original set of 1,788 submissions to 100 finalists
ADC 1 Round 1 Phase 2 == picking the 50 ADC 1 Round 1 winners from the 100 finalists

Okay, phew. :) With those definitions, here is where we are:

* We sent out the submissions to judging a few days after the submission deadline of 14 April, and judging began.
* Our 100 or so judges received the judging guidelines we provided, reviewed their assigned submissions, and reported data back to us.
* Late last week, we applied our outlier mitigation techniques, identified the top 100 results, and sent them on to the final, separate panel of 15 or so judges to score and produce the final 50 ADC 1 Round 1 award recipients.

So in other words, we are currently in ADC 1 Round 1 Phase 2 as defined above. Once data from the judges comes in, we will notify the 50 award recipients and ADC 1 Round 2 will begin.

It has not escaped my notice even on vacation that there have been a number of discussions on server hits and so on. Obviously we don’t have access to everyone’s server logs, and we can’t monitor what the judges have actually been doing (nor would we snoop if we could, since that seems really sketchy.) We’ve tried to automate everything we possibly can about the judging process, but the one thing we can’t automate is the actual act of assigning scores, since that requires a human’s brain.

The judges were given fairly detailed guidance on how to calibrate their scores, and what to review. For instance, they are aware that they are supposed to read documentation and do their best to test all the features. In the end, though, each judge is going to test to his or her own satisfaction. I’m not sure how reliable it is to correlate judge reviews with observed server hits. Some apps might have sporadic bugs that prevent network accesses. Some judges may have decided they didn’t need to see a particular feature. And before you cry foul, know that some people who have inquired about “missing” server hits have actually done quite well. Judges are just as likely to say “this is cool, I don’t need to see any more” as they are to say “this is so uncool, I don’t need to see any more.” On the whole, our judges have been excited to participate, and I expect that they are being as conscientious as they can be.

The one thing I can tell you with certainty is that I have answered quite a few private inquiries, and in all but one case the judges responded with legitimate scores, rather than scores that say something went wrong or the review was incomplete. Our only data points are what the judges give us, because that’s the only factor we can’t automate. Since the judges are telling us that they reviewed to their satisfaction, we can only take their word for it.

We’ve tried really hard to make sure that the only thing that affects scoring is what you put in front of the judges. But the entire goal of the ADC is to leverage plain old human judgment.

- Dan
P.S. - watch for gory details on the nuts & bolts of all this in the near future.