The Netflix prize was just won, making a post I had sitting in the back of my head a lot more timely. In case you’re unaware, the competition was to improve by 10% on Netflix’s own algorithm for recommending titles to customers based on their previous preferences.
But what’s special about 10%? I ask because improvement on Netflix’s algorithm was extremely quick. 5 % took less than 2 months, and multiple teams had reached 8% in a year. It’s been 18 months since then to reach 10%.
On the fact of it, that’s very strange. It’s true that a well designed prize would take long time to beat, and that it should take increasing quantities of time to make improvements as time goes on. But then it should be possible to overshoot and set a target that won’t be met. The timeline of the Netflix competition suggests that they did a very good job setting a target–not too ambitious, but not too easy.
But what’s the prior probability that any given target would be right? Not wonderful. So here are three possibilities: 1) It was luck. 10% was a nice round number, Netflix chose it, and the competition worked out right. 2) There’s only so much possible improvement to be had. More than 10% was unrealistic, and Netflix knew that, because a) there are theoretical results out there that suggest the systems can only be so good or b) the Netflix algorithm team had some know-how. Perhaps they already knew they could improve their system by n percentage points, and gambled $1 million on competition producing a better result. 3) More than 10% was achievable, but the way the problem was framed limited experimentation. Note that even well into the competition, there was a lot of collaborative discussion of ideas between the teams. Basic ideas would be shared, and then different teams would implement them, perhaps sharing tricks discovered on the way. Presumably this died down as teams approached the 10% goal (though the winner fused the two teams that had previously been closest). This kind of setup has the potential to make everyone take the approach of trying to eke out a few tenths of a percentage point from essentially the same approach as they’d been using. If so, that has the potential to create an artificial ceiling, as teams approach a local but not global maximum of fitness.
All of this is pure speculation by an outsider, but the phenomenon surrounding the 10% target caught my eye, and I couldn’t resist. Bottom line question: could Netflix benefit from a second round of competition, or have they tapped out the resources available for $1 million?
Update: Seems that the winning condition was a bit more complicated than I’d realized, though it doesn’t matter for what I’m saying in this post.