If you hold these goings-on up to the light, from one angle they look more like school yard tomfoolery than razor sharp academic and commercial innovation yet at a different angle it looks slightly more ugly.
Not to underestimate the phenomenal work that everyone puts in to their submissions but it seems like there is some raw human behavior at play which is, as far as I am aware, not out in the public domain.
Organizers of ImageNet’s Large Scale Visual Recognition Challenge (ILSVRC) announced on May 19th 2015 that a group entering the challenge had circumvented the ILSVRC policy of allowing 2 evaluations per week and on June 2nd, disclosed that there were at least 30 accounts used by a team from Baidu to submit to the test server at least 200 times.
At this point, all I can think is that it’s probably a good thing that those responsible for the spike in submissions are Research Scientists and not career crims!
If nothing else, what this did show was an obsession of the Baidu team to inject a much larger footprint on the ILSVRC to beat the error rate of 4.8% that Google had previously achieved, and which is now reinstated as the best performance on the dataset, by risking the submissions.
When the Wall Street Journal reported that Baidu had beaten the error rate set by Google on May 12th, it mentioned that Facebook’s Yann LeCunn felt that the ImageNet test was becoming ‘passe as a benchmark’. I wonder whether Baidu may now be thinking the same. Especially since ImageNet has also suspended Baidu from entering the contest for 12 months stating –
There is also concern about any new submission from Baidu that builds on top of these results. This includes submissions to all of the ILSVRC challenge tasks (classification, localization, and detection) as parts of the datasets are shared. We therefore requested that they refrain from submitting to the evaluation server or the challenge for the next 12 months.
The overall picture shows a thriving community of Research Scientists that are driven to improve the evolution of deep learning.. that is, from where I am sitting. I would be very interested to see if this episode will leave any lasting impact in the community. The New York Times reported
The episode has raised concern within the computer science community, in part because the field of artificial intelligence has historically been plagued by claims that run far ahead of actual science.
whilst inferring that this sort of thing could even lead to another AI winter. The big difference this time around being the commercial interests of the organizations that are funding research nowadays, completely driven by consumers, the majority of whom would have very little interest in Baidu having put in more submissions to a competition than they were entitled to.