I wonder what input factors you had to consider? If it is just a giant number churning exercise based off of the “easy to get out of the database” aspects, than you may not be considering factors that would give you the strongest insights. Demographics for instance. Also perhaps interaction (are there certain people that are “attractors/motivators” and others that are “driving people away”)?

I would not give up at finding this info out either. Perhaps fancy shmancy neural nets are not the way to get key insights, but instead doing some semi-quantitative work based on cases, interviews, etc. would lead to better insights. I also find that doing this sort of thing can often generate new hypotheses and ideas. Because you just learn things and generate new “factors” from getting a little dirty looking up close and personal at some cases.

It’s comparable to doing a manufacturing optimization problem. We could just use what data we have (now, easy, and in big numbers) from the shop floor computer system. That will give us some things…it will probably be robust (if you do out of sample confirmations). And you MIGHT learn something counter-intuitive, just because tribal wisdom doesn’t match the numbers. But pretty decent chance you might not be tracking a key feature. Also a good chance that there may be some problems somewhere in data input/tracking etc. that get missed…if it is all a hands length math game.

The next step up from that, might be to get on the white board and brainstorm, list ideas, diagram the process, make those fishbone causation charts, etc. and just list some possible factors. Then after doing that consider which ones have easy data to check on and which don’t (and if there are some that you think may be very critical…but lack data on…then you figure out some workarounds (e.g. cases) to at least improve your Bayesian hunches and try to validate/invalidate the hypotheses.

Perhaps even more of a step up is to do a walkthrough of the assembly line (or even take a shift or two and build the product personally!) This will probably give a few new ideas…and it may give insights on how to prioritize which factors to investigate, spark ideas for how to do workarounds where the factors are not tracked in the database, etc.

Of course, all the processes are important. And really it is a feedback loop as well.

that said…I’m still blown away how remote from actionable insights, the work here was. I went to the winner page and the discussion was all matrices and rows and set theory and such…nothing like…F=MA or V=IR or the like. ;-)

Sorry…if this is all motherhood and apple pie, but I felt it needed to be said. And I DO appreciate one person (you) boiling down some of the learning.

I want to point out a typo though, Benjamin and Fridolin Roth did not use random forests, but rather vanilla/standard linear regression. Unfortunately there turned out to be a mistake in the data set construction, it was not properly randomized.

As such Benjamin and Fridolin, no fault to them, stumbled upon this error and used it as the primary input to their model. This error happens to be equivalent to knowing a large fraction of the answer, future edits, before predicting it. Unfortunately this means that their model is invalid and can’t be used by Wikipedia to understand participation. See the following link for more detail:

http://www.kaggle.com/forums/t/980/wikipedia-participation-challenge-an-unfortunate-ending