Annual Fundraiser: Checking Banner Results
Hey All–
We’ve been tracking a huge amount of data during this year’s fundraiser so we can better understand which messages work well and which don’t. We have two sets of banners that we set each day to run on all Wikipedia languages. Set one is the English version; set two is all non-English versions.
We have two sets because we want our banners to run globally only if they are translated…which can take some time and volunteer effort. This is why our non-English banners rotate slowly. However, with English banners, we can build a banner quickly and put it up to see how it does.
Let’s go into some detail on selecting a rotation. On December 3rd, our rotation and results (pp = Paypal, cc = credit card):
| % of Total | Site Notice | Payment | Number of | Total | Average | Highest | ||
| WP views | Type | Donations | Amount | Gift | Donation | |||
| 12/3/09 | 20% | 2009_Notice17 | pp | 210 | $4,933.75 | $23.49 | $100.00 | |
| 12/3/09 | cc | 156 | $5,894.76 | $37.79 | $500.00 | |||
| 12/3/09 | 20% | 2009_Notice18 | pp | 725 | $11,807.41 | $16.29 | $1,000.00 | |
| 12/3/09 | cc | 454 | $10,145.52 | $22.35 | $250.00 | |||
| 12/3/09 | 40% | 2009_Notice30_bold | pp | 504 | $11,023.15 | $21.87 | $250.00 | |
| 12/3/09 | cc | 389 | $14,468.07 | $37.19 | $1,000.00 | |||
| 12/3/09 | 20% | 2009_Notice36 | pp | 207 | $4,650.90 | $22.47 | $120.00 | |
| 12/3/09 | cc | 147 | $5,890.83 | $40.07 | $250.00 |
As you can see, we had three different notices running at 20% and one banner, taken from one of the better notices from 2008′s fundraiser, 2009_Notice30_bold running at 40%. It did well throughout last week.
As you can see, 2009_Notice18 pulled in a huge number of gifts despite only showing 20% of the time. Also, it had a significantly lower average gift…probably as a result of the message itself. Despite the low average gift, people seemed to really respond to the message…and donated lots.
We are wary of banner-fatigue and saturation, where users might be tired of seeing the same message, so we changed banners around for the next day.
Looking at December 4th, 2009:
| % of Total | Site Notice | Payment | Number of | Total | Average | Highest | ||
| WP views | Type | Donations | Amount | Gift | Donation | |||
| 12/4/09 | 20% | 2009_Notice17 | pp | 192 | $4,280.08 | $22.29 | $250.00 | |
| 12/4/09 | cc | 144 | $4,778.83 | $33.19 | $250.00 | |||
| 12/4/09 | 20% | 2009_Notice18 | pp | 611 | $9,511.88 | $15.57 | $250.00 | |
| 12/4/09 | cc | 390 | $9,390.74 | $24.08 | $500.00 | |||
| 12/4/09 | 20% | 2009_Notice30_bold | pp | 266 | $6,573.39 | $24.71 | $1,024.00 | |
| 12/4/09 | cc | 228 | $6,696.20 | $29.37 | $238.75 | |||
| 12/4/09 | 20% | 2009_Notice36 | pp | 205 | $4,399.75 | $21.46 | $166.53 | |
| 12/4/09 | cc | 162 | $5,018.47 | $30.98 | $250.00 | |||
| 12/4/09 | 20% | 2009_Notice40 | pp | 320 | $7,795.45 | $24.36 | $1,000.00 | |
| 12/4/09 | cc | 187 | $6,113.04 | $32.69 | $500.00 |
We introduced 2009_Notice40 (“Thanks, Wikipedia.”) to the mix and cut back on another. All five banners in rotation are at 20%. Two of the banners are greatly outperforming the others. We like what #40 is doing…but #18 is still rocking…1001 donations while the next closest is 507 donations (#40).
Again, we switched things up, removing #17, and adding 2009_Notice22, a similar, but opposite message to #18, which has been successful.
December 5th, 2009:
| % of Total | Site Notice | Payment | Number of | Total | Average | Highest | ||
| WP views | Type | Donations | Amount | Gift | Donation | |||
| 12/5/09 | 20% | 2009_Notice18 | pp | 518 | $8,207.09 | $15.84 | $250.00 | |
| 12/5/09 | cc | 314 | $6,866.38 | $21.87 | $250.00 | |||
| 12/5/09 | 20% | 2009_Notice22 | pp | 166 | $4,634.09 | $27.92 | $250.00 | |
| 12/5/09 | cc | 135 | $4,938.74 | $36.58 | $250.00 | |||
| 12/5/09 | 20% | 2009_Notice30_bold | pp | 272 | $6,989.63 | $25.70 | $250.00 | |
| 12/5/09 | cc | 197 | $8,005.98 | $40.64 | $1,000.00 | |||
| 12/5/09 | 20% | 2009_Notice36 | pp | 195 | $4,440.20 | $22.77 | $191.00 | |
| 12/5/09 | cc | 157 | $7,092.57 | $45.18 | $1,000.00 | |||
| 12/5/09 | 20% | 2009_Notice40 | pp | 279 | $5,881.04 | $21.08 | $250.00 | |
| 12/5/09 | cc | 168 | $6,259.51 | $37.26 | $1,000.00 |
This day is a fascinating look at our banners and our user population. Notice the results of #18 and #22…they are similar yet contrasting messages. #18 is an quote from a small dollar donation (USD 1.95), acknowledging the sincerity of the gift. #22 is a quote from a high dollar donation (USD 200), emphasizing that a big gift is a small matter.
Compare the numbers of gifts for the two banners:
#18: 832 gifts, $15073.47 total
#22: 301 gifts, $9572.83 total
Not close right? But look at the average gift sizes:
#18: $15.84 for Paypal, $21.87 for credit card
#22: $27.92 for Paypal, $36.58 for credit card
That’s quite a difference. What was it about the message that would account for that? Is it possible that our more affluent donors were more interested in #22, while other donors were affected by #18?
Post your thoughts below.
Rand Montoya, Head of Community Giving
Thank you for your transparency on approach and results of fundraising. Although surprising to see such candor, it is completely in keeping with the spirit of your endeavor. Congratulations!
I recently made donation and have two items of feedback:
1) for me, the messages of the banners had a compounding effect and the trigger message to giving was one where I could identify with donor — i.e. I think you need mix of messages: all messages combined give context and help make connection between received value, independence of source, and budgetary need; individual testimonies help to lower threshold to giving.
2) now that I have given, perception of banners has changed — i.e. I now react positively to messages that show results of fundraiser and “less positively” to messages that appeal to giving. The former make me feel part of the success of the fundraiser while the latter are frustrating since I’ve acknowledged need and justified giving to myself but have already exhausted my budget. Perhaps increasing frequency of banners that give fundraising status and thank donors as you approach target may help lower banner fatigue. It may also help users celebrate being part of community that supports this independent source of information.
Thanks again!
I have written a blog post on this with detailed explanation.
“Wikipedia donators and the Anchoring heuristic”
http://saperduper.org/post/293243288/wikipedia-donators-anchoring-heuristic
Please feel free to post your comments!
We need to review the negative case data – when presented with a banner the user did NOT donate. It is possible that you have 100%conversion on #18 but only 10%conversion on #40. This data leads to the conslusion that ALL VISITS presented with one of the above banners donate. This is not the case. I have NOT clicked to follow any of the banners in this fundraiser.
#19, #20: It is indeed possible to turn off the fundraising banner. There’s a [hide] link at the top right of the banner.
#24: AFAICT we’re already implicitly measuring that, right? Say N people visited Wikipedia on a given day and saw a banner. Assuming all banners run at 20% (which is the case most of the time), the number of people seeing a certain banner would be N/5. If that banner got 500 donations, the number of people that saw that banner and didn’t donate is N/5 – 500. So for each banner, the number of people that saw it but didn’t donate is N/5 – D where D is the number of people that did donate. This N/5 is a pretty large number that we don’t know exactly, but that doesn’t matter because it’s the same for every banner. In short, if a banner got more people to donate that automatically means it got fewer people to not donate, and vice versa.
“Am I the only one, or is there something somewhat gross about engineering messages to extract the greatest possible donations? Maybe this is standard practice in fundraising, maybe it’s the right thing to do…but something about it doesn’t sit quite right.” says Steve Bennett.
If everyone thought that way charities that have saved millions of lives would all have closed down. And so would Wikipedia. And you wouldn’t be reading this. In fact, on the basis that we should never test what works and what doesn’t, all human progress startinbg with the discovery of fire or how to escape an angry saber-toothed tiger would never have got started.
Absolutely fascinating results! I was drawn by number 18, but finally donated using a different banner. Wikipedia is becoming more and more recognized as a legitimate source, especially for hard science, like finding and using equations, computer programming code, and hard biological facts like cellular respiration. It’s also an interesting source for cultures and country data. I pay hundreds of dollars for textbooks with the same information, or course I would donate.