Eating, Praying, Losing: Getting Beaten by a Voting Bot in 50 Minutes

If you’ve used the site in the past couple of weeks, you may have noticed that PadMapper recently got nominated for a Webby Award for best Real Estate site! We were pretty excited about this since our team is tiny compared to most real estate sites (it’s me and a summer intern), and the Webbies are one of the more prestigious awards that a website can win. At first we didn’t think we could actually win the People’s Voice popularity contest against a big company like Zillow, but at 13.5 days into a 14-day voting period, we’d gotten 37% of the votes in the Real Estate category, 5 percentage points ahead of the second place contender Zillow, and 12 percentage points ahead of the third place contender Apartment List. We’d gotten an outpouring of love from users and things were all around pretty great (thanks everyone).

Then at 12PM on Thursday, with 12 hours of voting to go, the Webby Awards decided close the doors and hide the percentages. The stated reason was to keep the anticipation high until they officially announced the winners 5 days later.

We were bummed that we wouldn’t get to see the end of the race, but then around 4PM, we figured out how to see the percentages again! Whether intentional or not, the voting site was set up such that if we logged out and logged back in again in the Chrome browser using Facebook login, we could see the voting breakdowns again. Logging in and out to look at voting results is a pain, though, so when we saw the percentages were basically the same as 4 hours before, we went back to work on more productive projects. Since the voting had basically stabilized to a ~1-2% fluctuation per day due to the large number of votes from the long voting period and because there were only 8 hours left for voting, we were feeling good about PadMapper’s chances of winning the popular vote. We were in such a good mood that we made ourselves lamb chops for dinner. With ice cream!

Then at 10:40PM, we got curious and we peeked again at the voting results. We were still in first place, but Apartment List had pulled up dramatically. We smelled something fishy. We’re not usually the suspicious type, but we’d had a bad experience dealing with Apartment List before, so we decided to track the voting closely.

Here is where things get crazy. Watch closely.

Below are the voting breakdowns for the Webby Awards Real Estate Category at various points on the last day of voting for the top 3 contenders, all times in PDT:

Previous day PadMapper (37%), Zillow (32%), Apartment List (~25%)
Just before the numbers disappeared at 12PM PadMapper (37%), Zillow (32%), Apartment List (25%)
When we found a way to see the numbers at 4PM PadMapper (37%), Zillow (32%), Apartment List (25%)
10:40PM PadMapper (35%), Zillow (30%), Apartment List (30%)
10:55PM PadMapper (34%), Zillow (29%), Apartment List (31%)
11:10PM PadMapper (34%), Zillow (28%), Apartment List (32%)
11:32PM PadMapper (32%), Zillow (27%), Apartment List (35%)
11:32PM – 12:00AM No fluctuation in votes

Webby Voting Data Points

Needless to say, we were a little emotional for the 50 minutes between 10:40 and 11:30. First crossing our fingers because the race was so tight, then calling “Foul!” on the couch in our living room. By the end, it was like getting to the end of the first Game of Thrones book and, all of a sudden, [redacted] lost his [redacted]! Then as now, we were shocked and unhappy.

Math says that new votes for Apartment List alone in those 50 minutes accounted for at least 7% of all votes in the contest over the 2-week contest. That’s 7% of the votes in 0.2% of the contest time. Aside from this sheer volume of votes, 5 other things seemed fishy:

A) The movement observed happened after 10:40PM on the West Coast. That means it was 1:40AM on the East Coast, on a weeknight.
B) Throughout this flurry of activity, Apartment List was the only site to gain in percentage, and PadMapper and Zillow remained 5% apart at all times.
C) The pace of Apartment List’s gains was linear over the course of the 50 minutes.
D) The movement suddenly stopped at 11:30PM with Apartment List at a 3% lead, 30 minutes before voting officially closed.
E) A quick check on Apartment List’s Twitter and Facebook pages showed no activity in the last 6 hours. No promoting, no calls for votes, nothing.

The total rise from their previous days’ numbers represent a roughly 50% increase in the total number of votes over the number they received in the previous two weeks, despite having next to no social media buzz on the last day.

So here are some guesses at how this could have happened:

1) Lots of avid Apartment List fans suddenly discovered the contest in its final hour and voted at 2AM on the East Coast and 11PM on the West Coast on a Thursday night, at a constant linear pace. Then suddenly no more votes for the last half hour of the contest.
2) Lots of people halfway across the world where it was daytime suddenly discovered the contest in its final hour and voted, at a constant pace, for a website that only serves the US.
3) Someone created a bot that pre-registered for a bunch of fake Twitter accounts and used those to vote for Apartment List.
4) Someone did the human equivalent and used Mechanical Turk or another form of cheap labor to buy votes for an hour or two using fake accounts.

The first two don’t seem very likely.

Only the people at the Webby Awards have the voting logs, but based on the data points we showed above, our best estimate is that the voting distribution over time probably looked something like this (with minor fluctuations):

Webby Voting Lines

Sadly, the voting is over, and the results may be set the way they ended up above. We ate a too-early celebratory meal, then we prayed that we could still win, then it seems that a bot or something else came and bulldozed over us with a truckload of fake votes. It’s been an interesting exercise to study the voting pattern of this situation. Whereas most social network-driven activity probably builds in a Poisson-shaped distribution, the votes for Apartment List were added linearly, with a sudden halt at 11:30PM. We hope that the administrators of the Webby Awards will disqualify the almost certainly fake votes and we’ll regain first place, but we haven’t received a response from the admins about this yet.

What makes this whole voting mess more troubling to people who don’t give a fig about our winning an award is that decision-making is increasingly based on internet reviews or votes. The aggregate opinion of the masses is often viewed as more accurate or trustworthy than that of a single reviewer in a magazine, since we expect that we may have different preferences from a single reviewer, but if everyone likes something, we probably will too. This event shows that an aggregate opinion online may actually be easier to fake than a single detailed review. This other story posted less than a week ago about a man being duped by fake reviews for a moving company on Yelp demonstrates that this isn’t always so harmless.

We are left to ponder on the broader question of the pitfalls of Internet voting and the necessity of transparency in things involving voting. If the Webby Awards had not decided to hide the voting distribution in the last hours of voting, would the perpetrator have risked exposure and still pulled this off? Or would s/he have simply done it in a more subtle way?

As a final note, if you decide to write about this, please don’t link to Apartment List as a result of this post. This event shouldn’t lead to an SEO windfall.

UPDATE: Thanks to those of you who pointed me to a blog post from a few days ago by Apartment List – the explanation they provide is that they asked people not to confirm their email addresses after voting, and instead forward the confirmations to them so they could confirm them near the end. That would explain this strange voting pattern, but I’m skeptical, since their public voting campaigns didn’t mention anything about this convoluted step, and I think the number of people they would have had to get to do this privately would have needed to be very large. The Webby admins doubtless have a record of when the votes were originally cast, though, and if it is legitimate, it’s a pretty neat trick to avoid counter voting drives.

UPDATE #2: PadMapper won the Webby for best Real Estate site! Still no response from them about the People’s Voice side of things on whether they checked out the votes, though.

UPDATE #3: Thanks to commenter Mike, who dug up a contest they ran for votes in exchange for a chance to win an iPad. Not as shady as running a bot or hiring people directly, but still basically buying votes. It doesn’t seem to have been extremely successful, though (39 comments from people who voted), and their directions don’t mention not clicking the confirmation email, so I don’t think this explains the vote spike in the last two hours. It seems to have been taken down, which is a bit suspicious. Here’s the cached copy: Cached contest page

This entry was posted in Uncategorized. Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.
  • Vote on HN

20 Comments

  1. evan
    Posted April 30, 2012 at 2:40 pm | Permalink

    Did you read their latest blog post? I’ll honor your request by not linking it, but I think you should read it before accusing them of wrongdoing.

    • Posted April 30, 2012 at 3:25 pm | Permalink

      Yeah, I saw it thanks to some people pointing it out over on HN (and thanks to you too) – I just updated the end of the post. That strategy would create a similar voting pattern, potentially, though it would probably have taken a large number of those secret votes to do it, so it’s a bit of a shaky explanation – none of their public voting campaigns mentioned that convoluted step, and it would have been hard to get people they didn’t know personally to jump through those hoops. If that’s actually what happened, though, I’d be pretty impressed.

  2. Ian Rogers
    Posted April 30, 2012 at 3:15 pm | Permalink

    Ouch. Sorry to hear that! That’s a bummer. I agree in it’s fishy-ness. Here’s the thing- if it was bot or Turk or something similar, then you guys know that you won the race and beat the competition anyways. Keep up the good work. If the competition has to do shady tactics, their success won’t last forever. Karma will reign!

  3. Posted April 30, 2012 at 3:42 pm | Permalink

    I find it kind of shame that companies/users go for ‘unnatural’ voting like those mentioned – using bot is obviously illegal and should be thrashed if/once detected. Using crowdsourcing such as mTurk for voting is also ‘not natural’ as there is no affinity towards the brand. Finally there is a possibility of quick advertising such as Facebook ads or others. Again the fairness for such should be under inspection. Esp. considering underdogs competing with bigger players, such quick and paid tactics potentially wipes off the competition.

  4. Roi
    Posted April 30, 2012 at 3:52 pm | Permalink

    Devastated! Seems like a foul play to me.

    If that’s what actually happened, I’d be more impressed for the fact that both of you fought til the end against bigger players here. If anything, their underdog clip fits you better.

  5. Shan
    Posted April 30, 2012 at 4:20 pm | Permalink

    AL’s post reminds me of this famous Chinese anecdote: http://history.cultural-china.com/en/38History6906.html

  6. Sergey Kornilov
    Posted April 30, 2012 at 5:40 pm | Permalink

    Sorry to hear that, hopefully Webby admins can sort this out. Unfortunately most of web votings are easily manipulated and what Apartment List did looks legal though still fishy.

    Anyway it’s a great achievement for two guys shop. Congratulations!

  7. Peter Holmes
    Posted April 30, 2012 at 11:51 pm | Permalink

    Seems easy enough to resolve to me. Either the logs either have a bunch of legit votes or they don’t. Do we need some sort of petition or can you just ask? Refusing to verify these accounts would seem to create a much larger problem for Webbys. Behold the internet.

    • Posted May 1, 2012 at 12:35 am | Permalink

      Maybe, they haven’t responded to inquiries about this, so I’m not sure whether they’re looking into it or not. But it seems like it would be fairly obvious if they were looking at the data.

  8. Phillip Birmingham
    Posted May 1, 2012 at 5:27 am | Permalink

    >it’s a pretty neat trick to avoid counter voting drives.

    I think it’s pretty sleazy, myself. I’m unlikely to be in the apartment market again, but I will remember this if I am asked for recommendations.

  9. aaron
    Posted May 1, 2012 at 5:39 pm | Permalink

    dude! you spoiled game of thrones for me, I’m on page 600/800! you didn’t redact enough. ugh!

    • Posted May 1, 2012 at 5:46 pm | Permalink

      It might not be who you think it is, I don’t think there’s too much foreshadowing if memory serves. If I actually did spoil something, though, I’m really sincerely sorry. I’ve reworded that part to have two redacted parts.

  10. Brennan Caballero
    Posted May 1, 2012 at 9:14 pm | Permalink

    By just looking at those graphs you can tell something unnatural happened. Sorry you got screwed out of your deserved award. However, glad to hear you got the judges award, that is something. I hope you can petition the Webby Award people to examine this phenomenon; I voted in a lot of categories and now I wonder if all that mattered was a well timed bot. Anyways, the cream will rise to the top.

  11. Paranoid About Polls
    Posted May 2, 2012 at 8:13 am | Permalink

    Or a more corrupt hypothesis:

    5) Webby Awards accept kickbacks for 1st place and close the voting so they can adjust the paid-for winner upwards. They lazily implemented this as linearly adding votes.

    • Posted May 2, 2012 at 11:39 am | Permalink

      I would doubt that. It would take such a small amount of money to buy 2000 votes on mechanical turk (at 5-10 cents a vote, it’s ~$100-$200) that there’s no reason for anyone to bribe anyone.

  12. Mike T
    Posted May 3, 2012 at 8:02 pm | Permalink

    I’m into the whole internet voting contest thing.. I’ve done okay with it, but I’ve seen all manner and sort of shadiness. I’ve seen this type of thing before. I was in a contest where another contestant was submitting friends’ addresses for them and asking them to forward the “confirm this vote” emails to her, presumably so she could click them later.

    If you still want to get your panties in a bunch, check out this google cache entry related to their strategy.

    • Posted May 3, 2012 at 10:15 pm | Permalink

      Heh thanks for the cached page, the original is giving a page not found now… I guess they deleted it for some reason, maybe because they claimed on their blog post that it was “friends of friends” that got them their votes, when they were actually persuading people on other sites to vote with the chance to win a free iPad.

      Something still doesn’t add up, though – that post doesn’t tell people not to click the confirmation themselves as they claimed, and it tells them to post a comment when people voted, and there are a grand total of 39 comments on there, not nearly enough to make up the 1000 votes they claimed, even if everyone voted 5 times. I think this contest and video were probably their original plans, but they failed to put them in first.

  13. Michael Aye
    Posted May 9, 2012 at 1:13 am | Permalink

    Is it really common for the contest to close its curtains close to the end? Did you check if that’s a common behavior? Because that’s where the fishiness started for me in your story.

    Thanks for padmapper.com, I’m currently in the market and it *actually* makes apartment search almost fun! Your interface is so good, it’s sick, man! ;)

    One suggestion though for the GUI: It was not clear to me that one has to click on My Places and My Profile a 2nd time to close the pop-up windows. (And if it’s not clear to me, Internet-user since 1993, I believe many will stumble there as well). Could you not maybe add a little (x) in the upper right corner to close the pop-ups? Cheers!

  14. Posted May 14, 2012 at 7:34 am | Permalink

    Eric, don’t worry about this award. You will win a bigger reward if you keep this track going. The thing is, Apartment list does not have what you have… Cleverness. You took an Idea that I’ve always liked, and put features in that are so forward thinking, and logical, that the other sites look like they were made for IE6. Add your program an LAYAR, and you’ll have the coolest app ever. People have been waiting for your site, ERIC! not anyone elses. I’ve been to many open mic concerts where the true artists didn’t win because their fans weren’t there, but EVERYONE knew who was the best. The thing is, it’s showbusiness. Get yourself an investment, and go kickass. You site is like hotpads, but I like yours more. It hits a specific market, and you’ll go far. If I had a million dollars, I’d invest in you.

    Take it easy, and let me know if I can be a part of your project at all.

  15. Mike
    Posted June 16, 2012 at 5:03 pm | Permalink

    The webbys are bullshit anyway. It’s always about who can stuff the ballot the most.

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>