We heard feedback loud and clear from the developer community on the Salesforce1 Hackathon we ran at Dreamforce.  We take feedback seriously and, as a result, our internal audit team conducted a comprehensive review of the eligibility requirements and judging process.  We want to discuss all of the issues transparently, explain how we made some key decisions, and share the results of the review.

The internal review found that the winning team, Upshot, met the hackathon’s eligibility requirements, and that the app they submitted adhered to the rules of the hackathon.  It also found that we weren’t clear enough with the final round judges about the use of pre-existing code.

So here’s what we’re going to do: we are declaring a tie and we are awarding each of the top two developer teams with the grand prize of $1 million.  Both Upshot and Healthcare.love built incredible apps on the Salesforce1 Platform and both deserve to be recognized.

 

Let’s address the biggest questions we heard from the community:

Did the winning team break the rules?

  • The internal audit team found that the app submitted by Upshot, the winning team, adhered to the rules.

  • UpShot as a team was eligible under the rules of the hackathon.  One of the members was a former salesforce.com employee, but based on his departure date from the company, he was eligible.

  • We reviewed the source code of Upshot’s submission.  It showed that Upshot started with some existing code, but the mobile app was created within the contest window and a majority of the submission was new code created during that time.

  • While the Upshot mobile app used some pre-existing code, this did not violate the rules.  Use of pre-existing code was allowable as long as the code didn’t comprise the majority of the app and didn’t violate any third party’s rights.  Our internal review determined that Upshot’s mobile app was created during the hackathon and met these criteria.

 

Was the Healthcare.love team eligible to participate?

  • We heard a concern raised about the participation of the Healthcare.love team because they are employed by a company in which salesforce.com holds a small equity stake.  We determined that the team was eligible to participate because such investment is immaterial and salesforce.com has no ability to control the referenced company.  As a result, it is not a “salesforce.com-related” entity under the rules.

 

Were the rules clear to everyone?

  • We should have been more clear to teams and to the final round judges about the use of pre-existing code.  This is complicated because essentially every app uses pre-existing code, APIs and/or libraries.  Our intention all along was that to qualify for submission, the use of existing code in the app was ok so long as that code didn’t violate any third party’s rights and didn’t comprise a majority of the app.  In other words, a mobile app previously created for another purpose would not qualify.  We didn’t do a good enough job explaining to final round judges how to take into account the use of pre-existing code.  We didn’t get this right.  We should have been clearer.  We don’t know whether this would have changed the outcome of the final round of judging or not.

 

Who were the judges and what was the judging process?

  • Our first and second round judges were all salesforce.com employees.  We trained them for 90 minutes on how to evaluate apps based on the criteria: innovation, business value, user experience and use of the Salesforce1 Platform.  We had technical staff available to build and run apps, review code and advise the judges.

  • During the final round, in an effort to ensure impartiality, five of the six judges were not salesforce.com employees.  But we didn’t adequately equip these judges with enough information to ensure that their scores for the “innovation” criteria took into account use of pre-existing code.

 

Did you review every app?

  • Every submission was reviewed.  In the first round, 73 judges conducted more than 1,000 reviews of 150 complete submissions.  First round judging took place from 6pm – 10pm on Nov. 20.

  • The top 20 advanced to the second round of judging which took place from 11pm – 3am.  In this round, 18 judges conducted 80 reviews to select the final 5 apps.  These 5 entered the final round unranked.

  • We instructed both our first and second round judges to evaluate the submissions using the apps’ description, screen shots and the demo video using the same four criteria: innovation, business value, user experience and use of the Salesforce1 Platform.  We generally relied on the videos for the initial rounds of judging.  In some cases technical advisors helped the judges inspect code, or build the apps with Xcode.  Our judges reviewed every submission, but we passed zero feedback to participants.  This was a big miss.

  • We sent an impersonal email to all participants to let them know that they did not advance to the final round.  We wanted everyone to know whether or not they needed to be onsite the next day, or if they could sleep in.  This was a lame mail to send at 3:30 in the morning.

 

Why no live demos?

  • Most hackathons include live demos for judges, but we didn’t.  This was a mistake.  The net of this is we didn’t give every team the experience they deserved.  We didn’t let them demo their work live, and they didn’t get feedback from peers and judges.

  • The gallery was closed and the judging was conducted in private, so participants didn’t know why they didn’t advance to the finals.

  • Here’s what we’re going to do: we are going to make the gallery public.  We have reached out to every team to ask their permission to post their submission.  The apps are amazing, and you can check them out here on http://salesforce1million.challengepost.com.

 

Were any teams given preferential treatment?

  • No.  No teams received preferential treatment on technical support or in the judging process.  Our technical staff was available on a first-come, first-serve basis.

  • Three submissions showing as posted after the 6 PM deadline were allowed through to judging.  Two of these posted within seconds after 6 PM but it could not be shown conclusively whether they had been submitted immediately before or after the deadline; their delay in posting may have been because of the queue in the system processing the many large last minute file submissions.  The third team had been actively troubleshooting a technical issue with the submission system operators when the deadline occurred. Given the circumstances, we felt it was only fair to allow them to proceed to judging.  None of these 3 entries were finalists.

 

Our commitment to you

The Salesforce1 Hackathon was at a scale we’ve never tried before.  We hope that participants recognize our goal was to run a fair hackathon.  The apps that came out of this challenge were stunningly good — far beyond the top 5.  Many of these should be in the Salesforce AppExchange or other app store and create a great business for their authors.  We don’t want your code; we want you to ship apps.  We want you to make awesome stuff and make money.

Here’s what we intend to do next.  First, we’re going to showcase all of the amazing apps online.  Second, we’re inviting every team to participate in our AppExchange Accelerate program to help teams get funded and get their apps to market.

Finally, we plan to do more Salesforce1 Hackathons!  We’ll partner with an outside firm to execute them.  It’s not our core competency; we’re in the business of APIs and platform.

We want to thank everybody for their hard work, honest feedback and amazing apps.  I hope that every developer that participated gives us feedback on how to do it better next time.  Please email us: [email protected]

Adam

 

tagged , , Bookmark the permalink. Trackbacks are closed, but you can post a comment.
  • http://blog.shivanathd.com/ Shivanath D

    @adam … Very clear post … ! in the end its all about developers being able to have fun and experience the Technology 1st hand !

    • Telm Perez
      • atn77

        lol up vote just because it’s a hilarious video

  • Colabi

    and what of the fact that healthcare.love was built by employees of a salesforce related company? expressly forbidden by the rules? as reported here: http://www.bizjournals.com/sanfrancisco/blog/2013/11/salesforce-hackathon-dreamforce-taptera.html

    • Ward

      The answer to this is in the article. “We determined that the team was eligible to participate because such investment is immaterial and salesforce.com has no ability to control the referenced company.”

      • Colabi

        unfortunately, either there is an agreement that follows the terms or there is not. i would argue that when the company received the money, they probably thought it was REALLY material. would they like to give that money away? if so, i can think of lots of people who will take it.

  • Colabi

    and what of the fact that healthcare.love was built by employees of a salesforce related company? http://www.bizjournals.com/sanfrancisco/blog/2013/11/salesforce-hackathon-dreamforce-taptera.html

    • Adam Seligman

      Hi, thanks for all the feedback. We want to be very clear on this. Salesforce.com has reviewed the claim that the entrants from Healthcare.love were ineligible to participate in the Salesforce1 Hackathon because they are employed by a company in which salesforce.com holds a small equity stake. The review determined that the entrants were eligible to participate because such investment is immaterial and salesforce.com has no ability to control the referenced company. As a result, it is not a “salesforce.com-related” entity under the rules. This rule was intended to prohibit participation by employees of salesforce.com subsidiaries, such as ExactTarget.

      • Colabi

        unfortunately, it wasn’t subsidiary. it was stated as salesforce related. and unfortunately, making an investment makes it a relationship. is that investment free of any and all controls? do you relinquish all rights to the monetary stake of that investment? if so, when did that happen?

      • Colabi

        btw, can you confirm if that small equity stake is 2 million dollars as was reported in Biz Journals?

        • NotThe1950s

          Well, let’s be fair– it’s now 3 million.

          • Colabi

            i thought they were just regular guys, but the heathcare.love people were the people closely related to receiving the investment (CTO, head of product). EricSF over at the TechCrunch article made a funny observation that:

            Here’s the description: “What is a related company? A company in which another company makes a long-term capital investment in order to gain control or influence.” http://www.investorwords.com/10871/related_company.html

  • Mike Leach

    Well done. Congrats to both UpShot and Healthcare.love.

    Eagerly awaiting more details on the AppExchange Accelerate program.

  • MikeBZ

    Very generous and gracious way to handle the situation. Programming competitions are really tough… more detailed opinion is here: http://www.mikebz.com/2013/12/the-dreamforce-million-dollar-hackathon.html

  • us0r

    “Finally, we plan to do more Salesforce1 Hackathons!” – Which this nonsense will be brought up. Get that spin machine going.

    • Colabi

      if it’s even allowed. an investigation will be starting soon with different agencies as to the conduct of this contest. they drew in people from around the globe for this mess. and sullied the image of this city and industry. letting another contest happen is an open liability.

  • Colabi

    Dear Salesforce, I realized that I may have misunderstood this whole thing. Would you just like to punch the developers in the face? It feels like that’s what you want to do. So if you need to punch a developer in the face, we can nominate one of us who will take it.

    I didn’t really think you could make a worse decision than rewarding one ineligible app. But you can. Just reward both ineligible apps!

  • Ted Patrick

    To be fair, this is Salesforce’s event, Salesforce’s rules, Salesforce’s rule interpretation… because it is Salesforce’s $$$. This was not a classic hackathon, it was something new and to be honest Salesforce deserves credit for innovating and deserves the license to make mistakes and iterate. I have yet to see an event like this from Oracle or SAP. Shoot even during the controversy, they error on the side of the developers, that says a ton. I participated and I plan to participate next time. My 2 cents. Ted :)

    • Colabi

      i think the problem is – assuming it was honest to begin with – was that they set expectations they had no intention of living up to. but now that we know it’s a salesforce world, i think most people will spend their time elsewhere. btw, to error on the side of developers on this one would have been to rejudge this thing.

      • Ted Patrick

        The expectation was 1 million would be given away. 2.3+ million was the final number. I entered expecting to pay $99, that was refunded. Expectations exceeded.

        • Colabi

          I think if you were like the Romanian team that brought 4 people and their hard work to America just to have their video watched (their rules, not ours), you would have a different point of view. Think you could have let Team Romanian submit their video online? Salesforce has pretty successful crapped on the dream and fundamentally misunderstood what pissed people off. Developers don’t trust these things. But they raised everyone’s hopes that it was a real thing. And those hopes were dashed on the rocks. It’s like It’s A Wonderful Life. But in this holiday version Mr. Potter kicks George Bailey and the town in the a$$.

          • Ted Patrick

            In the words of William Schatner, “Get a life”.

          • Colabi

            i get to be vocal because this isn’t my industry. you’d be surprised how many people are holding punches cause they have to do business with them.

          • Colabi2

            sorry Colabi1, but as an objective party that knows little about this “i get to be vocal” and point out that you sound bat-shit crazy.

          • Colabi

            those are pretty stern words from the objective anonymous observer who just copped my userid. or should i respond to you as Coward2?

      • MikeBZ

        Colabi, rejudging this thing would be the biggest mess. I am not sure how many Hackathons you were a part of or organized but the issue is that you only have 2-3 hours to judge a bunch of apps. You can have a problem with the way things panned out all you want, but you have no real suggestions on how to handle this situation in the moment. Endless recounts don’t work for hackathons. Participants themselves usually want to know by the end of the day whether they came out to be on top or not.

        • Colabi

          Really? The entries exist. Judges abound. This wasn’t my first dance. I was at Facebook the previous week and won. We sat through 2 hours of presenting – even their head of product to his credit – and everyone had a good time. That was for an iPad. Now, I’m no genius but I think for a REAL shot at a million dollars 100 or so teams would probably endure a weekend day where they got a chance to be seen and could come out as a legit winner. But then again, maybe their patience tops out at an iPad?

          Just everyone realize they just tried to spend their way out of the problem with A MILLION DOLLARS. In the wrong direction. Comicly wrong. Think 100K could have gone to a day event and 900K to the prize?

          • MikeBZ

            Colabi, your suggestion to create another event is a logistical and PR headache. As you have mentioned yourself some people flew form Romania for this event and would you suggest that they fly out again? Again this is not a payday but a competition. Only a few people win and most of the people will be disappointed. I wrote a blog post on this which basically ends with this – in all competitions work on a project that you would work on even without the prize.

          • Colabi

            Give that nightmare to us to manage with a budget. If they REALLY are interested in saving face, fly in one rep from each team for the day (still, we haven’t added up to that last million they just wasted). If that’s too much, Skype? Technology to the rescue. At the end of the day, there is a solution. And doing the right thing – the actual right thing – will stop this wound from festering.

          • Mike Leach

            Agreed. Most Salesforce Developers know that Dreamforce is the one true window for launching new apps every year. This explains why so many apps had pre-existing momentum going into the hackathon.

            There were many ‘hacks’ that were essentially a mobile façade to existing apps, data models, and APIs. So what. This made them better apps in the end.

            To your point, the healthy outlook and expectation should be to create something that can ultimately be monetized on the AppExchange and bring in $1M+ per year in subscription revenue. Don’t bet everything on a one-time, subjective contest where you have no control over judges.

            Sure, the $1M would have been a nice stepping stone, but the apps that truly met the “business impact” Hackathon criteria are now moving on to real customers and will eventually recoup their investment in the hackathon.

          • Guest

            I don’t think anyone would argue with this – you should work on an app that you think is worth while, and not depend on winning money at a contest. But this statement does nothing to address the issues raised about this contest.

            For the Salesforce faithful, it is understood that this is how Salesforce has always run the contest and quiet honestly why I didn’t bother trying to come up with a grand idea and try and implement it in the time frame given. Prior experience with the sf hackathon dictated that if I didn’t already have an app that I had been working on for the last year, there’s no way I’d be winning anything at this hackathon. It’s always been obvious that submissions were not worked on within the hackathon’s time frame something a seasoned SF developer would know but a new developer would not.

            The problem – the PR nightmare they are facing now is because they offered 1 million dollars and actually attracted new developers who had never used the platform before. Those developers, the ones SF should be eager to attract left with a bad taste in their mouth, that they have been had. That this wasn’t a real contest, just another part of the partner exhibition where SF awarded a million dollars to some of their partners / long time supports.

            What we’re saying to those new developers is: So what that Salesforce didn’t follow their contest rules to letter, it’s not really about the contest it’s about the learning and building marketable apps that you can make money from. If you win the million that’s just icing on the cake, so shove off and stop whining.

            I don’t think that’s a good message to send new developers to the platform, and I don’t think this is something Salesforce’s unpaid evangelists should be jumping to their rescue about (although it probably is worth a few brownie points ;) ). It appears that they didn’t follow their own rules, did some half hazard finger to the wind Judging and expect everyone to be happy about it because that’s just the way salesforce does things.

            If this is about trust, this does nothing to address the trust issue with developers outside the SF community, who thought they were entering a real hackathon, not another SF app showcase with a million dollar award to the best (looking) app with the most marketing like video.

  • mattlacey

    It’s pretty clear that with this the biggest fall-down was in the judging. The healthcare.love guys fell within the rules as determined by Salesforce, and since they were acting as a team separate to their company I really don’t have any issue with those guys taking out a prize. Many of us use and develop on the platform every single day, does that mean we should be ruled out just because we know it a little better than some?

    As for Upshot, it’d be great to know exactly what was built for the competition, I’m still a little skeptical about that one but their overall solution is excellent and really does solve a genuine user problem in a very effective way. There are other apps I might have picked above both of these entries but without full insight it’s impossible to say.

    “We’ll partner with an outside firm to execute them. It’s not our core competency; we’re in the business of APIs and platform.” — I, for one, would definitely enter another Salesforce hackathon if the judging were handled in a more transparent fashion and using a third party would definitely facilitate that. At the very least if another one were to be run then the name of the game would be transparency and openness, lest this happen again so it’s guaranteed to be run more proficiently. Am I gutted I didn’t win $1m? Of course, but then when there’s 150 entries the chances of winning are always going to be slim, but I’m game for another crack at it. Queue the down-votes ;)

    • Rupert Barrow

      +1 re. Upshot : to close the discussion and remove any doubt, it would be nice to know what part of the app was pre-existing and shown at that Meetup which took place before the hackathon.

      On another line, I wanted to point out that this was the first contest organized by Salesforce which was NOT restricted to citizens of US, Canada or UK : well done Salesforce (Legal).

    • atn77

      Great points, although I’m still sceptical that healthcare.love is within the rules. People are correct in saying the term used in the official rules was “related”. IANAL so I can’t say for certain what it means legally. It sounds like invested in would fall into the “related” category but maybe not :o) .

  • Aaron Auhin Bagchee

    As suspected. Judging based on idea and how well video presentation was done. Not looking at functioning app and code is criminal.

    • Aaron Auhin Bagchee

      Healthcare.love would not have won if they had a low tech video.

  • Steve Schneider

    The transparency is very nice to have. It is clear to me that Salesforce has been making a tremendous effort to be fair.
    Regarding the existing code issue, most coding is copying and pasting from examples. We need example code to get started, and we were leveraging Salesforce APIs and other libraries to quickly build apps with impressive features. It is possible that any entry could be examined and objected against, but at some point we need to recognize that the winning teams did excellent jobs and should be congratulated. It isn’t easy to watch others receive tremendously generous prizes, but that’s something we need to deal with and not bash Salesforce for. It is just human nature.

    Salesforce engineers were extremely patient and stayed around the clock with us to assist and make our DevZone experiences as pleasant as possible, and then many of them stayed up the last night going through all submissions. I’m sure it wasn’t easy for them, but I found them to be very friendly and professional and enjoyed my experience in spite of wrestling with bugs for so long and trying to work my phone and talk while taking a movie (I was a one-person team).
    Note that the playing field wasn’t completely even since there were teams of different sizes, and individuals like me couldn’t hope to get as much done as very large teams. However, that was the nature of the contest, and I went into it knowing that this was the case. Interestingly, the top two teams were on the smaller size. The nature of the contest was that there was a lot of variation in teams and projects, and this may have been the creativity that Salesforce was looking for.

  • Eric S.

    Adam, you claim that Taptera (the company the Healthcare.love people are in the senior leadership/executive team of) isn’t “related” because Salesforce doesn’t “control” it. But the definition of “related” in legal/financial use means “a related company is one in which another company makes a long-term capital investment in order to gain influence.”

    Salesforce may not control Taptera directly, but it is an owner, investor, and INFLUENCER. Unless something is extremely unusual, Salesforce can elect the board of Taptera and vote on shareholder issues. And that means that Salesforce is clearly able to influence Taptera’s direction.

    • Colabi

      it’s almost amusing that they spent a million going in the wrong direction. think it would have cost a million to rejudge the competition? i think it would have cost a little and they could have given the million to a real winner. instead, the tactic seems like ‘let us bribe you off with possibly getting financing for you that probably won’t happen’

  • Patrick Riley

    Adam, show me where in your official rules PDF doc does it say you can use any past composed code. There is no ambiguity there. The issue is Salesforce specific said you can’t start or use existing code before the date (still there in the rules PDF). I know that many very talented teams practically killed themselves followed that rule even though they could have used some of their older repos and did something much more impressive than Upshot. Then Salesforce respond to a forum question, weeks after the hackathon started, saying “yeah sure, just not a significant amount”. Most programmers were heads down in their code and didn’t check the optional forum. That’s wrong and you did nothing to explain or correct this.

    You should have taken a Google Android App Contest approach and given programmers many months to built something awesome for your platform if that’s what you wanted and award many apps. Instead, you guys messed up, made rules, cite a forum post as a rule change, are still wasting all of our time with this BS internal and external review, and not apologizing to the developers who worked hard following your rules.

    http://res.cloudinary.com/hzxejch6p/image/upload/v1384900393/Final_1M_Hackathon_2013_Rules_4_hcwqgs.pdf

  • Dan Appleman

    I think the process is as interesting (or more interesting) than the particular solution: http://danappleman.com/2013/12/02/to-be-a-customer-company/

    • guest1

      Dan, you are part of the salesforce ecosystem. A quick peek at LinkedIn shows you work for Full Circle CRM. Hmm, I wonder how much credibility you have?

      • Dan Appleman

        I’m also a Force.com MVP and author of a book on advanced Apex. So what you’re really asking is whether I have integrity or not. That’s for others to judge. I suggest you judge it based on what I’ve posted, and not who I may or may not be associated with. If you find anything factually incorrect about anything I’ve said, I will be glad to apologize and correct it immediately. If you find anything I’ve stated as fact that is just personal opinion, I will promptly clarify to distinguish the two.

        I also do not hide behind anonymity, so if you wish to question my integrity, at least have the guts to do it under you real name.

        • guest1

          what crap are you talking about “customer company” when they swindled intentionally/unintentionally thousands of their own customers/prospects and lost shitload of credibility? A “customer company” would have manned up, said they committed mistakes and re-do the hackathon judging. I don’t know who is running their PR but this is not a “customer company” in any shape or form.

          Yes I can use the real name when I am singing praises and playing a sycophant.

          • Dan Appleman

            I use my real name when criticizing companies as well – as Salesforce knows full well. I also find it possible to engage in a conversation and debate without use of name-calling and obscenity.

          • guest1

            This is what a customer company does :) No gimmicks!
            http://motorola-blog.blogspot.com/2013/12/we-owe-you-apology.html

            This is my last response on this thread. Good Luck!

          • Dan Appleman

            There I disagree. While it was indeed a nice apology, it failed to address the real problem that wasn’t technical – the fact that there was virtually no communication all day and thousands of customers were left hanging thinking the site could come up any minute. All Motorola is offering them is double the chance of possibly getting a phone – which given demand could still be a lottery – the phones may sell out in minutes. Note how they carefully never said how many phones they were offering, or how long they expect it might take for them to sell out.

            If they were a customer company, they would have addressed this – perhaps by offering discount vouchers to those who already had phones in their carts that they were unable to purchase, or guaranteeing at least one hour’s worth of availability, or some other mechanism to make sure that those customers who were loyal to them were able to get at least one phone at the offered price.
            The apology was nice – but the offer was, in my mind, a gimmick.

  • raj dn

    Adam,
    Can you publish the 20 teams which was in 1st round

  • Jim Raynor

    SHAME! SHAME! SHAME! If the original results were sham, the cover up has been horrible. To be honest, I never expected Salesforce to go back and admit that they goofed up but to award another million dollars to one of your invested companies – it’s a slap on the face of every other developer who participated in the hackathon and yet another testimony that the hackathon was rigged from the outset. You call this a ‘full and fair review’ – awarding million dollars to an ex-employee and one of your investments and not even disclosing the top 20 apps?

    @Colabi:disqus – You deserve a lot of praise for standing up and fighting for what’s ‘right’. This post and the comments highlight how Salesforce has decided to suppress all voices who’ve been demanding a fair judgement. You’ve the stalwards of the Salesforce & tech fraternity – Dan Appleman, Mike Leach, Ted Patric lending their support to the cover up – two wrongs don’t make a right!

    • Colabi

      Thanks, Jim. And thanks to you too. As one observer observed, I should get a life. I’ve got a life. What I haven’t got is a hobby! So between elections, fraudulent competitions beware!

  • hohoho

    .. it would be interesting if SalesForce could explain that “UPSHOT existing code did not comprise the majority of the app” … the fact that Upshot is using “BootStrap.JS” – it would means that “mobile part” is just a matter of including Bootstrap JS & CSS – and add class definition in the HTML, no?

    http://webcache.googleusercontent.com/search?q=cache:http://beta.upshotdata.com/demo

  • MrSpartanification .

    After a detailed analysis of this whole problem, i conclude that salesforce is the real winner of this HACKathon.

    @adam: salesforce should award Colabi some money, he seems pissed. as badger said in breaking bad, “why he can’t throw a brotha a bone”. :P

    • Colabi

      10 pts breaking bad reference! As Mike would say, ‘No more half measures’. RIP, brother. You should have gotten to Belize.

  • atn77

    Disclosure: (I didn’t participate in this years hackathon nor did I attend dreamforce, I’m just an outside observer, and developer on the platform: opinions are my own and don’t reflect my employers)

    To me it seems like measured transparency (or marketing transparency if you will). Enough to get most people off their back, not quite enough to satisfy the ones that are most upset. I wonder why salesforce isn’t being even more transparent. Why not disclose what they counted as “for the hackathon” and what they counted as “prior code”. What does mostly mean? (50 + 1%) How do you even measure that if the smarts of the app are already written and all that was needed was a mobile UI. UI’s can account for a lot of code but I don’t think in any of these projects the UI is what was innovative, most of the ui is provided by salesforce1, android or IOS UI systems?

    Why not also disclose the names of the judges and their affiliations, along with the top 20 finalists. Score sheets would be incredibly helpful in determining how it was all judged. But then salesforce would open themselves up to much deeper transparency that I hope would show the company in good light, but might be a scary proposition in a tightly controlled marketing environment. They’re trying to contain the fire not fan it I guess :o). I really hope this isn’t a Lance Armstrong type deal.

    I think it’s a good move for Salesforce to contract out to an outside company for this type of event. Should’ve thought of that before this, but of course hindsight’s 20/20 . It will remove a lot of suspicion in the future.

    One final note :) to us the average lowly developer 2.3 million seems like a generous amount of cash. To a multi billion dollar company, it’s a drop in the bucket. I also suspect they make much more than that off the spectacle that is dreamforce. It’s a fantastic event but they wouldn’t be doing it, if it didn’t result in lot’s of profit, and they wouldn’t run the hackathon out of the goodness of their heart ;) don’t kid yourself there’s something it for them.

  • atn77

    One more thought I feel like I’m watching the 2002 winter olympics figure skating contest. At least the judging got better after that ;o)

  • Dan Slowinski

    I don’t know where else to chime in, but I feel like adding my 2 cents just to know if there are others who feel the way I do or if I’ve got something wrong. (I apologize for the back and forth/unclear flow of the post, I don’t know how to better outline my thoughts)

    I participated in the hackathon and adhered to the rules to the best of my ability. I have won/lost many hackathons and never have walked away feeling more upset, I was actually content knowing I had not won up until I found out that 1) No one had logged into my app, and 2) Many others noticed their apps had not been run and that the winners seemed suspicious.

    I felt I had a reasonable chance of winning knowing that most likely nobody else had read the rules/would adhere to them (so the number of qualifying entries would be low, which 150 is very low considering they claimed to have over 6,000 spots available) and that I have some level of confidence in my abilities to churn out an app, At the very least I would learn how to develop on the salesforce platform.

    Having said that, and after reviewing many of the other entries/reports, I think that salesforce has poorly executed this event and are having a hard time to make things right because they don’t seem to understand the importance of following through on your commitments. There’s no real way to make this better, you can’t rejudge at this point, there’s really no easy way to make this right. The primary issue is that people entered a contest and their entry was not even considered (I understand that Adam has posted that these entries were reviewed numerous times, but you would think that even once of those times someone would have logged into the app I created once).

    Some flaws with what’s going on in these updates:
    -Not once does Adam point out that some of the entries are ineligible because the videos are too long. (the rules said no videos over 2 minutes), I saw a few that were over 2 minutes and so that 150 number should have already been lower. You would think this would’ve been the first thing they said to rule out reviewing some of the entries.
    -There are no other statements about entries being disqualified due to the strict rules (being able to be understood, in english,etc).
    -The judging criteria are not even used to justify the 5 people who were supposedly picked at the top. Are they trying to tell use all these people scored 100% based on their rubric?
    -I don’t know how they got 70+ employees to commit to judging at such late a time to be a reasonable judge in a contest they can’t participate in because of the rules that were haphazardly adhered to (submission by 6pm, team checkins by 2pm, no longer than 2 minute video) Either my video was watched once, or they loaded it on one computer and passed it around. The numbers provided here just do not easily map to the observations of other participants. I’m more curious as to how they got so many people to give some objective score and rank all of the entries based on some composite score. Given that they have so many judges, they would have had to have written down these scores somewhere and average them otherwise there’s a complete ‘magic black box’ here where entries where somehow judged and 20 remained, then another magic black box that left 5 entries. This is the transparency the community has been asking for from the start, the winning teams being ineligible just is the icing on the salesforce fail cake.

    So the way I see it, Salesforce realized they messed up, tried making it better by giving 2nd place a million (unsure if this is in addition to what they were already given or not). which basically says to me that their perception suggests that people have an issue with how the top 5 entries panned out. (as their final decision seems to only affect those in the top 5, which just exacerbates the initial claim that the top 5 were preselected/falsely selected).

    What would I like to see? I would be completely content with my flight and maybe some other costs reimbursed just for wasting my time. There’s no way they can re-work this mess-up effectively. They could easily give each team eligible or not 1k just as a ‘we’re sorry it got this bad and to involve you.’ (hell with $1 million they could give every team of the 150 over $6k and it’s still better than last place)

    I find it funny that given it’s a software company, they should have some idea about how other events are run, so I don’t buy the ‘this isn’t what we do’ because there are many other companies that host events that are run much more fairly with lower stakes involved and are more transparent. (This is key here, you can’t have a software company in/near SF and NOT know how to run a hackathon).

    My lesson learned, don’t bet on others losing by not following rules when the house doesn’t even follow them. I am frustrated that the rules/criteria were so explicit and then at some point didn’t even matter. Besides that, no one knows why they didn’t win OR why the teams that won placed where they did (the scores/rubric should be somewhere).

  • raj dn

    Adam,

    Any plans to announce the 1st round qualifiers (20 teams as per your blog post)?

  • Dan Slowinski

    I would love to email you at [email protected] but when I tried, I got a canned auto-reply:

    Hello ,

    We’re writing to let you know that the group you tried to contact (hackathon) may not exist, or you may not have permission to post messages to the group. A few more details on why you weren’t able to post:

    * You might have spelled or formatted the group name incorrectly.
    * The owner of the group may have removed this group.
    * You may need to join the group before receiving permission to post.
    * This group may not be open to posting.

    If you have questions related to this or any other Google Group, visit the Help Center at http://support.google.com/a/salesforce.com/bin/topic.py?topic=25838.

    Thanks,

    salesforce.com admins

    Is there another email to send feedback to?

    • Colabi

      I think what you’re seeing is that this is a house of cards. That the only thing they really care about is the paint, or marketing, you see from the outside. And that nothing legit actually exists. Even a simple email address.