We heard feedback loud and clear from the developer community on the Salesforce1 Hackathon we ran at Dreamforce.  We take feedback seriously and, as a result, our internal audit team conducted a comprehensive review of the eligibility requirements and judging process.  We want to discuss all of the issues transparently, explain how we made some key decisions, and share the results of the review.

The internal review found that the winning team, Upshot, met the hackathon’s eligibility requirements, and that the app they submitted adhered to the rules of the hackathon.  It also found that we weren’t clear enough with the final round judges about the use of pre-existing code.

So here’s what we’re going to do: we are declaring a tie and we are awarding each of the top two developer teams with the grand prize of $1 million.  Both Upshot and Healthcare.love built incredible apps on the Salesforce1 Platform and both deserve to be recognized.

Let’s address the biggest questions we heard from the community:

Did the winning team break the rules?

• The internal audit team found that the app submitted by Upshot, the winning team, adhered to the rules.

• UpShot as a team was eligible under the rules of the hackathon.  One of the members was a former salesforce.com employee, but based on his departure date from the company, he was eligible.

• We reviewed the source code of Upshot’s submission.  It showed that Upshot started with some existing code, but the mobile app was created within the contest window and a majority of the submission was new code created during that time.

• While the Upshot mobile app used some pre-existing code, this did not violate the rules.  Use of pre-existing code was allowable as long as the code didn’t comprise the majority of the app and didn’t violate any third party’s rights.  Our internal review determined that Upshot’s mobile app was created during the hackathon and met these criteria.

Was the Healthcare.love team eligible to participate?

• We heard a concern raised about the participation of the Healthcare.love team because they are employed by a company in which salesforce.com holds a small equity stake.  We determined that the team was eligible to participate because such investment is immaterial and salesforce.com has no ability to control the referenced company.  As a result, it is not a “salesforce.com-related” entity under the rules.

Were the rules clear to everyone?

• We should have been more clear to teams and to the final round judges about the use of pre-existing code.  This is complicated because essentially every app uses pre-existing code, APIs and/or libraries.  Our intention all along was that to qualify for submission, the use of existing code in the app was ok so long as that code didn’t violate any third party’s rights and didn’t comprise a majority of the app.  In other words, a mobile app previously created for another purpose would not qualify.  We didn’t do a good enough job explaining to final round judges how to take into account the use of pre-existing code.  We didn’t get this right.  We should have been clearer.  We don’t know whether this would have changed the outcome of the final round of judging or not.

Who were the judges and what was the judging process?

• Our first and second round judges were all salesforce.com employees.  We trained them for 90 minutes on how to evaluate apps based on the criteria: innovation, business value, user experience and use of the Salesforce1 Platform.  We had technical staff available to build and run apps, review code and advise the judges.

• During the final round, in an effort to ensure impartiality, five of the six judges were not salesforce.com employees.  But we didn’t adequately equip these judges with enough information to ensure that their scores for the “innovation” criteria took into account use of pre-existing code.

Did you review every app?

• Every submission was reviewed.  In the first round, 73 judges conducted more than 1,000 reviews of 150 complete submissions.  First round judging took place from 6pm – 10pm on Nov. 20.

• The top 20 advanced to the second round of judging which took place from 11pm – 3am.  In this round, 18 judges conducted 80 reviews to select the final 5 apps.  These 5 entered the final round unranked.

• We instructed both our first and second round judges to evaluate the submissions using the apps’ description, screen shots and the demo video using the same four criteria: innovation, business value, user experience and use of the Salesforce1 Platform.  We generally relied on the videos for the initial rounds of judging.  In some cases technical advisors helped the judges inspect code, or build the apps with Xcode.  Our judges reviewed every submission, but we passed zero feedback to participants.  This was a big miss.

• We sent an impersonal email to all participants to let them know that they did not advance to the final round.  We wanted everyone to know whether or not they needed to be onsite the next day, or if they could sleep in.  This was a lame mail to send at 3:30 in the morning.

Why no live demos?

• Most hackathons include live demos for judges, but we didn’t.  This was a mistake.  The net of this is we didn’t give every team the experience they deserved.  We didn’t let them demo their work live, and they didn’t get feedback from peers and judges.

• The gallery was closed and the judging was conducted in private, so participants didn’t know why they didn’t advance to the finals.

• Here’s what we’re going to do: we are going to make the gallery public.  We have reached out to every team to ask their permission to post their submission.  The apps are amazing, and you can check them out here on http://salesforce1million.challengepost.com.

Were any teams given preferential treatment?

• No.  No teams received preferential treatment on technical support or in the judging process.  Our technical staff was available on a first-come, first-serve basis.

• Three submissions showing as posted after the 6 PM deadline were allowed through to judging.  Two of these posted within seconds after 6 PM but it could not be shown conclusively whether they had been submitted immediately before or after the deadline; their delay in posting may have been because of the queue in the system processing the many large last minute file submissions.  The third team had been actively troubleshooting a technical issue with the submission system operators when the deadline occurred. Given the circumstances, we felt it was only fair to allow them to proceed to judging.  None of these 3 entries were finalists.

Our commitment to you

The Salesforce1 Hackathon was at a scale we’ve never tried before.  We hope that participants recognize our goal was to run a fair hackathon.  The apps that came out of this challenge were stunningly good — far beyond the top 5.  Many of these should be in the Salesforce AppExchange or other app store and create a great business for their authors.  We don’t want your code; we want you to ship apps.  We want you to make awesome stuff and make money.

Here’s what we intend to do next.  First, we’re going to showcase all of the amazing apps online.  Second, we’re inviting every team to participate in our AppExchange Accelerate program to help teams get funded and get their apps to market.

Finally, we plan to do more Salesforce1 Hackathons!  We’ll partner with an outside firm to execute them.  It’s not our core competency; we’re in the business of APIs and platform.

We want to thank everybody for their hard work, honest feedback and amazing apps.  I hope that every developer that participated gives us feedback on how to do it better next time.  Please email us: hackathon@salesforce.com

Adam