Salesforce.com operates its own data centers, and our infrastructure teams are as passionate about innovating as the mobile, apps, and platform groups that take the stage at Dreamforce. The DevOps Hack Day was a part of our larger DevOps internal mini-conference that Dave Mangot shared a couple weeks ago. I wanted to provide you a window into how we approach the unseen engineering activities of deploying, managing, and monitoring tens of thousands of nodes of physical infrastructure.
Engineering executives may raise the visibility of infrastructure challenges, but it’s our individual contributors that form the innovative solutions. Traditional methods of managing enterprise architecture and commodity hardware are error prone, risky, and don’t scale well. Application developers higher up in the stack have become accustomed to having an API on top of every piece of infrastructure in the data center. Our global customer base expects 24/7 uptime and near real time updates on subsystem stability issues. The DevOps Hack Day was a time to step back from prioritized and planned work, advertise an undeveloped idea to the other participants, and form a cross-functional team to deliver original solutions to the pressures described above.
Salesforce culture airs on the side of recognizing everyone, so there are multiple ways to win a hack event: the Benioff Award (innovation); the People’s Choice Award; the Made it Look Easy Award; and the Superhero Award. In the spirit of the DevOps movement, the judges gave special consideration to teams comprised of engineers from diverse roles in the technology organization. Below are highlights of the DevOps Hack Day winners and honorable mentions for outstanding effort.
Access control lists testing
Software-defined networking is an area of interest primarily for pre-deployment testing of switch and firewall configurations. One team hacked on open virtual switching technology and enabled testing of access control lists on developer workstations, which was previously impossible. This category of infrastructure innovation allows us to find patch regressions before they get deployed in an environment with expensive production-replica hardware.
Optimizing a workflow engine
Salesforce uses a workflow engine to orchestrate changes to production hosts. Time saving optimizations were made to these workflows in the hack day, including the ability to parallelize independent tasks.
Secure intra-zone message passing
Security and customer trust are everyone’s first priority here at salesforce.com. Data centers are a highly monitored and tightly controlled environment that present a challenging set of constraints for an infrastructure software developer. During the hack day, a new idea for secure intra-zone message passing was implemented, complete with encryption and authentication. Working code is a better starting point for a discussion than slides. With this prototype an interdisciplinary group of engineers and executives from security, data center automation and networking can better collaborate on the requirements for the system to join production in a future release.
Auto-responding chat robot
Another team of engineers added an intelligent auto-responding chat robot to our release communication channel. It can tell any participant who the manager is on duty; whether the rollouts are on schedule or nearing completion; and, respond to other unstructured queries for operations information.
Puppet-in-a-Box enhancements
Increasing numbers of engineers are using a Salesforce innovation called “Puppet-in-a-Box”, which allows any server role to be simulated in a local container or virtual machine. Usability improvements and support for other dynamic scripting languages were added during the hacking time.
—-
After every hack day, innovations are open sourced, patented, and put on the path to production. Unstructured time for hacking and special projects is a foundational part of our innovative culture and it’s a big part of why this is my #dreamjob. If you join Salesforce Engineering, you can use one of the biggest lab environments in our industry to demonstrate your best ideas at the next hack day. Find your #dreamjob on salesforce.com/tech.