Chris Stolt is one of the original founders of Enterprise Heroku Support and the co-founder of Elm Insights. Today we are sitting down to talk with both him and his fellow co-founder, Phil Ripperger. In our conversation, we talk through the early days of Enterprise Heroku Support including some of the horror stories they’ve been through. We also discuss their current product, Platform Experts, over at Elm Insights.

These two men were a part of the original customer support team for Heroku when it was first bought by Salesforce and have worked together ever since. Tune in for an incredible conversation between us three.

Show Highlights:

  • How Chris and Phil built the customer support department for Heroku during its acquisition.
  • The benefits of engineers going into customer support.
  • The unique technical challenges Heroku presents.
  • How the Enterprise Heroku Support team had to practice transparency.
  • How Shaq boosted their traffic.
  • How they plan for big events like Shaq’s Twitter announcement. 
  • How they supported Trailhead’s success.
  • Why they started Elm Insights and what Platform Experts does for developers.
  • Why Slack was their tool of choice. 

Links:

Episode Transcript

Chris Stolt:
Heroku had hired to do support, move them out of support and into the engineering org where we built that team from the ground up. And so I believe the first person I hired after I joined was Phil.

Josh Birk:
That is Chris Stolt, one of the original founders of enterprise Heroku Support and co-founder of ELM Insights. I’m Josh Birk, your host for the Salesforce Developer Podcast. And here on the podcast, you’ll hear stories and insights from developers for developers. Today, we sit down and talk with Chris and his fellow co-founder, Phil Ripperger about the early days of Heroku Support, some of the worst stories that they’ve gotten into, and we’ll talk about their current product over at ELM Insights, Platform Experts. But we start with some of the early years with Phil describing about how he joined that team that Chris was just talking about.

Phil Ripperger:
Yeah. So this was, what, November-ish 2010.

Chris Stolt:
Yeah.

Phil Ripperger:
Yeah, it was right at the end of the year. And I remember that day coming into work. Actually it was morning I was eating breakfast and the news… I was just checking probably TechCrunch, who knows? But the news about Heroku being acquired by Salesforce hit and then walking to Engine Yard and everyone was kind of just in shock that the main competitor of Engine Yard had just been bought by Salesforce.

Josh Birk:
Oh, wow. Yeah.

Phil Ripperger:
And Chris had left maybe two weeks prior, maybe a month. The timing’s a little bit fuzzy, but it wasn’t long before I remember thinking, “Well, that was a good time on Chris’s part.”

Josh Birk:
Nice, nice.

Phil Ripperger:
And then it was probably two to three months later that I reached out to Chris and moved over to Heroku. I think I started March 15th, 2011. So I think I was employee number 30 or 31, but first official support hire. And then those two engineers that he mentioned, at that point they were able to move over to engineering full-time. So shout out to Terrence and David for doing support and engineering in the very early days.

Josh Birk:
Got it. Got it. So let’s talk a little bit about those early days and Chris, maybe follow up with your instinct to get the engineers back into engineering. And then now it’s just kind of occurring to me that you were basically trained to create a customer support team slash department for a company that’s just been bought by Salesforce. What was your plan at the time? What did you know needed to have happened?

Chris Stolt:
A couple of things. So first that the way you phrase the question that can sound really daunting, right? You’re joining Salesforce. Salesforce is knowing all about their customers. One thing that was made very clear at the onset of that acquisition was that for at least a long period of time, measured in a number of years, Heroku was going to be kind of incubated in its own sort of bubble from Salesforce. So I didn’t have to start integrating immediately, but I did have to try and keep par with their level of customer support and attention to the customer.

Chris Stolt:
So really what I did is, I thought about it in a couple of ways, which is who’s our customer and how do we take care of them? And on Heroku, that customer is a developer, especially in those days, right? So all we had running on Heroku at the time of the acquisition was RailsApps or really RubyApps. And it was developers deploying their own apps. We didn’t really have any big businesses starting to use us yet, full on using us yet. It was in the works and was starting to happen.

Chris Stolt:
So we tailored Support to do developer support. And so what that means is we needed to bring on really ace talent, technical talent onto the team, because what we had is people that are very well-known engineers out in the developer community, opening up Support tickets and asking questions. And we have to have a staff that can hold their own technically with these people that are opening up these cases, right? So we look for people like Phil that had a strong developer background that we could pull in.

Chris Stolt:
The one thing that I had to do too is think through, okay, “Well, those people like to be engineers. Why do they want to do support?” We can map out a career progression into engineering. It’s not a guarantee, but it’s there and it can happen. And it really started with initially moving David and Terrence out and saying, okay, “This can work. We have prior art for it. We know how to make it happen.” And then after that, I looked at, once we stopped up the team of it, how can I get the next person in to engineering purposefully, mindfully.

Josh Birk:
Interesting.

Chris Stolt:
So that way that is almost like a C team for engineering. So people can come in, grow their careers on the direction they want to go in. And it worked for some peoples. Other people, they don’t want to go down that path. And people like Phil kept working in my organization, helping me grow the organization and moved up into leadership. And so there was multiple paths for people to take, but it helped us always be able to attract that talent that we really needed to.

Josh Birk:
Nice. So that’s interesting because now I’ve interviewed a surprising number of people who have gone from customer support and Salesforce to be Salesforce developers. And then they kind of go into your consultancy. I think I’ve got a couple of PMs on there, but it’s always been kind of an accidental route. Making it that purposeful A) sounds kind of unique. And B) what do you think are some of the positives there? Do you think that you’re making better engineers or was it really just so that you could get technical talent with a carrot and a stick?

Chris Stolt:
It’s really both, right? First of all, you want to make something really good for the employees and for the people you’re attracting. And so with that plan in place, we were able to go out to possibly even customers, people that were deploying things under Heroku and fund some of their top engineers and say, “Hey, you’ve been a top engineer, like a lead engineer maybe. Why don’t you come over here and help us scale thousands of apps? You’re going to see a larger breadth of problem than you ever will working on a single lab.” And it’s not easy. You really have to help some of the biggest, some of the top 10 internet sites scale. This is hard stuff.

Chris Stolt:
So there’s that, but then there’s also, “Hey, we’re moving people from support into engineering.” So yes, it makes them better engineers because they’re approaching engineering after that with customer empathy, top of mind, right? They were in those kind of so-called trenches of support and said, “I’m not going to do this thing because I know that’s going to be a bad customer experience. And I don’t want my customers to have that experience because I know what it’s like because I had to hear it from them firsthand for a number of years before I got over here.”

Chris Stolt:
So there was that. And then for me, personally, as I continue to scale up the team, there was like this sort of long game for me, which was really paid off over the years, which is I now have a bunch of ex-employees spread throughout the engineering org. I have these contacts everywhere where I have really good longstanding relationships with them. And so does the support team, so we can have inroads and make headway when support needs to, into parts of engineering org. I think traditionally in a lot of companies, you can find some resistance when support says, “Hey, can you guys go fix X?” Engineering org says, “Well, no, that’s not a priority.” In support relief, it is. And that can be a point of contention. And we really try to break down those barriers by mapping that stuff out and saying, “Well, we’re going to be doing this for a number of years. So let’s do some things that might take years of pay off, but when they pay off, it’ll pay off dividends.”

Josh Birk:
Nice. Nice. And then this is definitely to both of you to kind of dig into that a little bit more. Heroku is a pretty unique offering just in general, in the tech world. How does that then translate into like unique technical challenges?

Phil Ripperger:
Yeah, I can take that one. I think the uniqueness of Heroku is that a lot of the developers that we work with, they would come to us expecting to be able to adjust server settings like they had in previous roles or previous companies. Like we need to be able to configure the patch configuration file for this app or the Nginx comp file, or we need more RAM or swap or CPU or whatever. And Heroku didn’t have… Well, technically you can configure a patch of your Nginx config files. It’s a little bit different. But Heroku’s concept of a dyno and taking care of the routing layer and restarting processes that fail, that was a unique challenge, I would say early on because it was just such an unheard of thing. Even deploying with git, I think, the git push Heroku deployment method that I think Heroku can be credited with inventing, that was a concept early on that we had to explain and work through with customers.

Josh Birk:
Nice. I have to say, I think git push Heroku is something that made me addicted to git. That was the gateway drug for me to just do everything in git so thank you for that.

Chris Stolt:
It was really a game changer. The world hadn’t seen deployments done that way end-to-end.

Phil Ripperger:
I would say to Salesforce credit, and this is worth calling out… We in the early days, probably for, well, I don’t want to put a timeframe on it, but let’s just say multiple years, Salesforce had a very hands-off process with Heroku and basically gave us freedom to build and grow the company. So that’s why Chris was able to build support organization in the way that he just described, because he basically had complete freedom. We had complete freedom on who to hire. We would go look for people who were smart and doing interesting things and maybe they were doing freelance work or maybe they weren’t, but we basically had free reign. The engineers did and the support side did. So Salesforce should be credited with giving us a hands-off approach for the first few years to build things out.

Josh Birk:
Got you.

Phil Ripperger:
And support customers, how we wanted to.

Josh Birk:
Nice.

Chris Stolt:
Yeah. I would even go so far as to say, because of the type of people that we were hiring across the board and within Heroku, you then think about people going to conferences and those giving talks in all sorts of different positions. And you’re on a talk and you’re like, “Whoa, that person just gave that awesome talk, but they work in support? Let me go talk to them.”

Josh Birk:
Nice.

Chris Stolt:
And then you can get some additional interest that way.

Josh Birk:
Yeah. Nice.

Chris Stolt:
One thing I would add on over the uniqueness is also Heroku ran exclusively, still goes on top of AWS was the obvious choice. At first, there wasn’t really any other cloud infrastructure providers that were viable at the time. And then as things like Azure and GCP came on the scene, there was choice in infrastructure provider, but we were choosing to obfuscate that, to obstruct that away from the customer, so they didn’t have to think about what kind of infrastructure their app was running on. It just runs on Heroku. It runs on a dyno, which is just a container. But we were doing that way before containers were a thing, before Docker was a thing. So we called it a dyno. It’s a container, you need to think about how many containers you need and is your app running efficiently?

Chris Stolt:
And so when there were problems with Amazon, we had to think, “Oh, well, we can’t point the finger at Amazon. We cannot blame Amazon for this [crosstalk 00:12:23].” That was our choice, not our customer’s choice. So that failure is really honest. So we want to talk about it transparently so we know what’s going on. And that was always a really, I think, thin rope to walk on to strike that balance between being transparent, but also not coming off like we’re just blaming an underlying infrastructure for provider.

Josh Birk:
Got you. Okay. Well, let’s talk a little bit more specific. Let’s talk a little bit about your greatest hits. For instance, and I’m asking the audience to hear about this in all caps, but tell me about the day Shaq tweeted.

Chris Stolt:
Yeah.

Phil Ripperger:
That was a fun day.

Chris Stolt:
That was a fun day. So a customer comes to us. This is two days before Shaq tweeted. They come to us and they say, “We can’t tell you what’s going on, but a really big celebrity is going to announce something really important on our app.” And their app was… Phil, I don’t know if you remember the name of it, but the app exactly was video for Twitter or Twitter for videos. So you do little short video segments, maybe not so different than TikTok, although this is way before TikTok.

Chris Stolt:
And then so we rush. We hear this stuff all the time, but we don’t think it’s going to be a big deal. But we always took people seriously. So we kind of rushed to help them get some stuff in place. We reminded them a little bit more heads up would have been good. And then all of a sudden they’re like, “Okay, it’s live.” And our servers just lit up and we’re like, “What is going on?” And it turned out that Shaq tweeted his retirement announcement and he has like millions of followers, right? And he didn’t just tweet that he was retiring. He tweeted, “Hey, I have something important to tell you.” And dropped a link to the video. And it was running on Heroku by midday. It was the headline article on ESPN is like, “Shaq’s retiring” with a link out to the video.

Phil Ripperger:
To that video.

Chris Stolt:
Yeah. Everybody was hitting it. And it was a really fun day in Heroku. Because we were all excited about getting all this traffic and we weren’t in the press. We were so excited that one of our customers was getting that much attention and driving that much traffic.

Phil Ripperger:
That was probably the first big event that we can think of to have happened on Heroku that Heroku powered. There were other sites. Urban Dictionary has always been one of the great examples of a huge site that runs on Heroku from the very beginning. And that was there. And your parents may or may not have heard of Urban Dictionary, but everyone knows who Shaq is. And so just to be able to talk about that like, “Oh yeah, we helped power that event and that announcement.” That was a fun.

Chris Stolt:
And the other big differences, a site like Urban Dictionary, they know how much traffic they have. They’ve already optimized certain parts of the app to handle that much traffic. It’s pretty steady, right? The Shaq thing. They literally went from zero to 60, whatever, right? I don’t remember exactly how many.

Phil Ripperger:
What’s the appropriate scale from obscure to epic?

Chris Stolt:
Yeah. But they went from nothing, almost no traffic, to Shaq level traffic in an instant.

Josh Birk:
I like that, Shaq level traffic. So how did it go? Because it sounds like it went well and don’t take this the wrong way, but how is that possible? What goes into planning for something like that when there’s no way that you’re going to know that ESPN is going to throw their logs into this fire?

Chris Stolt:
Well, so the infrastructure on Heroku was different back then. So the secret is that we were running Varnish. And Varnish is a caching layer that was in front of everything. So it cached in the video and was serving the video very fast. Now, the right way to do that would have been, let’s get that video let’s cache in the CDN and have the CDN be delivering it and not be driving all that traffic through something like Heroku where, because to stream the video, you’re not actually processing a web request and that doesn’t have to come through your web app and stuff like that. So there’s actually a better architecture. But at the end of the day, I think it was really Varnish that say saved the day and kept things up and running.

Josh Birk:
Got it.

Phil Ripperger:
It went well given the amount of traffic and the popularity. But overall, it was a success story, a fun story to tell.

Josh Birk:
So talking about preventing the boom a little bit, we have kind of a weird shared history. I wrote this no Duchess prototype that never ever actually had a single user and a bunch of really dedicated people turned that into Trailhead. And I want to be clear here, there’s a rule on project Trailhead. Josh doesn’t support production code. So Josh doesn’t write production code. The closest I got was writing a Ruby CLI to help process the challenge checks. And I think some of that code got morphed into the rails process that that Trailhead actually works on. So if anything went really weird launch, I just want you to know was probably not my fault. But at the end of the day, Trailhead is a Heroku app. So what was it like to support Trailhead’s success at launch?

Phil Ripperger:
Yeah, that’s actually a really good story because it’s an internal project. It’s important. I mean, Trailhead today is a super important product for Salesforce. I think…

Chris Stolt:
Yeah. Salesforce thinks it’s such an important project that they rebranded all of their branding to Trailhead theme branding.

Josh Birk:
Yeah.

Phil Ripperger:
Yeah. That’s a good point. I think Chris, you came to me, I don’t remember when, but this probably would have been shortly after you had written your code, Josh. But Chris came to me and said, “Hey, there’s this internal project. They’re on Heroku. You need to make sure that they’re successful.”

Phil Ripperger:
And so I happened to be actually at the Heroku office in San Francisco that week. I met with some of the project owners and product people for Trailhead. And this would have been early spring, probably around, maybe around this time of the year. And they said we have to be ready by Dreamforce. And so we kind of took that date and worked backwards from there and just took every best practice that we knew about and had had gone through with all of these fun stories that we’ve been sharing and many more. Applied that to Trailhead running on Heroku and made sure that when it launched at Dreamforce, everything was ready to go. And that included load testing and scaling up the database, the number of dynos, every best practice that we could think of.

Phil Ripperger:
And it was ultimately a success and it’s only grown from there. And I think the last time I checked in with the Trailhead team was probably about this time last year. And it was just like… We switched to quarterly check-ins and they told me, “Yeah, we’re fine. I don’t know if we have anything for you this quarter, everything was worthwhile on Heroku.” So.

Josh Birk:
I mean, it went from a very small, extremely scrappy, extremely dedicated team to an entire department. Do you remember, just out of curiosity… Because I remember that Dreamforce so clearly because I was the weird guy running around asking if anybody had seen the Trailhead slide in the Keynote, and if they’d actually tried it yet. And most people had no idea what I was talking about. And then fast forward one year later, I literally can’t walk onto the expo floor without people coming into me and talking about Trailhead. Do you remember what that year one traffic… What their expectations were for that?

Phil Ripperger:
I don’t remember specific numbers. I remember a bump after Dreamforce. It really took off though when it turned into a product that could be sold for customers to use internally. That was kind of the inflection point.

Josh Birk:
Got you.

Phil Ripperger:
And when that happened, everything was ready. So it wasn’t like that caught anybody off guard, at least as far as the Heroku infrastructure was concerned.

Josh Birk:
Got it.

Chris Stolt:
Yeah. If I recall correctly though, there was added traffic and almost like a step function, like pop up, run for a while, pop up, run for a while, pop up. And I think it correlates really well to… You announce something at Dreamforce, people will get interested in it. They don’t necessarily drop what they’re doing, start using it. But then more trails get out there and start circulating. So you get an increase in traffic and almost right after Dreamforce sets off their roadshow, right? They do all the mini Dreamforce events and remote events around the world. And they started kicking that off in that last, almost an another entire year. And each one of those is more people talking about it, more attraction, more events, where it’s getting reinforced to folks. So we started seeing that traffic kind of kick up.

Chris Stolt:
One of the things I think is cool about Trailhead is, unlike some of the other things, when you see, like a Superbowl commercial, a big spike, and then you see like a drop off and a really long tail until they shut it off. And things like trails. Here’s some more traffic. Time goes on, here’s some more traffic.

Josh Birk:
More traffic.

Chris Stolt:
Here’s some more traffic. And you can kind of start to anticipate some things coming up. Like okay, if you keep growing at this rate, you’re going to need a bigger database by this time. Or you’re going to certainly need to look at charting data and all these other things that we can go into and hope at that time helped the Trailhead team really figure out.

Josh Birk:
Yeah, I think I’m going to remember, “We’re going to need a bigger database.” I liked the trails reference there.

Chris Stolt:
One of the unique things with Trailhead having run on Heroku, and this is only unique internally to Salesforce. Peel back the curtain a little bit for the listeners. Support, generally speaking, and CSG, Customer Success Group inside of Salesforce does not support internal projects. I didn’t necessarily like that approach because we’re supporting our biggest customers. Why not support these internal projects? And that view was held with the Heroku leadership at the time. So the Trailhead team had been introduced to my team and we were like, “We’re going to take this.”

Chris Stolt:
Traditionally that would have just kind of gone over to the engineering team who, while more than competent at understanding how to scale some of those things, approaches it a little bit differently. Whereas, our team approaches… We’re going to advise the customer to success. And that’s how our thoughts are ingrained in what we do. The engineering norms may be little bit more focused on “What problem were you having? We know how to solve that problem. Tell me the problem we’ll solve it.” And we’re like, we know how to anticipate those problems. So the uniqueness was that the Trialhead team even got to work with our team because traditionally, if we had followed the status quo in Salesforce, we wouldn’t have worked with an internal team at all.

Phil Ripperger:
Until something went horribly wrong.

Josh Birk:
Until something went horribly wrong. Well, thank you. And I appreciate your style of preparation to avoid the boom, especially when it came to something like Trailhead. Okay. So let’s focus a little bit more on your current gig. Both of you struck out on your own post-Heroku. What’s the current company like, and what was your impetus to do that? Was it just like looking for some brand new challenge kind of thing, or what was the reason for the new company?

Phil Ripperger:
I think we started ELM Insights and more specifically ELM Insights’ kind of the parent company for what we’re doing. But Platform Experts is a Heroku add-on and it’s meant to be a community built around Heroku customers, developers, add-on providers and basically anybody who uses Heroku, who loves it, who’s an expert. And then we also build some tools inside that community that can look at your app and provide a recommendation.

Phil Ripperger:
And Chris and I, I think we saw a need for that, kind of in our last few months, maybe last year or so at Heroku. We wanted to do it. We never really had the resources or the time to build something like that. So it became kind of a natural next step for us. And we built a community in Slack. So you get access to the community through a Heroku add-on. And then we also built a lot of the tooling as a Slack bot. And now that probably looks like a great decision after Salesforce by Slack, but we actually started on all of this before that. And then we woke up to the news one day and we were like, “Okay, we’re building with more Salesforce tools.”

Chris Stolt:
Yeah.

Josh Birk:
Phil, you specifically, I feel like Salesforce keeps doing this to you. We keep buying companies that are important to you. So to frame this, of course, I’m legally bound to talk about Slack as if it is just any other company right now, but that frames this question very nicely, because what was the appeal to Slack? Why did that turn into your tool of choice?

Chris Stolt:
We either looked at [inaudible 00:26:29] and a few others.

Phil Ripperger:
Discord.

Chris Stolt:
Discord, sorry, Discord and a few others. And ultimately it came down to sites where I think a lot of people are already. And if you’re already using Slack, adding one more channel, one more workspace, as the term terminology they use, to get access to some Heroku help and say stuff like that, it’s really easy. It’s right there. You don’t need to fire up yet another app on your computer while you’re trying to work and get other stuff done. So it just kind of made sense. And then the bot integration inside is really good.

Phil Ripperger:
Yeah. The API is really, it’s easy to develop against the libraries are already there for interfacing. I mean, we lived in Slack while at Heroku. That was our day to day. Basically everything that we were doing in the support group happened in there. And we could talk to engineers in there. So I think it was just a natural progression to…

Phil Ripperger:
Oh, we actually, we also had, this is probably key because this was the genesis of the idea. We brought a couple select customers into the Heroku Slack using the official invite mechanism. And at the time it was a decision that I made. Chris gave me the approval. That the team that I have built up and was leading at the time was the data solution architects. So we were focused on just data, either Postgres, Redis or Kafka.

Phil Ripperger:
And some of the customers we worked with had a very large complex databases. And so rather than going back and forth and back and forth and tick it, we would say, “I thought it would be easier to bring them into Slack and discuss issues as needed there.” And that turned out to be a great decision looking back on that, because we had a customer in Slack for over a year. As far as I know, they’re still there. And it was a great relationship because they would ping us if they ever had a question, but it would go quiet for days or weeks at a time if we had a new database or a new feature that we thought they might be interested in and we could coordinate with them. And it was a really good example of what was possible as far as supporting customers via Slack. And so when Chris and I left, we decided to take that idea and turn it into the Platform Experts community.

Josh Birk:
Nice. Nice. Okay. So the add-on is called Platform Experts. And people can find it on the Heroku add-on library, right?

Phil Ripperger:
Correct.

Chris Stolt:
Correct.

Josh Birk:
And that’s our show. Now we will have links in the show notes to the Platform Experts product that they were just describing. Now, before we go, did ask after Chris and Phil’s favorite non-technical hobby and it turns out they really both liked the outdoors.

Chris Stolt:
My mine’s easy. It’s skiing.

Josh Birk:
Nice. Colorado. Makes sense. Yeah.

Phil Ripperger:
Yeah. I would say skiing or hiking depending the season. So that’s kind of easy to answer.

Josh Birk:
As somebody who has to survive the seasons of the Midwest, I completely understand. I want to thank Chris and Phil for the great information and conversation. And as always, I want to thank you for listening. Now, if you want to learn more about the show, head on over to developer.salesforce.com/podcast, where you get to hear old episodes, see the show notes and links to your favorite podcast service. Thanks again, everybody. And I’ll talk to you next week.

 Get notified of new episodes with the new Salesforce Developers Slack app.