Episode 32: Open Source Release Management with Jason Lantz | Salesforce Developers Podcast

Jason Lantz is the Senior Director of Release Engineering for the nonprofit arm at Salesforce.org. Today, Jason and I talk about managing and releasing open source projects for the nonprofit community. We also discuss the Nonprofit Success Pack (or NPSP), open source tools for automation, and hosting open source community sprints.

Show Highlights:

How NPSP transitioned from a repository of metadata to a managed package.
Some early efforts creating open source tools for automation and building them on the platform.
The challenges in finding a cloud storage solution for open source release management.
The building and developing side of open source projects.
Open Source Sprints, what it has been like taking sprints virtual and one side project that came out amazing
An overview of the Open Source Commons

Links

Episode Transcript

00:07

Jason: Coming to salesforce.org provided me with the opportunity to apply my technical skills to benefit 10s of thousands of nonprofits. And you know, at the time we were just doing nonprofit, not education. Now, it’s education as well. And that was six and a half years ago.

00:22

Josh: That is Jason Lantz, Senior Director of release management for the nonprofit arm of Salesforce, salesforce.org. I’m Josh Birk, a Developer evangelist for Salesforce out here on the Salesforce developer podcast, you’ll hear stories and insights from developers for developers. Today, we sit down with Jason to talk about managing and releasing open source projects for the nonprofit community. And we start with the big one, the nonprofit success pack or NPSP, and how it transitioned from a repo of metadata to a managed package.

00:50

Jason: Yeah, so I wasn’t here during that transition. But, you know, I think I’ve got a pretty good understanding of that at this point. Yeah. And, you know, I guess one thing that And I’ll point out because I think this kind of shocks most people, because I think most people in the Salesforce world when you talk about unmanaged pack, or when you talk about open source, they’re like, oh, you’re talking about an unmanaged package. And our experience has actually been, to some degree unmanaged packages are almost the worst way to distribute open source. And the reason for that is because Salesforce makes it really easy to do customizations inside of your own org. And that’s what I mean, that’s a huge part of the Salesforce ecosystem is really good at doing that. But if you’re building an open source project, part of the reason you’re making it open source is so you can get contributions back, you know, especially kind of many years ago, it was not very easy to get that stuff out of your Salesforce org and contribute it back. So what happened was this complete fragmentation where everybody started from the nonprofit template, but then customized it heavily in their own instance, and, and the contributions back to that core were much less once it’s a managed package and it’s run through it. An open source GitHub repository, that’s public. Now, if you want to change the Manage package code, for some reason, you go contribute back to the GitHub repository, and then that comes out in the next release. Now, that does require you as the publisher of a managed package, to really think through what you’re putting in the package, and what assumptions and decisions that’s baking into the package for the your users. So you have to start thinking about your package more as a platform, that people are going to want to do different things, you don’t bake in implementation decisions, you build a framework for those implementation decisions to be able to be made. So it requires thinking about it in a different way. But you know, that’s how we’ve been able to scale NPSP. So when we started our product delivery team, like about six and a half years ago, our main goal was to get NPSP on to a release train philosophy where we’re cutting new releases every two weeks and push upgrading them out to all users. So Wow. And we started Doing that about five years ago, and I’ve been doing it since. So we’re push upgrading to, you know, 40,000 plus scrappy sandbox orgs. And then six days later 40,000, press production orgs. And that’s actually across our entire product line. And you know, the nice thing about that is, yes, it’s a little bit scary, you’re getting updates every two weeks, right? And, but on the flip side of that, every update you’re getting is smaller. And right. So if something goes wrong, it’s a lot easier to debug, the changes that were made are much fresher, and our developers behind so if something does come up, you know, we’re able to kind of quickly turnaround to get a new release out. And we’ve had actually a really amazing track record over the years of not pushing errors in production releases using this whole approach. But the I guess the key thing to point out is, the only way that that’s been possible is through an investment in automation, which was there from the very beginning. So I think that’s, that’s really essential to being able to do it.

03:59

Josh: Right, because early on, there were some efforts to open source tools for automation and building on the platform as well. What were those kind of efforts like?

04:09

Jason: Yeah, so originally when I started, my role has been, I think my original role was lead developer, lead release operations engineer. It’s really confusing title. But essentially the whole time. Kevin Romer, who hired me here, recognized that if we were going to scale this team in the way that he had the vision to scale it, we had to invest in that kind of automation to be able to run automated builds, do continuous integration, use feature branches for development in the repository, and all of that sort of stuff. So early on, you know, the time that I was hired, everything was based on the force.com migration tool was all the kind of documentation out there and running Jenkins. So that’s the way that I started I started writing a whole bunch of stuff in ant took this rather silly line in the sand of I should not write Java code, I should only write all the logic in in XML, horrible choice. You know, we ran with that for about two years. And but we managed to accomplish a pretty amazing process. That’s that’s really I mean, the process that we architected out in the first six months, yeah, is largely the same process that we’re using today. We’re just we’ve evolved the tooling quite a bit. But you know, because this was built for an open source project, it was all out in a public GitHub repository, and the code name of NPSP version three, this this attempt to build out a development team get onto a two week release cadence and agile development practices and stuff like that. The codename of that project internally was cumulus. And in fact, up until just a few months ago, the GitHub repository was still called cumulus. The fun joke internally is cumulus is the cloud that makes it rain resistant and that was my understanding of the reason why that was fixed. So the tooling that we had all of our build scripts to create continuous integration, sort of naturally became called cumulus ci. And initially, it was at base scripts. And, you know, I had other ISVs that heard about what we were doing, and we would reach out and, you know, pay tonight, you know, use your your scripts, and, you know, I was like, sure they’re there in our GitHub repository, right? Go copy the bill, dot XML, copy this lib directory copy, you know, these other things, that was a version properties file that you put some configuration in, and you’re off and running. And they did. And then they would come back to me like, you know, three months later, six months later, and experiencing some issue in the build. And it was an issue that we had already fixed in our build scripts. But again, we ran into this problem, just like with the nonprofit template, when everybody had to forge their own implementation and customize it, where we had fixed that problem in our build scripts, but they started from a copy seven, seven months or six months ago, of our build scripts that didn’t have that fixed. Gotcha. So that sort of led to we tried an app to build a kind of standard packaged approach that people could extend. And then they could update the standard packaged approach. But really, the the current incarnation of cumulus ci started about three and a half or four years ago, which was converting all of that code into Python, which definitely made me very happy, it was so much easier. And the fun inspiration for that was we had some logic that would crawl through metadata files and generate a package XML file.

Josh: Gotcha.

Jason: And that logic because of because I had drawn this line in the sand of like, we’re only going to do it in a had to be this kind of nasty combination of XSLT 1.0 to parse through all the files and generate some files on the file system. So it took just ran NPSP, it took about 20 seconds to generate a package XML by crawling through the metadata, right. And in our builds, it was it was around the That aura came out, we started adding a bunch of extra files in and we started hitting Java heap bears. And I didn’t know how to get around those. I tried all sorts of tweaking of it. But like that was a little bit out of my realm. And I just hired the first member onto my team. And he actually come out in the staying with me for the first week. And we were like, you know what, let’s see if we can rewrite this in Python. And it took a day to rewrite that logic and Python. And in a very configurable way, that’s still kind of in cumulus ci, now, yeah, and that XML generation logic was probably half of the heavy lifting of what our scripts did. So it was like at that point, Well, okay, cool. Let’s just search for a rewrite all of this stuff. We added all the logic from our add scripts largely recreated in Python in about a month. Wow. So we started down that path. And then kind of the next evolution in that was well, where do we run this stuff? And you know, we we have Jamie started with Jenkins, we’d switched over to Atlassian bamboo cloud. And because I really didn’t want to be a sysadmin, I wanted, you know, a cloud system where the stuff runs because I’ve built scripts not run a system, right? And then, you know, we had been running in bamboo cloud and right around that time Atlassian announced the end of life of bamboo cloud and, and that was the offer that you know, will give you server licenses to run it on your own system. But I was like, I don’t want to be assessed. So that’s what I do. And I went to, you know, one of the other things in addition to just the awesome environment of.org that has kept me here at my job was my boss. For many years, Kevin Bremmer was very open to crazy ideas that are well thought out. I have a lot of crazy ideas. Some small fragment of them are actually good ones. But I really thought through and I have a bit of a reputation of like, just wanting to build things, right. So I we had been kind of Thinking on this idea for about three, three or so months after we had all of our build logic running in this Python framework in the what’s now cumulus ci, and of like, why don’t we just build our own ci system? That’s just a Python web app that sits on top of all of this and runs it and runs out on Heroku. And, you know, I, you know, for the longest time, I was like, No, I can’t take that. I’ll get shut down. There’s no way that that’s gonna work.

10:29

Josh: yeah, I okay, I have to ask, how well documented does it idea that crazy have to be to get sold?

10:35

Jason: I still have the two page Google document to bring the Kevin and it was, you know, I basically went through and laid out like, here’s the different cloud ci systems that are available. And, and there was one challenge in particular, that was really hard to find a cloud ci provider that could handle which was this challenge of concurrency blocking. So in, in Jenkins, there’s a thing it’s Build concurrency plugin, where you can say like these two jobs can’t run at the same time. And only one instance of this job can run at a time job. And if a bunch of jobs or if a bunch of builds queued up, those other jobs would just queue and wait until the you know, the running job finished. And then they would run gonna kick off. Yeah. And and that basic functionality actually doesn’t exist in most cloud hosting providers. Because most cloud hosting systems are designed with this idea that your code is built inside a virtual machine. And that’s not the case in the Salesforce world, right. And so it seems like a really simple difference. But it threw this huge wrench into our ability to find a cloud hosting provider. And the specific challenges. You know, this was actually we were starting scratch orgs were starting to come out we had the early dev preview access to scratch org. So we were starting to build on scratch org so we could run our feature branch builds in parallel, now full concurrency, but we couldn’t do that with a packaging or builds. So when we wanted to go upload a package, there’s a single packaging org, and you don’t want two builds running against it at the same time. Really bad things could happen.

Josh: Right?

Jason: So that was a big part of the pitch. I actually, I think it was. It’s still in cumulus ci about 40 lines of Python code that handles all of that logic. That was the blocker for me to find any cloud hosting system. So my pitch to Kevin was, give me two weeks to prototype this. Yeah. And this was around around Thanksgiving at the end of 2016. And he’s like, okay, I’ll bet two weeks. Let’s see. And I had at the end of that two weeks prototype of cumulus ci running on Heroku. With worker dynos spinning up scratch orgs. And calling cumulus ci flows, which are kind of our named orchestrations and cumulus. I’d run a build. And, you know, didn’t have much of a UI around it at that point, but it was working and I was able to show that Kevin Kevin’s like, yeah, roll forward with it and nice. So we went into practice. With what’s now meta ci, and we went into production with that in January of 2017, and had been running all of our projects through it, since I think we’ve run something like 240,000 builds against it, or through that system. Almost all of those with scratch orgs, we burst up to about 100 concurrent builds at a time. You know, each one creating their own scratch, org and running. And you know, it has been awesome. And because it’s running on Heroku, Heroku just bills you down to the second. So we built auto scaling into meta ci now, so it monitors the build queue and then spins up worker dynamos for you, and then runs the builds. And whenever the queue goes back down to zero, it scales everything back down to zero. And so, you know, it’s it’s remarkably efficient at being able to burst up that much. And, you know, we see those bursts usually towards the end of a sprint when everybody’s wanting to merge their stuff in and there’s a lot more demand on the builds, but we don’t need 100 You know, build dinos sitting around full time wasting resources, like we need to be able to burst to that capacity, but that’s not our normal load.

14:07

Josh: Well, let’s, let’s talk a little bit about the building and developing side of it as well.

14:15

Jason Yeah, these are one of my favorite events and things. So I mentioned you know, the Sprint’s in the salesforce.org, where I have to catch myself all the time and in trying to say like, Oh, we built this amazing community because really the community built itself we’ve just been here to, to kind of facilitate it and help it along. But starting, I guess about three or four years ago, we hired Judy Stone, who is amazing. She’s currently the Director of our customer centric engineering team. And at the time that we hired her she was we have the the power of us hub, which is our Salesforce community with, you know, hundreds of thousands of nonprofit and education users in it and partners and staff from salesforce.org all like self supporting each other. It’s an amazing community. Judy was like the most prolific poster.

15:04

When when we hired her eyes, you know, like, I’d wake up in the morning and like all the questions Judy had already answered. And, you know, she she’s east coast. So she had a, you know, a little bit of an advantage on me. But it was it was, it was remarkable. Yeah. And so we hired her as our kind of open source Program Manager. And she helped, you know, sort of grow the sprint program, and really it formalized A few years ago, we started running these open source sprints a couple of times a year, and the sprints generally like it for the last two years. We’ve run three sprints a year in the US and then one in EMEA. And I haven’t attended any of the meal ones I would have loved to, I think in Amsterdam and in Paris, both places that I’ve never been before so but in the US we have these three sprints a year, their two day sprints, usually somewhere between 100 to 200 attendees and a good mix between our nonprofit and education customers, our partners in the.org ecosystem and staff from salesforce.org. So we get a lot of our, you know, developers that are working on the products, they’re attending the sprints. And it’s a very unconference type event starts out with just kind of brainstorming of ideas of what could you know, what could be built that would help nonprofits or what are the skills even in a non technical sense, like, there’s the last couple of Sprint’s there’s been a group focused on admin soft skills, just creating documentation on soft skills for admins. But there’s also teams that are, you know, they want to build something really cool and I think it would be impactful and want to build it as something open source. So they get together at Roundtable, spend two days hacking away at something, and there’s a demo session at the end. And I mean, everybody that I talked to, that comes to one of these Sprint’s like by the end of it, it’s just, you know, this, this amaze, it’s been this amazing experience for them, and we have a lot of regulars that keep Coming back to sprint after sprint. One of the challenges though is, we started to realize about two years ago that what was happening when people were building stuff at the sprint is oftentimes somebody would spin up like a scratch, or I’m sorry, they spin up a sandbox, or they’d spin up a D. org. And everybody would kind of share the login and go in and build something in in in that org over the two days. But then that thing that they built would kind of live and die in that org. Right? And, you know, the idea of like, how do I get that, in my org? How do I sustain this as a project was was a real challenge. And, you know, we started to realize, like, hey, we’ve sort of figured this out for our internal projects, figured out how to run an open source project through GitHub and built all this tooling to do it. Yeah. What if we made that tooling available to our community? I mean, the tooling was already open source, but like, what if? How do we enable our community to be able to use this tooling so that these projects can kind of be run through version control can be sustainable? And last year, we started Have a half day training session before each one of the sprints where I get to do a hands on training session with like up to 50 people, mostly Salesforce admin level users, then, you know, a fair number of them had not really worked in command line before, you know, bit anxious about working in command line. had never used Git or GitHub and maybe had gone through the Salesforce dx trails, but most likely not. So never spun up a scratch org, right like that. And, you know, the way that we looked at it is every single member of our community that we train up to use that process has an exponential multiplier effect for every sprint going forward, right? You know, they could go around and sit at one of those roundtables and help make sure that the things these awesome things that are being built by that team are something they can carry on to the next sprint and you can work on in between sprints and, and so you know, we did that first training class was a little bit rough. I didn’t really have the material together all that well, but the last two actually went Great. In Detroit, we had I think about 30 people. They got all the way through the process of, you know, creating a new GitHub repository using cumulus ci to set up the repository and spin up a scratch or go make changes in the scratch org and the declarative user interface, using OCI to retrieve those back into the repository, commit them to a feature branch, submit a pull request review of pull request, merge it, you know, and I mean, to me, it was just so cool to see this roomful of people that had never been exposed to any of that after three hours, right, be comfortable with that process. And then in Philadelphia, which was our last sprint last year, there was 50 people that went through that training and the curriculum for that training actually is the pretty much the first three modules of the cumulus ci trailhead trail that we launched in March and then the additional three modules beyond that. Kind of extend upon that idea. So it’s the trail is pulled from the story of a food bank that has has built some customizations on top of NPSP to run a food bank, and they want to share that with another Food Bank that’s not on Salesforce. Yep. All right. So how do you how do you go about doing that as a as a Salesforce admin, because, you know, most most admins are trained in change sets. And deployments between these connected works like a sandbox and production or sandbox and other sandbox works. But how do you share something that you built outside of that structure with like a totally different org? Right. And so that’s the the story that we tried to walk through in the trail. And we really tried to target that trail towards people that are admin users that maybe don’t have a familiarity with the command line. We try to keep the the commands that you run as simple as possible and approachable. But I think, you know, hopefully, it’s, it’s just another way of getting more people enabled to be able to use this tooling, participate in these open source projects.

21:01

Josh: Right, right. During the sprints themselves, have there been any kind of interesting side projects that have come up? Yeah, there’s a ton.

21:10

Jason: One, the one that amazed me the most at the Philadelphia sprint is Lars Peiffer from Attain One of our partners who is a sprint regular he’s been in like every sprint for I think the last two years. I love going to the in person Sprint’s and seeing them every day. And Chris had this crazy idea at the last one, which I guess that’s the idea. And honestly, myself and one of my team members who was there, heard this idea announced at the beginning and we’re like, there’s no way we’re gonna stay away from that project. There’s no way this is gonna happen. But the idea was to use flows to build Apex tests. Oh, you know, and you know, the idea was that admins should be able to write tests in Salesforce. And by the end of the sprint, Chris was demoing a working test written in flows that goes and creates objects, checks the values on the objects after a trigger runs in the background and stuff like that. And it’s still up there in a public GitHub repository. I just tweeted about it. Last week, somebody was asking about, you know, how it’d be really cool if admins could write tests and flows and it was like, hey, and, you know, it still has a lot of work to do. It’s very much just a prototype of like, you know, is this possible but yeah, I mean, the the ingenuity of this community that we have. I, you know, I think part of what amazes me about the nonprofit tech space in particular, and also the education tech space, is just, I think that you get a different type of innovation. When you have people that are motivated for different reasons and You know, when it’s, it’s really like a passion project, like, it’s something that like you recognize this thing that I’m doing is gonna make the world a better place. You know, it’s not just about like, you know, generating more sales or making somebody more money which you know, has has huge value. I don’t mean to downplay that, right. The types of innovation the, that you get when people are just out, you know, when it’s personal to them, you know, it’s, it’s like the statement about like a, you know, an army that’s defending their own territory. But it’s better than an army that’s that’s trying to invade somebody else’s territory. You know, I think there’s a lot of that aspect in our community, which is just really cool to see.

23:36

Josh: Nice. And what is it been like these we’re recording this an interesting time. So what’s it been like taking an open source spirit to bring virtual?

23:45

Jason: Yeah, we, our first sprint this year was supposed to be in Atlanta at the beginning of April. And unfortunately, that had to be canceled. It was devastating to us to have to cancel the in person event because You know, we have members of our community that are just personally paying to conference, right? Yeah. And so that was a really hard decision. But it was absolutely the right decision for, you know, the health of our community and such. And we decided that we were going to do an open source sprint or a virtual sprint instead. And I remember having this conversation with Judy stone, where she told me we had 100 people register 105 people register. She’s like, generally turnout on open or on, you know, virtual events is about 30%. Right. So, I’m anticipating we’ll have about 30 people show up. Yeah. So the first day of the sprint, I load up the zoom call, and there’s 100 people. And it was just awesome. We used a quip doc to kind of set up breakout rooms. And this was all like, designed in two weeks time or something like that, this whole structure for doing it, but I am I think it speaks to our community that everybody was you know, willing to give it a try willing to innovate and try to figure out how to make this work and it really did it was it was cool there were a number of projects that got worked on and some really great work that was done those individual breakout rooms were kind of like a roundtable at the sprint. part that I’m sad about about not having the in person events is you know, one of my favorite parts of the Sprint’s is you know, our team at Salesforce. org is is all remote we’re almost all remote and so I work from home I don’t have regular interactions with like even my team members in person or members of our community in person so you know, the ability to just like go hang out at the hotel restaurant in tow bar and like have a conversation with somebody after the sprint and get to know them better is awesome and there were definitely some like social calls and stuff like that people stayed on for like a happy hour after at the end of the sprint chat a little bit but but that part I think is irreplaceable. Yeah, but On the flip side of that, we had attendees from all over the world, we had an attendee from India at our sprint that would have never been able to make one of our in person sprints. So I think what we learned from that was, you know, our community is awesome. And they’ll bear with us and figure out how to make it work and the right way to probably do sprints and think about Sprint’s, going forward is a mix of both. Yeah. You know, when we when we get to a point where we can actually meet in person again, right.

26:27

Josh: Okay, so let’s move to a different topic. Talk to me about what the Open Source Commons is and what it does.

26:35

Jason; Yeah, so open source Commons has kind of two things. It is the umbrella term for our community. Gotcha. And for these community sprints, and it is you know, one thing to emphasize is open source Commons is a very individual oriented thing. So our partners who are sending their staff to attend the opensource Commons Sprint’s are sending their staff or facilitating their staff as needed, To participate in the sprint, it’s not them as an employee to participate in the sprint, you know, we really want to have that, you know, these are people that are there out of their personal passion kind of approach to the community. And the other side of open source Commons is sort of came from, you know, going back to the days of like Ant and Jenkins and running all of this stuff on our own right. And having come from a, an open source Python background, I remember many years ago, Travis CI, when when it came into the ecosystem, and started running, you know, it started out as a system to run builds for open source projects. And there were thousands of open source projects that ran on it. And it was awesome because most open source projects can set up their own Jenkins server and run it and have the infrastructure for that. And it provided just this huge value to the open source community, right. And so I always had this idea of like, wow, we’ve we figured out how to do continuous integration for sales. first project, I’d love to be the the Travis CI for our community essentially or something like that. We’re providing that kind of a service to them. But it was always tricky to figure out from a legal and security. You know these other standards. Yes, yeah, yeah, exactly. And, finally, about two and a half years ago, Judy, and I managed to convince Kevin that this was a crazy idea that was worth. He’s like, fine. Yes, figure out figure out a way that we can do this. And so the open source Commons kind of program is, I guess, to some degree like incubator for these open source projects in the nonprofit education ecosystem. So there’s a set of qualifying criteria that really focuses on the sustainability of the project, like these projects are part of the Salesforce org open source Commons. So our name is attached to them, even though they’re community run projects, but our customers are trusting those projects based on their trust in us. So we want to make sure that somebody’s not gonna, you know, it’s not just that you build something really cool, it’s that you’ve got a team of people that are dedicated to maintaining this project over time, you’re not gonna leave our customers stranded. And if you did happen, you know, everybody on the project won the lottery at the same time and they moved to a private island. And other people in the community wanted to come forward and maintain that project, right, you could transfer ownership of the packaging org over to those new maintainer. And so we had to figure out this whole model for sustainability. And, you know, the advantage of why you would want to go through that process is kind of twofold, one for visibility. So we have the SF do community organization on GitHub, where official open source Commons projects get hosted. And then also because we run all of our tooling, so meta ci, I was talking about earlier, also meta deploy, which is our web based installer that’s installed salesforce.org. There’s an open source comments tab on there that you can go out and install Open Source Commons application. Gotcha. So we try to basically if, like the challenge in the nonprofit education ecosystem is not people’s desire to share, nonprofits have an inherent desire to share food banks want to share with other food banks in the county next to them. You know, pet rescue organizations want pet pets to be rescued, no matter which organizations doing it. And so there’s this inherent desire to share, it’s just that people don’t know how to. And that’s the thing that we try to solve with open source Commons and through these communities, Francis really, we will help you share these things, we will take some of the pain of like, you know, creating releases of your managed package and stuff like that, if you’re not familiar with doing that. Now, it’s just a click of a button through meta ci and you get your your release of your package and you know, we’ll manage the playing it out to meta deploy and making it available for people to go out and install for you got it. And you run the project through GitHub, people submit pull requests, you just review the pull request, merge it and You get a beta release automatically. So a lot of the, you know, what we built in order to make, you know, my team’s life easy to support all of our internal development teams. We also want to put that into the hands of these open source projects.

31:13

Josh: That’s our show. Now when I asked about his favorite non technical hobby, Jason admitted to being a former DJ and really into music production. And he brought up an interesting factoid that there is an interesting combination of skills when it comes to being a musician and being a programmer.

31:28

Jason: Because you have to think about things that you abstract like you’re running a composition through your head and then hearing it play. And University of North Texas here in the Dallas area in Denton was one of the one of the top music composition schools in the country. So they were like, hey, go hire music composition majors out of out of u and T. And it was apparently a wildly successful program.

31:50

Josh: My thanks to Jason for the great conversation. Of course, my thanks to you for listening. Now if you want to learn more about this podcast, head on over to developer.salesforce.com/podcast where you could hear old episodes see show notes and links to your favorite podcast services. Thanks again and I’ll talk to you next week.

Get notified of new episodes with the new Salesforce Developers Slack app.