Episode 42: Salesforce DX Architecture Considerations with John Daniel

John Daniel is a Salesforce architect for the B3 group. Even before working with Salesforce, he had encountered programs that had constant issues and technical debt. He found that Salesforce projects brought unique challenges when it comes to designing at the org level and maintaining properly divided applications that respect a separation of concerns. 

 

John was also part of the group of developers who brought about the first-ever Salesforce DX implementations. Today, we’re talking about his experiences using Salesforce DX for technical architecture. 

 

Show Highlights:

  • What happy soup is and why it’s a favorite among the Salesforce community.
  • What vertical slicing the next phase, horizontal slicing, are.
  • John’s styles and suggestions for naming Apex classes, Visualforce pages, and Lightning Web Components.
  • How he handles things that are dependent upon each other.
  • How functional aspects play into the declarative side of the veil.
  • What the open-source group is and what it does.

 

Links

 

Episode Transcript

John Daniel:

Just had a small group of developers and they were keen and eager, and they wanted to bring order to the house, and they were willing to do what it took. And so we just started going after it.

Josh Birk:

That is John Daniel. Salesforce architect for the B3 group. I’m Josh Birk, developer evangelist for Salesforce, and here on the Salesforce Developer podcast, you’ll hear stories and insights from developers, for developers. There, John is talking about one of the first ever Salesforce DX implementation.

Josh Birk:

So today we’re going to talk about his experiences using SFDX for technical architecture. And to start well, we’re going to start with the term, it’s something of a favorite amongst the community, happy soup.

John Daniel:

So happy soup is this term that has surfaced in the community to represent, or mean, the Salesforce org and all of its metadata, but it’s just this big pot of blended, whatever you want to call it, spaghetti’s code. There’s no beginning to end. It’s just a big cauldron, it’s a big soup, and it’s a mess and it’s all together, and it represents everything.

John Daniel:

It symbolizes things like the interdependencies and circular references that are so often found in Salesforce development. Things like when you get like a Visualforce page that has an Apex controller and talks to another Apex class, it runs a query against an object. And that query against that object has an action item override to another lightning page, or Visualforce page or whatever.

John Daniel:

And so, all of a sudden that first visual force page now has a dependency on a totally different object. And it’s not really intended to do it, but it blends it all together. It’s a term… I think Salesforce marketing does not like that term. I think they’ve tried to squash that term actually quite a bit, but I don’t think they were successful.

Josh Birk:

Yeah.

John Daniel:

It represents a disorganized org, or an org that it’s very difficult to understand where things begin and end.

Josh Birk:

I mean, it kind of sounds like the architectural equivalent of spaghetti code.

John Daniel:

Mm-hmm (affirmative).

Josh Birk:

Got you.

John Daniel:

Absolutely. Absolutely. I’m not sure what kind of a soup is spaghetti and soup. [crosstalk 00:02:10].

Josh Birk:

Right. It feels like there’s something there. And is this just really kind of an aspect of overall technical debt? Or is there kind of a deeper cause to creating this soup?

John Daniel:

I think it’s a mixed bag of reasons. I think in my experience, I’ve seen orgs that… It’s been interesting. Because Salesforce, and the Salesforce phenomenon reminds me a lot of, again, that I got started in the world, and in development in mid ’90s. And so Microsoft Office, and specifically Microsoft Access was a big thing back then.

Josh Birk:

Right.

John Daniel:

And Microsoft Access was great in the ’90s because it was very simple and it was Wiziwig, and drag and drop, and a business type person could easily come in and set up an app, and off to the races you were going.

Josh Birk:

Yeah.

John Daniel:

But Access had this terrible reputation. It wouldn’t scale. It was always having issues. You couldn’t have a concurrent record without Access. And turns out, it wasn’t a problem with the Access program. It was a problem with the way the application was designed.

John Daniel:

Because I came across one guy in particular who was truly the embodiment of a Microsoft technical architect. And this guy could run entire enterprises on a single access database and be performed as all get up. Because he knew how to properly craft an application. And so when I came into Salesforce, I saw the same type of behavior. I mean, you’ve got a lot of people out here who have made arguably awesome careers, but they’ve only ever known the world of Salesforce.

John Daniel:

And they probably came up the ranks through starting out as admins, or whatnot. And they somebody asked them to go and write a trigger to do, whatever, HelloWorld function or something like that. And they’ve been organically learning how to pull it together. But, have they been in position where some of I guess the old school development shops where they focused on things like the Gang of Four design patterns, and Martin Fowler’s Patterns of Enterprise Development in architecture. And they just haven’t had the exposure to some of these concepts.

John Daniel:

And so they don’t know any better. I guess the other thing I see a lot of times is these orgs that have lots of life, and you actually get multiple architects in there. And they have multiple architectural visions. And so that’s a TIFF and type of dynamic. I mean, you’ve got guys who come in with very good, real ideas, but they don’t have complete control over the org. So it’s difficult.

John Daniel:

I mean, one of the benefit I had over at the law firm was, whatever the buck stopped with me. But I mean, I didn’t have godlike power, but we had a clear, single vision about how we wanted to pull that together. And we were unified and we can make some things happen.

Josh Birk:

Right. Nice. All right. Well, let’s talk some specifics when it comes to adoption. And you’ve talked about in the past about vertically slicing and horizontally slicing, what does vertically slicing mean?

John Daniel:

Vertically slicing is part of, I guess, a mental exercise about how do you… When you’re trying to take your org, take your happy soup, and you’re trying to put it into multiple packages, the first thing you definitely don’t want to do is just take your entire org and put it into a single unlocked package.

Josh Birk:

Right. Because at that point you’ve pretty much codified the happy soup.

John Daniel:

Exactly. Yeah, exactly. And it’s just sort of like, okay, what was the point actually? How do you take your code base and start splitting it up and organizing it? What patches do you really look at? And so vertical slicing was the term that I gave, basically things like, take a look at your org and see what applications are there.

John Daniel:

So if you’re in the old classic mode, it’s the drop list in the upper right hand corner; all the different applications there. Or if it’s in lightening mode, it’s the tic-tac-toe box on the left. What are the applications? And start thinking about your org as just verticals of each of those different types of applications.

John Daniel:

And then the first exercise out of the gate is always to identify all of the metadata that you have in the org, and then pin it to these different verticals. Start figuring out what is supposed to be going in that, and then identify things that really belong to every vertical. And so you’ll deal with that as a next phase.

Josh Birk:

Got you. So then what is the next phase? Is that the horizontal slicing?

John Daniel:

That’s usually the horizontal slicing. The horizontal slicing is more about layering your application and layering it by purpose.

Josh Birk:

Okay.

John Daniel:

Because one of the hallmarks of a good code base is a really good naming convention.

Josh Birk:

Okay.

John Daniel:

You should be able to look at the name of a class, or a flow, and understand exactly what its purpose is doing. And really you should make sure that it doesn’t have multiple purposes. So that when you get into the design concept of separation of concern, it’s all about each thing has a specific purpose, and it’s about organizing your codes [inaudible 00:07:38] that. All too often, I’ll see a developer come out there with like a lightning component, and an Apex controller to back it up.

Josh Birk:

Yeah.

John Daniel:

Yeah. But he’s doing business logic in the controller, he’s doing queries, and he’s doing DML updates, transaction control. And again, it’s just sort of like get into that work. Think about it more as the controller’s purpose is really to just massage data into a way that either the VF page, or the lightening component can consume it. It’s not about processing the logic. Its not about doing DML. It’s not about even querying that kind of logic.

John Daniel:

And so again, if you’re properly layering your code, then you have far less waste, you have far less duplication. So if you have a process that manages the updation, you might update a critical account with a specific discount or something like that, maybe an opportunity or something like that. And so you probably don’t want to put that into like the Visualforce controller of that manages the opportunity. And the controller that’s backing up the lightening UI, and then the invocable method class that you’re running from the flow. And then you’ve got the rest API.

John Daniel:

You don’t want to do this discounting every single time. Each one of those guys are basically clienteer type aspects, and they should all be calling some level of a service tier to manage the actual applying of the discount integrate he proper way. Therefore, you can keep that logic in a single way.

Josh Birk:

Yeah.

John Daniel:

So, horizontally slicing is about understanding what your layers are, what are the client layers? What are the service layers? If you follow the enterprise patterns that Andy Fawcett was promoting in his book, it’s all about the domain logic versus service level logic, and just organizing it that way.

Josh Birk:

Yeah. It’s interesting the naming convention, because I feel like there’s an additional challenge to that when it comes to Salesforce development. Because if you go back to like Java development, there’s a certain level of structure that’s enforced on your code because you’re putting things into packages, you’re putting things into folders.

Josh Birk:

And so you kind of have like, if they’re all in the same family, then they’re all in the same folder and you don’t have to worry that much about the naming convention, right? So what are some of the styles and suggestions you have for like naming Apex classes, and Visualforce pages, and lightning web components to help keep that straight?

John Daniel:

Yeah. No, I hear you. I mean, because like I said, in Java, there we had the added benefit of being able to move the code into sub-directories [inaudible 00:10:14]. In the classic metadata project, it’s the SRC folder, and the classes folder.

Josh Birk:

You’re right.

John Daniel:

And in that folder you have 3000 Apex classes.

Josh Birk:

Exactly.

John Daniel:

And it’s difficult to understand and organize that. But one of the things that we adopted was what we called a… For a while we called it a sub namespace and we call it a class prefix. Basically, it’s very easy to look at a code base and identify the code that is owned by whatever third party managed package is out there. Because a third party managed package has a namespace on every bit of metadata that’s out there. So it’s easy to sort it by the namespace.

John Daniel:

And we were looking for some sort of mechanism to do the same thing. Because there again, that first year at the law firm, we didn’t have DX. We didn’t have DX source format, which allows us to move things into sub-directories as well. So what we came up with is with this concept of a prefix, that we would just append on to every class and really every SObject and every flow and everything.

John Daniel:

And it was again, identifying what our packages, and our modules of the org look like; or the applications. And so even though we didn’t have DX and packaging out the gate, we knew it was coming. So we started organizing things, and we sort of logically figured out what kind of a package model, or a package hierarchy that we wanted to do.

John Daniel:

And so what that did was allow us to organize things like there was common library utilities, there was common service layer type things that [inaudible 00:11:57]. And then like at the law firm, there was one process that was all about intake. That was the processing. It was everything from the point where we get the phone call saying, “Hey, I want to sure my brother-in-law or whatever.”

John Daniel:

And all the way to the point where we said, “Yes, we do want to take you on as a client.” And so there’s all that processing there. And that was the intake process. And so pretty much everything related to that code, or that section of code was all prefixed with the word intake.

Josh Birk:

Got you.

John Daniel:

And then when it became a lawsuit or it was also called a matter, we would prefix that logic there. And so it was very simple to organize and say, “Yeah, this piece of logic goes to this general IDM package.” And it was important because we knew when we were going to start separating things out into various packages, we wanted to isolate say the library code first. Because we knew we were going to codify the library package, the very first thing. And so it was easy to identify and pull that out and start mingling, and we didn’t have all the extra noise of the rest of the org.

Josh Birk:

Right. And then I guess going back to talking about like circular references, and that if the whole goal is to create boundaries that can then line up with second gen packaging, like how do you handle things that are dependent on each other and are creating these circular references?

John Daniel:

It depends on the metadata type in question. So there are certain things that they’re very easy to separate. For example, you might have a custom SObject.

Josh Birk:

Right.

John Daniel:

And the custom SObject or standard SObject like account, you might have a custom SObject, and call it FUBAR and it might have a couple of different fields. Well, technically the way Salesforce is set up, it’ll allow you to deploy the fields as a separate deployment of that from the main SObject. So what that afforded us to do, what our ability to do was, we might have one SObject and it’s part of a package, and we knew it was going to be deployed. And we knew that this package would always be deployed first; and so that would set up the SObject. And we knew that other packages could add fields to the SObject.

John Daniel:

We did it a lot with the account and the case objects as well. It’s just we didn’t have a package that pre-installed the account, because it was already in the org. But we would add new fields to that org, depending on which package was installed. And so that kind of logic helps us to when that second package is there, then we know we can rely on it, but we need to make sure that all our code that relies on it, exists in the package that has the field.

John Daniel:

And we needed to make sure that… We did some testing actually where we deployed say the first package, which didn’t have this extra field. And that ensured and validated that we didn’t have logic that was depending on a field inadvertently that it shouldn’t be.

Josh Birk:

Right. Got you.

John Daniel:

And then you’ve got other things like with basically what we found to be the silver bullet was the concept of dependency injection.

Josh Birk:

Okay.

John Daniel:

And so what we did was we, again, piggybacking off of the proper layering that the Apex common enterprise framework gave you, it had an example of dependency injection already there. And that was sort of the inspiration for it. The inspiration there was the Apex common framework has a hook that allows you to swap out maybe a selector class, which is responsible for querying, or swap out a service class, or a domain class. And you could swap them out and replace them with a mock version of it. So that’s where the utilization of the ApexMocks framework, which is the other, there’s a sister framework to it.

John Daniel:

And so the pattern was sort of there already. And it sort of inspired us to really think about how we’re going to handle dependency injection. And that sort of led it to, we realized that the original version of the Apex common framework had a static mapping, if you will, or service locator pattern, basically. And that static mapping sort of worked in a non DX world, but we had problems with it as soon as we started breaking out the packages.

John Daniel:

Because the premise was that everything in the code base depended on the application factory for getting all of its instances of everything that you needed. But the application factory was statically mapped in the old world. And so that was our first problem that we had to solve. And so we needed a way to create the mappings, but in a flexible way so that if I don’t have a package installed there, well, the mapping doesn’t exist. And so we needed the packages to just carry the mappings with them.

John Daniel:

And so what we ended up with was custom metadata types, and custom metadata data type of records. So just like I was saying that with an SObject, you can have an SObject that comes out in one package and add more fields in another package. We did basically the same thing. We defined a core custom metadata type record. And we put that in the [inaudible 00:17:16] foundation package or the lid package that we have.

John Daniel:

And we adjusted our own code logic so that it, instead of trying to rely on static mappings, what it would do is dynamically query this custom metadata record, or this custom metadata type. And then whatever records it finds there, it would filter them down based on the use case. And then invariably, there will be a class name that was part of that record field.

John Daniel:

And so what we would do is then do the command to instantiate a class on the fly. And we would instantiate that class on the fly and basically utilize it there. And so what the beautiful thing was the custom metadata records didn’t have to be there with the base package. But say we have another package that we install, it had the custom metadata records. It didn’t have the custom metadata type defined, cause that was already part of those-

Josh Birk:

Because it was already done.

John Daniel:

It was already done. It was already part of the org. And the unlocked package said, “Well, you have a dependency on say the lid package,” is the paradigm that we had at the law firm. And so we were guaranteed that that base custom metadata record would be there. And so when we install our package, all of a sudden these records would show up, and the next iteration, all of a sudden these records would be available and you could spin up the logic accordingly.

John Daniel:

And so that dependency injection was what we started with. It was a crude form of it initially at the law firm. And when [inaudible 00:18:42] and I gave a talk at Trailhead DX 2018, about what the journey was, it piqued the interest of Andy Fawcett. And specifically the dependency injection part of it. And so he started talking to myself and another guy named Doug Ayers. And that led to, he was like, “We really need to get this dependency injection concept out there. And in a way that’s consumable to everybody.”

John Daniel:

And so that led to the creation of the Force DI, or to Force Dependency Injection Framework.

Josh Birk:

Got it.

John Daniel:

And so it was based off of, he had seen another framework out in the Java world. It was the Google Juice Framework. And basically that does dependency injection in the world of Java. So he’d modeled it after this. And then Doug Ayers brought some absolutely just amazing adjustments to basically achieve dependency injection in flows.

Josh Birk:

Oh, okay. So that was actually going to be my next question, because we’ve talked a little bit about data. We’ve talked about functional things like Apex classes and the client side. So then how does this kind of play into it with the declarative side of development?

John Daniel:

Yeah. So again, it was an amazing piece of work. It opened my eyes to the power of a dependency injection. I’m admittedly, I’m a coder at heart. I’m the guy who goes to the programmatic side all the time. If you give me a beer, or give me something to drink, I’ll get into a lovely philosophic argument about declarative versus programmatic.

John Daniel:

That aside, I was stunned when Doug air’s finished with his work on the Force DI project, because he opened my eyes to what could really be done in the flow area. And it was crazy there. And basically it gives the ability to, again, insert, and change and add things dynamically, which means that they’re loosely coupled. And then that means that you can put some of the components in one package, and other components in another package, and add functionality just by the presence of which packages are actually installed at any given time.

Josh Birk:

Okay.

John Daniel:

That was the way we set it up. And then basically once that was done, we basically went back to the framework that we originally created at the law firm and adjusted it, so it was using Force DI. And then that led to what is now called the AT4DX repo, or framework. And so it depends on the ApexMocks and Apex Common, but also on the Force DI, and it’s a very, very, very nice recipe if you will. Architectural recipe for setting up very involved dependency hierarchies, and it’ll work.

Josh Birk:

Got it. Okay. So we’ve mentioned it many, many times, and I really want to give like a real proper shot out to it. So, tell me about the Apex Enterprise patterns, the Open Source Group, like what is it and what does it do?

John Daniel:

We had all these different frameworks and Andy Fawcett used to work at FinancialForce, and then he left there and went over to Salesforce. And then I was there at FinancialForce, and FinancialForce has been doing a lot of work with various technologies. They set up the Open Source Repo, but they had a little bit of staff turnover and a few other things. And so the repos, the ApexMocks and the Apex Common repos, it was a while to get some response as far as changes on those repos. And everybody’s got to put food on the table and earn a paycheck. So, asking people to do this in their off time, it’s a little crazy.

John Daniel:

So last fall, so this was 2019, Andy Fawcett, myself, Steven Wilcox from FinancialForce and a few others at FinancialForce, as well as David Esposito from Patron Technology.

Josh Birk:

Got you.

John Daniel:

We got together and we started talking about the idea of the two repos, ApexMocks, and Apex Common being sort of put under community control, if you will.

Josh Birk:

Got it.

John Daniel:

And so that was the Genesis of the Apex Enterprise Pattern Group. And then once that was set up, the Force DI project was added to it. And then later on the AT4DX framework was added to it. And the purpose is basically, just a bunch of us developers who don’t get to develop enough, because I’m a Salesforce architect now, so I don’t get to develop all the time.

John Daniel:

But we get to do it and maintain these repos. And we’ve been able to start to modernize them and take advantage of some of the newer things that are available. We’ve got plans for things like incorporating change data capture, and we’ve already got some platform events, and platform cache built into the things.

Josh Birk:

Nice.

John Daniel:

We’ve converted all the projects to a Salesforce DX source format.

Josh Birk:

Nice.

John Daniel:

And so, yeah, it’s been a good move. Definitely a good move.

Josh Birk:

If I go to the repo, what do I see? Is there like a roadmap or anything? Do you have like us to kind of… Like if I wanted to raise my hand, and see where this is going, is that all up on GitHub?

John Daniel:

Not yet.

Josh Birk:

Okay.

John Daniel:

Funny you should ask. I was talking to some of the guys about an hour ago, in fact, about that very thing. Part of our focus right now is there’s been a lot of articles written by lots of different architects, and technical leads about how to use various facets of the frameworks, the ApexMocks, the Apex Common, Force DI. And so what we’re doing right now is starting to get all of those organize. We’re going to be contacting some of these authors and seeing if they’re okay with us actually transferring those blog posts over to those repos. So they’re all being a one stop shop.

Josh Birk:

Got it.

John Daniel:

We are going to set up a more holistic Wiki. Again, the whole idea of, what is the Apex Enterprise patterns about? And this is where you start. Again, we’ve got to put food on the table first. But the goal is by the year, we’ll get that in there. And there’s some other things like the next project I do there is to set up the CI, because we get a lot of pull requests coming through. And it’s important to understand, not only did the pull request work for this particular project, but did the pull request really work for all the downstream projects?

Josh Birk:

Right.

John Daniel:

Because, when you think about it, if there’s a pull request to ApexMocks, which is technically the bottom of the stack, well then does it impact Apex Common? Does it impact the Apex Common sample code repo? Does it impact the AT4DX? Does it impact the AT4DX sample code repo? And so it’ll be good. We’ve got our eye on GitHub actions. I’m going to play with that shiny new thing and play with that a little bit, and get that all set up.

Josh Birk:

That’s our show. Now, before we go, I did ask John about his favorite known technical hobby and it turns out it’s something that sounds delightful in the age of a pandemic.

John Daniel:

I love going to the beach and driving on the beach. I go out to the outer banks, North Carolina a lot, and there’s several places you can just drive on the beach. But I love going to the beach. I’ll fly stunt kites and hang out on the beach, and just go all kinds of things. I’m quite content to just sit on the beach, and bring my laptop if I need to, or something like that.

Josh Birk:

Well, beaches are still closed here in Chicago. So I guess that will have to do. Thanks for listening, and my thanks to John for the great conversation. If you want to learn more about this podcast, head on over to Developer.Salesforce.com/podcast, where you can hear old episodes, see the show notes, and have links to your favorite services. Thanks again. And I’ll talk to you next week.